Command Line via Lasso

The text to support the talk at the LDC in Amsterdam 2009

Pier Kuipers, Visual ID, Dublin, Ireland

Overview

When building web applications with Lasso, there comes a time when you realise that there are things that cannot be achieved with Lasso alone. In the distant past, this became obvious as soon as you were trying to manipulate images or work with PDF files. Thankfully Lasso now has a lot of this type of functionality built in.

However, there still remain situations where you need to invoke the help of other applications to meet your requirements. In a server environment, these helpers will typically be command line (or cli for "command line interface") applications that function without the need for user interaction.

This talk aims to provide information on how Lasso can access these applications, as well as where to find them and how to install them. Since I have been developing on the Mac OS X platform since time began, I am only able to discuss my personal experiences in that particular environment.

The need for command line applications

There are numerous reasons why you would resort to command line applications to assist Lasso. Typically, you need to do something that Lasso simply cannot do, like converting a Video file from one format to another. Or you may wish to use applications where Lasso already provides similar functionality, but not quite in the way you need it.

Some examples:

• Working with Video files.
• Working directly with images in ImageMagick
• Interacting with Subversion or other source code control software
• Working with obscure file formats such as Microsoft Word

Learning to use the command line

Before you can tell Lasso to run commands from the command line, it would be useful to know what you're doing. A good understanding of the Terminal application on Mac OS X is assumed - there is plenty of material available online and in print to get you started with shell scripting and the like. Absolute beginners are advised to start with a book such as "UNIX for Mac OS X" by Dave Taylor, published by O'Reilly. There are also plenty of online resources to find out more about "shell scripting".

How Lasso can access the command line

There are three ways in which Lasso can access the command line:

• As of version 8, Lasso comes with the [OS_Process] module to provide this functionality.
• In addition, the third party [PassThru] module is being used by large numbers of Lasso users to provide the same feature.
• Then there is also the slightly contrived option to use Lasso's [Include_URL] tag to call scripts written in other languages to provide additional functionality.

 

Security Implications

Providing access to the command line brings security risks with it - the most obvious one is that the command line may provide access to the server machine's entire file system outside of the website root. This is one of the reasons why it may not be possible to have access to this feature in certain hosted environments. Although you can experiment in your own development setup as you wish, you may want to check with your hosting provider to see if you can actually take advantage of access to the command line.

The Lasso user and file permissions

When accessing the command line via OS_Process or PassThru, you will have to keep in mind that you are acting as the Lasso user - in other words, the "invisible" user account that was created when you installed Lasso. As a result, the directories or files on which you are performing your commands need to be accessible by Lasso. In the examples provided with this talk you will find a shell script called "hello.sh". Lasso will not be able to use this script if the permissions are not set correctly - they should look something like this:

-rwxr--r--@  1 lasso  admin    18 28 Sep 23:01 hello.sh

You can "run" this script in the Terminal by cd-ing into the directory, and then typing the name of the script preceded by a period and slash, and hit enter, like so:

cd /path/to/script/folder
./hello.sh

If the permissions are set correctly for the Lasso user, you will get a "Permission denied" error since you are not in fact that user. If however, you are having difficulty trying to get Lasso to run the script, try running this command in the Terminal to set the permissions as required by Lasso:

cd /path/to/webfolder; \
	sudo chmod 744 hello.sh; \
	sudo chown lasso hello.sh

Obviously you will need to replace the /path/to/webfolder with the actual path.

Keep in mind that the use of a Lasso module that accesses the command line will typically require authentication of some sort - so you wrap your code in an [inline] to take care of this:

	inline(
		-username=#os_process_user, 
		-password=#os_process_password
		);
		---command line stuff here---
	/inline;

There as another important issue to consider when running cli applications with Lasso - the lasso user is "homeless" and has no root privileges. What this means, is that some command line applications that require access to some sort of Graphical User Interface (be it the Mac's Finder or the X-window system on Unix - or indeed X-window on OS X) will not work when called by Lasso. Something that works fine when you're typing commands in a terminal window may inexplicably fail when Lasso runs the same commands, and this may be because the Lasso user is not actually logged in. Similarly, commands that require authentication:

shell('sudo ls -al');

will not work - there's nobody home to type in the password. Note that the above example uses the Shell tag to run the command, explained further on.

Trying to use cli apps that require login to remote servers - Subversion over ssh and rsync spring to mind - provide their own challenges. You cannot issue a command that requests a password and then simply provide the password in the next command: the reason is that every time you issue a command, Lasso starts afresh - it's as if you were opening a new Terminal window and typed your commands.

If you wish to run commands that require authentication to a remote server, you will need to set up shared keys between your Lasso server and the remote server in order to remove the password prompts.

OS_Process

Although Lasso comes with this module as standard, it does not function "out of the box". Because of the security implications, it actually stays in the box when Lasso is installed, so to speak. After installation of Lasso, you will find the OS_Process module in a folder called "Extensions" - On Mac OS X, the module is called "OS_Process.dylib". You will need to manually move this file into the LassoModules folder and restart Lasso.

When the OS_Process module is first loaded, it will create a group called OS Process Users. A user must be in this group to be able to utilize the process type.

See the Lasso manual for full installation instructions and information on setting up users and groups. The manual also includes some examples of running commands through OS_Process. In my opinion, running shell commands with OS_Process can be a little cumbersome due to the number of parameters required. Jason Huck's Shell tag provides welcome respite and is discussed later on.

The example below lists the contents of the current directory:

	local('fullpath'='/Library/WebServer/Documents');
	var('os'=(OS_Process(
		'/bin/ls', 
		(Array(
			'-al', 	
			#fullpath+(response_path)
			)
		))));
	var('script_output'=($os->Read));
	$os->Close;
	'<pre>'+$script_output+'</pre>';

Note that the first element of the OS_Process command is a string indicating the full path to the "ls" cli app - /bin/ls - followed by an array with two elements to provide the arguments to be passed to this app - "-al" plus the path to the directory.

PassThru

Lasso developers have been taking advantage of Steffan Cline's PassThru tag since long before Lasso itself provided the OS_Process tag. PassThru is donation-ware and can be obtained from http://www.execuchoice.net/downloads. Installation is straightforward, just copy the EC_passthru.dylib file into the LassoModules folder and restart Lasso. The beauty of PassThru is its simplicity - you simply feed it a string with the command as you would in the Terminal:

PassThru('ls -al');

to list the files in the current directory. No need for separate strings and arguments in an array, everything is just presented in a single string.

Include_URL

Although this Lasso tag does not provide access to the command line per se, it can be utilised to access scripts written in languages other than Lasso to provide additional functionality. I once managed to find a local developer to write some code for one of my applications - but he wrote it in Perl. It did exactly what it was supposed to do and I did not have the time the rewrite it in Lasso. Instead, I used [Include_URL] to call the script with the necessary parameters to extend Lasso's (or rather my application's) functionality.

We need to use this tag rather than a straightforward [include] for the simple reason that the page calling the tag is processed by lasso, and as a result Lasso would also attempt to process the code in the included file also. With [include_url] however, the code referenced by the URL will be processed entirely separate from the page calling the tag. The URL could be calling some code within the same site but processed by some other language (such as Perl in my example), or it could simply pull in some information from an entirely different, remote server.

The question is sometimes asked if Lasso can coexist on the same page with code from some other language - and with [include_url] this can indeed be done in a roundabout way. A very simple example is a Lasso page running some PHP code - where the Lasso page looks like this:

	'Lasso says today is ' + (server_date) + '<br>';
	local('fromphp'=(include_url('http://'+(server_name)+'/info.php')));
	'PHP says today is '+#fromphp;

And the info.php page referenced by the [include_url] tag looks like this:

< ?php echo(date('m\/j\/y')); ?>

Calling the Lasso page will then return something like the following result in a browser:

Lasso says today is 9/25/09

PHP says today is 09/25/09

Of course it should be noted that the page called by [include_url] will more than likely not be able to use the Lasso session variables, and any required parameters must be passed in the URL, something like

	include_url('http://www.mydomain.com/info.php?parameter='+$var);

The Shell tag

To simplify the use of the OS_Process tags, you can use Jason Huck's [Shell] tag which is available from TagSwap.net. You will also nee to download Jason's [response_localfolder] from the same site. Install both tags in LassoStartup and restart Lasso to activate the tags - not forgetting to also install OS_Process, of course.

Using the Shell tag you no longer have to worry about the various parameters required by OS_Process since these are now handled by the tag. As with PassThru, you can issue command in almost the same way as you would use them in the Terminal. For example:

	shell('ls -al');

will list the files in the current directory.

Running a sequence of commands

More than likely you will want to run several command in sequence to achieve your goal, for example cd into a directory and then perform a few actions on the files in it. It may be tempting to try to run two commands after each other, something like

	shell('cd /Library/WebServer/Documents');
	shell('ls -al');

- only to find that the result is not what you expected. As I've explained earlier on, the reason is that every time you issue a command, Lasso starts afresh as if you were opening a new Terminal window and typed your commands.

One solution is to string several commands together following the rules of shell scripting, something like

	shell('ls -al /Library/WebServer/Documents/');
	shell('cd /Library/WebServer/Documents/; ls -al');
	shell('cd /Library/WebServer/Documents/ && ls -al');

An important consideration in using cli applications via Lasso, is to get used to specifying full paths to the files you're working with every time.

For complex interaction with the command line however, you would be better off writing a separate script and use your Lasso tag to call that script instead.

Where to find command line applications

The internet is awash with developers producing all sorts of utilities and applications accessible from the command line. A good starting point when looking for certain functionality is http://sourceforge.net/. Google has also jumped on the bandwagon, http://code.google.com/hosting/. Not everyone is part of these mass collections however, and Googling can bring up interesting results. For a large number of years now, Swiss developer Marc Liyanage has provided Mac OS X installer packages for Unix cli apps on his website http://www.entropy.ch.

Some command line applications, such as ffmpeg, constitute large projects maintained by a number of developers. Others are little gems put together by some guy in an attic somewhere. That's OK, but you may want to check if there has been any recent activity or if there are any reports about the software, before you decide to build your entire application around something that may no longer be supported.

If you're on Mac OS X, don't be put off by some of the system requirements you may come across. If it's a Unix-style application, it may well be possible to successfully build and run it on Mac OS X.

Some of my own personal favourites include the following:

• catdoc - http://www.wagner.pp.ru/~vitus/software/catdoc/ - Reads the textual content Microsoft Office documents
• ffmpeg - http://ffmpeg.mplayerhq.hu/ - Reads and converts video files
• ImageMagick - http://www.imagemagick.org - Image manipulation software, "Photoshop from the command line"; Used internally by Lasso for its [image] tags
• PDFtoText - http://www.bluem.net/downloads/pdftotext_en/ - Reads the textual content from PDF documents, amongst other features
• GhostScript - http://www.ghostscript.com - Converts PDF and EPS files to raster image previews, amongst other features

Installing command line applications

If you want to "compile" or "build" some recently discovered cli applications, you will first need to install Apple's Developer Tools (XCode) which can be downloaded free of charge from the Apple Developer's website - http://developer.apple.com/mac/. It is quite a large download and usually comes with Mac OS X Server or as an optional installation on the client version, so you may already have it on disk. Installation will add some background utilities that are necessary to compile applications from source.

There are several ways to go about the installation of your newly found utilities. Some applications come as proper Mac installer packages, but they are far and few between. It is sometimes possible to download source code that is meant for some other Unix-style OS and install it successfully on Mac OS X.

Once you get your hands on some Unixy source code, it will probably be called something like "unixapp-1.2.3.tar.gz", which is a gzip compressed "tarball". You can unpack this type of file in the finder, which will give you a folder called something like "unixapp-1.2.3". Inside of this folder you will find a bunch of files, most of them with meaningless names and extensions, but just ignore that. Although it doesn't really matter where you put this folder - you can usually delete it when you've finished installing - I tend to keep these source code folders inside a folder called "source" in my Applications folder.

To build or compile your application, use the Terminal to cd into the folder you just unpacked and run the commands as follows:

cd /Applications/source/unixapp-1.2.3
./configure
make
sudo make install

You will see a whole range of commands flashing across your terminal window in between each command, and if all goes according to plan you can now run your commands.

Dependencies

Of course not every cli app is completely standalone - it's possible that some other cli apps are required to build the app of your choice. For example, ImageMagick requires various libraries such as JPEG and PNG to work properly. Usually, these kind of parameters are added to the configure command but are not for the faint-hearted. If there is no installer package available, then a better option to deal with this kind of thing is to use MacPorts.

Where command line apps live

Once you have finished installing your application, the next move is usually trying to do something with it. So you try and type in one of the commands your app is supposed to provide. Say you've installed ImageMagick and want to convert an image from one format to another - the command would then be "convert". So you type "convert" in the terminal and hit enter - and now one of two things may happen:

• ImageMagick kicks in and tries to run the command " but because you haven"t specified any parameters it will just spit back all sorts of other information about versions and help and what not. But at least you know it"s working.
• The terminal reports: command not found

When the latter happens, al is not lost. It does not necessarily mean that ImageMagick is not installed, just that your "shell" doesn't know where to find it. So the question is: where IS my application? Usually, cli apps get installed into an invisible folder called bin (for binary). That may still cause confusion since there are various places where this folder may exist, such as /opt/local/bin, /usr/local/bin, /usr/bin etc. Your "shell" account will usually "remember" these locations so that you can simply type the command without specifying the path, but sometimes you may have to explicitly set this path - and the way this is done varies between operating systems and versions. On Mac OS X 10.5 it requires adding a line like this like this:

export PATH=/usr/local/bin:$PATH

to the invisible .profile file inside your Home folder. This will allow you to type in the name of any script inside the /usr/local/bin folder without having to type the full path.

What's important, is that not only do you need to know the path to your cli app, but so does Lasso. You can "ask" the Terminal to lend a helping hand by typing "which convert" and hit enter:

which convert
/opt/local/bin/convert

If it is indeed installed, this returns the path to the default installation of the "convert" binary - but keep in mind that it is entirely possible to have more than one installation of the same cli app. For example, I have two versions of ImageMagick installed, one installed from a Mac installer package and living in /usr/local/bin, and another installed with MacPorts (see below) and living in /opt/local/bin.

All this leads to one important consideration: when using command line apps via Lasso, get into the habit of specifying the full path to the binary, e.g. /opt/local/bin/convert.

Ideally you would store this path in a global Lasso variable so that it can easily be changed, should you reinstall the cli app or move your code to a different machine:

	var('pathTo_ImageMagick'='/opt/local/bin/');
	shell($pathTo_ImageMagick+'convert yada yada');

MacPorts

If you are unsuccessful in compiling your cli app, all is not lost. Most popular command line applications are available on MacPorts - a very useful and user friendly tool for Unix stuff on Mac OS X. Go to their website http://www.macports.org/ and install the program. As with the installation of "raw" cli apps, you need the XCode Developer Tools (see above).

The macports website provides ample information on how to install and run the macports program - which, incidentally, is itself a command line app.

The biggest advantage of using MacPorts, is that it will automatically download and install any dependencies that your app may require. Using the MacPorts program is delightfully simple - to install an application, you just type "sudo port install" followed by the applications name and hit return, enter you password, and MacPorts does the rest.

By default, MacPorts will install the cli applications in /opt/local/bin/ - this is mainly to make sure that its installations do not interfere with any default installation that may already exist in /usr/local/bin/, and also that the apps in /opt/ do not get overwritten by possible software updates.

Fink

Fink - http://www.finkproject.org/ - is a similar solution to Macports. I Haven't used this one myself, but just want to point you in this direction for completeness sake. Again, the XCode Developer Tools are required.

Lasso Variables and CLI Apps

Hopefully you will be doing more with command line apps and Lasso than just listing the contents of a directory with ls -al. More than likely you will end up building long strings consisting of numerous variables and bits of text, which you then eventually feed to your cli app. Make sure that you leave room for spaces in between parameters as you would when typing commands in the Terminal.

There are two things I would strongly advise when working with cli apps:

• Build your entire command and store it in a variable, and only then "feed" it to your cli app. This will allow you to log the contents of the variable and take a look if something goes wrong. You'll be able to copy the text output from this variable and paste it into a terminal window to run it - this way you'll quickly discover what, if anything, went wrong.
• When or building variables that contain file paths (other than the full path to the cli app itself), always, always, always surround each file path with quotes. I have been bitten more than once by commands that failed mysteriously and intermittently, only to discover that the file path sometimes contained spaces - and this will cause mayhem.

Taking those two point on board, your code should look something like this:

local('cmd' = ($pathTo_IM+'convert "'+$original+'" "'+$copy+'"'));
log_critical('Command running is:  '+#cmd);
shell(#cmd);

Resources and Further Reading

• OS_Process - see the Lasso Manual or the online reference
• PassThru - http://www.execuchoice.net
• Shell Tag - http://tagswap.net/shell
• SourceForge - http://sourceforge.net
• MacPorts - http://www.macports.org
• Fink - http://www.finkproject.org/
• Apple Developer Tools (Xcode) - http://developer.apple.com/mac/

Author: Pier Kuipers
Created: 29 Sep 2009
Last Modified: 25 Mar 2011