Monitoring Lasso with monit

Setting up Lasso and monit (a system monitoring tool) on OS X

This process has been developed and tested on Mac OS X. Monit is also available on linux systems but I have not tested on linux at all. The basic process should be the same but the start and stop scripts will probably need to be adapted.

This assumes that you currently have Monit up and running on your Mac OS X machine, as well as Lasso installed and working.

On my example server I have Lasso configured using multiple sites. Site 1 is a placeholder and shouldn't normally serve any requests. Site 2, 3, etc handle all normal traffic, and requests are routed based on the hostname. Setting Site 1 up this way is generally considered good practice, and can help you to detect when you make mistakes in your apache.conf files, amongst other things.

For this example setup I'm going to use:

http://site1.mydomain.com (Site 1)
http://site2.mydomain.com (Site 2)
http://site3.mydomain.com (Site 3)

We'll need these unique domain names when restarting an individual site later on.

The servers IP address is 192.168.1.1 and is on my internal network 192.168.1.0/24 .

Step 1 - Install Start/Stop scripts

We'll need a start and a stop script which will be called by monit.

Create these 2 bash scripts in your general purpose scripts folder and check that they are executable. I'm going to use /usr/local/bin/ for these examples

/usr/local/bin/startLassoService.sh

#!/bin/bash
#
# Start Lasso Script for use with Monit
# VER. 0.9 - http://www.hotblack.co.nz/
# 2009 wade at hotblack.co.nz
#
#=======================================

case $1 in

0)
	sleep 6

	rd_pid=`ps -o pid,command -au lasso | grep [L]asso8Service$ | awk '{print $1}'`
	if [ "$rd_pid" ]; then
		echo "Lasso Service is running"
	else

		sleep 2
		killall Lasso8Service
		sleep 6

		/Library/StartupItems/LassoPro8/LassoPro8 start

		sleep 2

		ctime=120
		counter=0
		while [ $counter -lt $ctime ] && [ ! -f /tmp/LassoService8_0.pid ];
		do
			sleep 1
			counter=$((counter+1))
		done
	fi
	;;

*)
	sleep 5
	/usr/bin/curl --max-time 15 http://$1 > /dev/null
	sleep 2
	/usr/bin/curl --max-time 25 http://$1 > /dev/null
	;;

esac

exit 0

	#!/bin/bash
#
# Kill Lasso Script for use with Monit
# VER. 0.9 - http://www.hotblack.co.nz/
# 2009 wade at hotblack.co.nz
#
#=======================================

LassoPID=`/bin/cat /tmp/LassoService8_$1.pid`

case $1 in

0)
	sleep 2
	killall Lasso8Service
	sleep 2
	/Library/StartupItems/LassoPro8/LassoPro8 stop > /dev/null
	sleep 14
	killall Lasso8Service
	sleep 2
	rm -f /tmp/LassoService8_*
	;;

*)
	/bin/kill $LassoPID
	rm -f /tmp/LassoService8_$1.pid
	;;

esac

sleep 2

exit 0

My example server is a G5 2.3DP setup with 8 Lasso sites. The sleep commands (in particular sleep 14) are based on how this machine has behaved. You may want to tweak the values in this script based on your machine specs and load.

Step 2 - Get Lasso to create .pid files

As Lasso starts, each site loads and runs any lasso files in the main LassoStartup folder, as well as any files in it's own LassoStartup folder. We'll use this behaviour to automatically write the pid for the current site to a .pid file in /tmp/. Each site will write it's own .pid file as it loads.

We'll also get each site to check the pid for the main Lasso process (I'm going to call it Site 0) and write a .pid file if it doesn't exist or the .pid file is out of date.

Add the RD_lassoService_pid ctag (from tagswap.net) to your main LassoStartup folder (/Applications/Lasso Professional 8/LassoStartup/) then restart Lasso. This tag also requires the OS_Process tag to be installed.

server:/ wade$ cd /Applications/Lasso\ Professional\ 8/Tools/
server:Tools wade$ ./stopLassoService.command
server:Tools wade$ ./startLassoService.command
server:Tools wade$ cd /

server:/ wade$ ls -l /tmp/Lasso*
-rw-rw-r--  1 root   wheel  70792 May 27 16:19 /tmp/LassoService8.out
-rw-rw-r--  1 lasso  wheel      5 May 27 14:40 /tmp/LassoService8.pid
-rw-rw-r--  1 root   wheel      6 May 27 14:38 /tmp/LassoService8_0.pid
-rw-rw-r--  1 lasso  wheel      6 May 27 14:39 /tmp/LassoService8_1.pid
-rw-rw-r--  1 lasso  wheel      6 May 27 14:40 /tmp/LassoService8_2.pid

You should have a series of LassoService8_X.pid files, where X is the siteID (0, 1, 2, etc).

LassoService8.pid is created by Lasso itself, but as each site loads it writes to the same file, so we end up with the pid of the last site to be started. Not very useful so ignore this file.

We can now monitor the CPU and memory usage for any individual Lasso site process.

Step 3 - Configure monit

Monit is configured to use /etc/monitrc for the main config file. I use monit's ability to include addtional config files. This is useful since it makes it easier to disable particular processes from being monitored, by simply renaming the .monitrc file in question. These additional config files are located in /etc/monit

erver:/ wade$ ls -l /etc/monit/
total 112
-rwxrwx---  1 admin  admin   570 Apr 24 01:58 apache.monitrc
-rwxrwx---  1 admin  admin  3603 Apr 19 04:58 lasso.monitrc
-rwxrwx---  1 admin  admin   419 Apr 19 04:56 mysql.monitrc
-rwxrwx---  1 admin  admin   468 Apr 19 04:56 svn.monitrc~

Note: The svn.monitrc~ file is currently disabled and svn won't be monitored by monit.

/etc/monitrc

set daemon 60 with start delay 60
set logfile /var/log/monit.log

set httpd port 2812 and use address 192.168.1.1
	signature enable
	allow localhost
	allow 127.0.0.1
	allow 192.168.1.0/24
	allow admin:monitpassword
	allow monit:monitpassword

check system Dev
	if loadavg (1min) > 4.5 for 3 cycles then alert
	if loadavg (5min) > 3.5 for 5 cycles then alert
	if loadavg (15min) > 2.5 for 10 cycles then alert
	if cpu usage (user) > 90% for 10 cycles then alert
	if cpu usage (user) > 80% for 15 cycles then alert
	if cpu usage (system) > 45% for 5 cycles then alert
	if cpu usage (wait) > 20% for 3 cycles then alert

check filesystem Boot with path /
	if space usage > 75 % then alert

include /etc/monit/*.monitrc

/etc/monit/lasso.monitrc

check process lasso_0
	with pidfile /tmp/LassoService8_0.pid
	start program = "/usr/local/bin/startLassoService.sh 0"
		with timeout 120 seconds
	stop program = "/usr/local/bin/killLassoService.sh 0"
		with timeout 45 seconds

	if changed pid then alert

	if totalcpu > 15% for 2 cycles then alert
	if totalmem > 75 MB for 2 cycles then alert

	if 3 restarts within 5 cycles then timeout


check process lasso_1
	with pidfile /tmp/LassoService8_1.pid
	start program = "/usr/local/bin/startLassoService.sh site1.mydomain.com"
		with timeout 120 seconds
	stop program = "/usr/local/bin/killLassoService.sh 1"

	group lasso
	if totalcpu > 25% for 2 cycles then alert
	if totalmem > 100 MB for 2 cycles then alert
	depends on lasso_0


check process lasso_2
	with pidfile /tmp/LassoService8_2.pid
	start program = "/usr/local/bin/startLassoService.sh site2.mydomain.com"
		with timeout 90 seconds
	stop program = "/usr/local/bin/killLassoService.sh 2"

	group lasso			
	if totalcpu > 25% for 2 cycles then alert
	if totalmem > 150 MB for 2 cycles then restart
	depends on lasso_0

We need to use the unique domain names/URLs for the different sites in this config file.

When a site needs to be started/restarted, the site URL (site1.mydomain.com, etc) is passed to the startLassoService.sh script, which then uses curl to load a page from the site. If the site isn't currently running, this will cause Lasso to start it (which in turn creates the new .pid file). The string passed to the start script will need to be a URL that is processed by Lasso. If your default index file isn't automatically processed by Lasso, then change it to something that will be, like:

	start program = "/usr/local/bin/startLassoService.sh site2.mydomain.com/index.lasso"
		with timeout 90 seconds

With my sites and servers, Lasso is generally well behaved. It hasn't gone wildly out of control and swamped the CPU. It's memory usage does however continaully grow over time, so periodically restarting individual sites frees up some memory and has prevented my machines from hitting virtual memory. For this reason I've set monit up to send alerts if any limit is exceeded, but only to restart if the memory usage is higher than allowed. My example server has a relatively low traffic volume, so I've set the cpu and memory limits quite low. You will need to set your limits and actions to what ever values are appropriate for your setup.

Create Site 0 pid (optional)

I've also modified the Lasso8Service.sh to create the .pid file for Site 0 when Lasso first starts.

Edit /Applications/Lasso Professional 8/LassoAdmin/Lasso8Service.sh with your favourite text editor and add

	rd_pid=`sleep 8; ps -o pid,command -au lasso | grep [L]asso8Service$ | awk '{print $1}' > /tmp/LassoService8_0.pid` &

near line 33, just before the line that reads

	cd /Applications/Lasso\ Professional\ 8/

So the script should end up roughly like

	umask 002

# RD .pid Changes
	rd_pid=`sleep 8; ps -o pid,command -au lasso | grep [L]asso8Service$ | awk '{print $1}' > /tmp/LassoService8_0.pid` &
# RD .pid Changes

	cd /Applications/Lasso\ Professional\ 8/
	sudo -u lasso ./Lasso8Service > /private/tmp/LassoService8.out 2>&1

Now when Lasso first starts up (even before any sites are loaded) the LassoService8_0.pid will be created.

Sorted. 8)

With all of this set up you should now be able to monitor an individual lasso site process, or even Lasso itself. For even more fun, get M/Monit and M/Monit for iPhone and you can monitor your servers on the go. If anything needs further explanation or is just plain incorrect let me know on LassoTalk or direct email (wade at hotblack.co.nz)

Author: Wade Maxfield
Created: 27 Apr 2009
Last Modified: 16 Mar 2011