Monitor the right things

I started re-reading Release It! by Michael Nygard this morning on the commute into work. In his first chapter he talks about a (very small) issue that turns into a colossus and takes down an airline's check-in system. The system had a monitor configured and performing checks on it, but it turned out that it wasn't checking the right things (it was looking at the http port on transactional servers when it should have been looking at the RMI port).
It totally reminded me of something that happened about a month ago. We have a bunch of web applications that run on our production server. After fine tuning our monitoring to look at pages that the application has to apply logic to to server up (rather than a static home page) we found that our monitoring corresponded much closer to complaints from users.
Think twice about what you want to monitor and where to point it.

IIS and Nagios

Working with IIS on a particular app, we had a situation where the behavior of the web server seemed to be inconsistent with how we had configured it. We had configured it to allow Authenticated access based on Integrated Windows authentication only, no anonymous access. When we browsed to the page we wanted to view on the web server itself, the page came up fine. But when we tried to browse to the page from another box, we were required to authenticate. This was not expected since it should have been using the Integrated Windows authentication to log in transparently.
After comparing it with another server that worked and seeing that most of the GUI configuration looked identical, I started googling and found this article that answered the problem for the most part. The only difference we found was even the order of the different NTAuthentication providers matters - the working server had "NTLM,Negotiate"; the server that didn't work was set up as "Negotiate,NTLM". Changing the broken server to look like the other server fixed our problem.

Our infrastructure teams has been doing some changes and moved the configurations for the email servers a bit. Unfortunately, this wasn't transparent to my Nagios/GroundworkOpensource installation (since I was lazy and hadn't reconfigured the email 'from' address for host). What ended up happening was all the Nagios emails ended up getting sent outside the internal network (because the domain name was still 'localhost.localdomain') and getting caught by the public antispamm appliance we have. I ended up switching notifications to use 'service-notify-by-sendemail' instead of 'service-notify-by-email' and then overriding the host in the command to point to our local email server. And that worked. Similar to this post on the groundwork forum.

Groundwork/Nagios and Rockstars

I've implemented Nagios at a few different places. In my current position, I've implemented Nagios with Groundwork, and I'm really happy with it. Groundwork provides a 'community' (free) version of their tool on a VM imaged instance that you can just plunk into a VM client. We use the npre_nt agents to monitor specific services and metrics on a number of windows servers. I reconfigured some of the vbscripts in the remote agents to call different methods in the WMI api which I've detailed in this forum. I've also customized the check_mssql to call specific tables looking for issues in production data (querying for held up business processes and corrupted documents). In order to do this, I had to download freetds and install it on the VM, overriding the default installation target with /usr/local/groundwork/. Then the freetds.conf file gets put into /usr/local/groundwork/etc and you can configure the servers you want to call in there. I put two configs for each server I wanted to call in that .conf file as I wanted to be able to test from the command line. I found that groundwork ends up calling using the IP of the box - so each server conf has a proper domain name config, and a IP name config.
One other thing about configuring the Groundwork/Nagios installation - I had to configure the CentOS to point to our exchange server so the email notifications would work.

I've read a couple of of good tech books lately (I now own them both). Release It! by Michael Nygard, and Secrets of Rockstar Programmers by Ed Burns.

Got Something done today!

...or at least, it felt like it. After doing housekeeping with our production application yesterday at work, today was a 'sit down and code' day. Boy did it feel good. I had told myself that I had to get this charting solution finished today or it would turn into a waste of time. It turns out that FusionCharts are super easy to implement, and boy, do they look sharp.

In my 'solution', I'm using Nagios to provide the data, with its 'service-perfdata' process. That file gets big quickly. I just enabled it in the Nagios configuration on Friday last week, and I already had 12 Megs of data. Going through the Nagios documentation and forums, there doesn't appear to be any mechanism to 'roll' this file or archive it or buffer it.... It had me stumped for a while. Then I realized I could just do a simple 'tail -1000 service-perfdata.out > service-perfdata.out' in a shell script with a cron job, and presto - a simple fix.

So, mental note to self.... whenever you're coding a shell script (or any kind of script) that gets run by cron, make all the file paths in the script(s) absolute. I keep on running into this problem. Cron is ignorant about everybody's profile and position.

I just started coding with Perl this summer on this project, and I really like it. Working with arrays is tricky and not very intuitive. Otherwise, though, Perl is fantastic for whipping up something quick. Today I found out that you can use && and || in conditional 'if' statements. That was good. Also, something I've noticed in the past and ran into again today - the 'chop' and 'chomp' functions in perl don't seem to work like the documentation says they should. I ended up having to something like this to get rid of spaces '$var =~ s| ||g' - basically using sed (or the like) to get rid of spaces I didn't want. Perl's localtime() function to convert unix to to something readable worked slick for me today. Another thing I ran into not too long ago - the proper way to concatenate strings in Perl is with '.'

Rode my bike into work today. Just had the pedal axle fixed, and wow, what a difference. The aphids are all over the place right now. After a ride they are just plastered all over me. It's hard to not breathe them in while one rides.

Nagios and things part 2

So I got to work this morning (with a fresh mind) and after a bit of googling figured out my problem in 30 minutes. Basically it boiled down to adding a reference to ORACLE_HOME in the bash script itself, and the full path of SQLPlus. After that, things ran fine even with changing the permissions of the scripts and related files back to normal and running Nagios with the Nagios user.

One of the 'nice to haves' for this little monitoring things I'm working on in Nagios is graphing capability - so we can see trends of the performance data at a glance. Just before I left today, I discovered that Nagios might install with this capability commented out. I let you know for sure in my next post.

While I was working on solving the graphing problem a different way, I got focused on some graphing addins for Nagios that required perl modules I didn't have. I learned that cspan is basically like 'yum' for perl. Didn't know what it was before today. In fact, I haven't had any exposure to perl before this project. It's all good - learning new stuff every day.

I also learned a big way to correct my golf swing today. I need to grip hard with my left hand and the right hand is just for support. I was driving much better and more consistently after I made that adjustment.

Nagios and things

I've done a couple of installations of Nagios now, and I quite like as a monitoring tool. Today I was working on coding a custom plug-in so we could keep an eye on a particular KPI. This involved running something of a complex query in Oracle and then sending the results to Nagios.
Writing the shell script wasn't difficult. The Nagios plug-in developer documentation was clear enough for me to follow. It took me a few minutes to figure out how to return the codes that Nagios is looking for - 'exit 1, or exit 2, etc'. However, to try and get Nagios to automatically run SQLPlus is a totally bear. I'm going to sleep on it tonite. I've tried changing perms on all the files, even tried changing the ownership to oracle and setting the SUID - no dice. Tried running Nagios as root (not recommended, but I had to try), and set all the Oracle environment variables in the .bash_profile for all the users I could think of that might possibly be running that code. I tried doing a su -c 'call my script here' oracle, and that didn't work either.

Oracle posts online regarding SQL*Plus 43 initiation errors etc. are pretty sparse. In my opinion, it's just more fodder for people who advocate using 3rd party free ware. At least if it's broken, somebody has likely run into the same issue and posted it online. With Oracle and Tibco, it seems like they guard their support forums with their life (maybe Oracle not so much so).