Starting a SRE (Site Reliability Engineering) Practice

SRE & Your Business 

When considering implementing an SRE department, it's one thing to read Google's SRE book and implement it the way they did in their IT centric organization.   I've found the implementation to be a bit different in organizations where IT is not the entire driver behind the business. I think it's very likely to be more of a challenge to implement in an environment where IT could be considered more of a utility to the business rather than the business moneymaker. Frankly, the majority of businesses out there are like this. In my experience, I discovered that a SRE team can encounter a fair bit of push-back to new 'priorities', 'automation ideas', and 'procedures' from other IT departments within the enterprise that have already established IT operation procedures and routines. Our team needed to evangelize and prove themselves as an SRE department before getting traction in getting our agenda considered with other teams.

Monitoring 

Members of new SRE teams might be excited about all the opportunities for automating in an organization. As a fledgling SRE team, you actually might find that much of your early automation ends up revolving around the monitoring and triage of production incidents. After all, the 'R' in SRE stands for Reliability. One of the first things our team did that created value beyond our practice in the enterprise was create monitoring dashboards from tools like AppDynamics and Splunk. These dashboards displayed and alerted on pertinent, inter-team SLO's and SLA's - many of which we derived from production incidents we tracked. Tracking inter-team SLO's was important because we discovered that inter-team incidents/problems fell through the cracks until our SRE team started owning them. Correlating each production incident to a monitor/alert we created helped us ensure: 

• We could be proactive if similar circumstances aligned towards another analogous incident occurring, and 
• We could likely pinpoint how to resolve the new issue because it had happened in the past and had an existing incident ticket and past resolution. 

Keeping an automated eye on the enterprise systems in production, we discovered more insights intrinsic to reliability in the organization:

• We began to anticipate and predict new production incidents. This gave us a leg up on being proactive in preventing the incident, or at the very least, the resolution. Managers and operations teams found this information valuable which gave us a foot in the door for further discussions to socialize our agenda. 
• We began to see correlating alerts across disparate departments associated with specific incidents, which allowed us to drive the problem to a root cause, and in some cases, several root causes with contributing factors in disparate configuration, architecture, and code. 

Automation from an operations/monitoring perspective definitely trumped any automation we did to improve IT deployments.

Challenges to Reliability 

In an organization where the business is not IT centric, Google's concept of the Error Budget sounded great, but was much more difficult to enforce, or even get any kind of commitment on. The main reason was really quite simple - the business that made the money drove what features/projects went into production and when, no holds barred. The business held the trump card - they made the money - and so long as we could keep things running that paradigm didn't seem like it was going to change. Other challenges to reliability that we ran into early on included:

• Either non-existent or out of date documentation which caused us problems on multiple fronts, including network upgrades and developing/consuming internal APIs for application services. 
• Configuration management issues Issues with data integrity across multiple systems. 
• Lack of discipline and followup surrounding post mordems 

Collaboration 

Once high priority incidents were discovered that weren't owned by teams or a department, our team took ownership of those incidents and drove them through to resolution. This generally meant communicating and working towards the resolution with many different teams across the enterprise, and helping those teams understand the root cause and enabling/empowering those teams to build the fix & resolution. This kind of cooperation allowed us to get a good reputation within the organization and helped the teams responsible for the root cause feel like they also 'owned' and contributed to the solution in a positive, empowering way.

Expectations 

Getting positive results (ROI) from a new SRE team doesn't happen overnight. Unless you've handpicked seasoned developers from your existing development/operations team, it will take time for your newbies (even if they have past experience as an SRE) to get a grasp on your business & technology domain. Their ramp-up time may depend on how you've structured the implementation of the team itself. Some questions that might have a bearing on this:

• Is it an entity on its own, or embedded across the development/operations teams? Google and Facebook use embedded team models. We started with an autonomous team, but began to head towards an embedded model. 
• Are other teams aware of what the SRE team will be doing and how this new team can potentially help them? 
• How current is your documentation?

Random Thoughts About the DevOps Movement in IT - Part 2

The Phoenix Project (image of the book on the left) was recommended to me over a year ago as a good read about DevOps.  It emphasizes the benefits of Dev Ops to management, shareholders, and a company.   Impressions I had of the book were:

• Its a great, fun read, however it skims over the difficulties and trials of the actual automation of the technical systems and software development process - where the rubber really hits the road.  It seems to practically romanticize the idea of automation in IT a bit - like DevOps is the goose that will lay your golden egg.  Unfortunately, there's significantly more work to get that golden egg in my experience.
• It also glosses over how to get the Security team on board with what DevOps wants to do.  In many companies, the Security team holds the trump card and if they decide to change all your certs from 128 bit encryption to 2048 bit encryption (don't laugh, I've seen it happen and we had to regenerate certs for all applicable servers in all envs).  Their wish is your command unless you can convince someone influential that that kind of encryption is overkill in a non-prod environment.

Finding and Cracking That Golden Egg
Hurdles I've encountered enroute to the DevOps 'golden egg' are:

• Silo-ed Application Projects.  
• Lack of consistent naming conventions for deployment artifacts and build tags/versions is majorly detrimental to the automation process. 
• Lack of understanding of the dependencies between application projects (API contracts, or library dependencies).  If you don't understand your dependencies between applications, they may not compile corectly, or they may not communicate properly with each other.
• Lack of consistent development methodology and culture between teams.  If you have one team that doesn't get behind the new culture, and they insist on manually deploying and 'tweaking' their code/artifacts after deployment, they risk the entire release.  Getting everyone on board with culture is challenging.  
• Overusing a tool.  When you have a hammer (a good DevOps tool), everything is a nail.  Generally, many of the DevOps tools out there all have their niche.  You could potentially use Chef, Puppet, or Ansible to do all your provisioning and automation.  Is that the best solution though?  I'm inclined to say no.  How much hacking and tweaking are you having to do to get all your automation to work with that one tool?  Use the tools for what they are best at, what they were originally made for.  Many of these tools have Open Source licenses and new functionality is being added to them all the time.  While it might seem that the new functionality is turning your favourite tool into the 'one tool to rule them all',  you might have a big headache getting there, trying to push a square peg in a round hole.  
• Lack of version control.  Everything must be stored in a repository.  The stuff that isn't will bite you.  VM Templates, DB baselines, your automation config - it all needs to go in there.
• DB Management.  All automated DB scripts should be re-runnable and stored in a repo.  
• Security.  How to satisfy the Security Team?  How to automate certificate generation when they are requiring you to use signed certs?  How to manage all those passwords in your configuration?
• Edge Cases.  Any significantly sized enterprise is going to have/require environments that land outside your standard cookie cutter automation.  Causes I've seen for this are:
• Billing cycles - we needed a block of environments that were flexible with moving time so QA didn't have to wait 30 days to test the next set of bills and invoices.  
• Stubbing vs. Real Integration - Depending on the complexity of integration testing required, there may be many variations of how your integration is set up in various environments - what end-points are mocked vs. which ones point to some 'real' service. 
• New Changes/Requirements - Perhaps new functionality requires new servers or services.  This can make your support environments look different that your development environments.
• Licensing issues.  When everything is automated, it can be easy to loose track of how many environments you have 'active' versus how many you are licensed for.  License compliance can be a huge issue with automation - check this interesting post out 'Running Java on Docker?  You're Breaking the law!'
• Downstream Dependencies   This is where the Ops of DevOps comes into play.  Any downstream dependencies that your automated system might have need to be monitored and understood.  You can't meet your SLA with your client if your downstream dependencies can't meet that same SLA.  Important systems to consider here are: LDAP, DNS, Network, your ISP, and other integration points.
• YAML files.  Yes, perhaps they are more terse than storing your deployment config in XML.  However, I'm at a bit of a loss to see how they are better than a well named and formatted CSV file.  Sure, you can 'see' and manage a hierarchy with them.  But you can do the same in a CSV file with the proper taxonomy.  YAML files utilize more processing power to parse and have extraneous lines in them because their trying to manage (and give a visual representation of) the hierarchy of the properties.  I've seen YAML files where these extra lines with no values account for a significant percentage of the total lines in the file, making the file more difficult to maintain and prone to fat fingers.  Several major DevOps tools use these files and I really can't see a good reason why except that they were the 'new, cool thing to do.'

IOException - Disk Quota Exceeded error - yet there's lots of disk space

I use a Mac for work.  I've run into an interesting issue with making backups of client web sites and zip files with my Mac...  It seems that whenever I've zipped a file and backed it up to my Mac, I get these wonderful __MACOSX folders - that's 2 underscores at the beginning of the folder name injected into the folder mix.

Restoring a site from one of those backups on a linux machine can have adverse consequences because of those 'injected' __MACOSX folders.  It seems that certain flavours of linux really don't like that folder name and as a result will not write a single file to the machine any more.... at all.  Instead, it throws an a 'Disk Quota Exceeded' error.  You can try touching a file, you can try unzipping some other files - whatever you do, you'll get that error.   The error didn't go away for me until I deleted all __MACOSX folders that got copied up to my machine in the zip file.  Once I did that - it was all good.

The interesting thing about this error was I could have tons of disk space still on the machine, and yet it would still throw that error until I removed those offending __MACOSX folders.

There are various commands out there for removing these folders from zip files created on a Mac... something like this:
zip -d your-archive.zip "__MACOSX*"

Dell Latitude Diagnostic Error Code 2000-0142 Hard Drive Failure

'Apparently' the hard drive on my laptop failed yesterday.  Out of the blue I got a couple of popup windows complaining about some dll not being able to load, and then the windows blue screen of death.  I thought maybe my machine got too hot, but I tried rebooting after letting it cool off and it didn't do anything after the Dell splash page during the boot.  So I ran a BIOS diagnostic and got the following error:
Msg: Error Code 2000-0142
Msg: Unit 1: Drive Self Test
Failed Status Byte=77
I called Dell and they basically confirmed what the diagnostic test said - my hard drive was fried.

Not wanting to give up on my hard drive that easily, I googled to see what I could do.  I discovered I could try to repair windows using the Windows Recovery Console.  So I found my Windows CD and booted from it and went to the Repair option.  Once I had entered the Admin password, I got a command prompt and after a bit of searching around I discovered that most of my hard drive seemed to be intact.  This, however, was still in the dos prompt of the Windows Recovery Console.  I couldn't back up my files to a USB from there.

I tried chkdsk, chkdsk /p, and chkdsk /r - those all worked successfully.  Yet my machine still wouldn't book.  So I preused the list of commands available to the Windows Recovery Console (here) and tried a few more - bootcfg, fixboot, and fixmbr.  In the end, it was fixmbr (which repairs the master boot record on the specified disk) that ended up working for me.

My laptop is 4 years old now.  I removed the battery a couple of months ago, and with problems like this starting to happen, I going to start shopping for a new one.  Hopefully this info helps someone out there though.

IIS and setting NTAuthenticationProviders and authNTLM

More than once now I've run into situations with the NTLM configuration in IIS where it seems that the GUI setting doesn't do all you need it to to properly configure integrated windows authentication for a web site. Earlier this week I was asked to look into a situation for a particular project team where Firefox wasn't prompting them to enter a username/password in one environment (they'd just get an http 401.2 error), and yet it was giving them the prompt in another environment.  (IE was running fine in both environments, by the way).

I went through all the IIS GUI configs in IIS and the two boxes looked identical.  Since this appeared to be an issue with integrated authentication, I ran this script:

cscript c:\inetpub\Adminscripts\adsutil.vbs GET W3SVC/WEB_SITE_Number/ROOT/NTAuthenticationProviders

which showed me that the working server had nothing set (wasn't set), while the server that was giving them problems returned with 'Negotiate,NTLM.'  I subsequently set NTAuthenticationProviders to '' and it worked by using 'set' instead of 'get' in the above script and providing an empty parameter.

Note:  I understand the above setting to be different than the one below.  In more than one instance at work I've had this issue and proven that this is the case.  The script below, from my understanding, the same as the Windows Integrated Authentication checkbox on the Directory Security tab in IIS.

cscript c:\inetpub\adminscripts\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/authNTLM TRUE

Excessive logging kills AppPool (log4net)

We had an issue where one of our Application Pools (IIS 6) was 'dying'.  The interesting thing about it was while the application itself was dead (error page, sometimes showing an outOfMemory exception, other times showing a meaningless exception), the appPool itself appeared in the GUI to be running.  The more troubleshooting I do with Windows OS's, the more I do not trust what the GUI says.  Appearances in Windows can be deceiving.  For several weeks - until we understood the root cause of the issue - are resolution to the dead application was to restart the 'running' appPool. 

A side note - this appPool was running on a Network Load-balanced server, on the dominant node.  The appPool for this particular application only ever failed on the dominant node.  It never failed on the other node of the cluster, so we had reason to believe that the 'load' had something to do with the root issue.

In the end, we discovered the root cause of our dying application by accident.  We were cleaning up our files on the prod servers when it became apparent to us that log4net on another application (and another appPool) was set to log 'ALL'.  This resulted in 1.5Mb of log created about every 5 minutes.  I had a hunch that all this logging was having a memory impact on the server - we certainly didn't want to be logging at the 'ALL' level anyway, so we created a change request and fixed that issue ASAP.  Then we crossed our fingers, hoping that our dying AppPool issue would go away as well. 

It's been more than two weeks since we implemented this log4net change and our flaky appPool hasn't died once.  Lessons learned:

- Watch for excessive logging on prod servers
- Excessive logging can apparently cause other appPools to through outOfMemory Exceptions!

Windows VM False Performance Monitor Stats

We almost exclusively run VM's now in most of our environments. Recently, we added two new Windows VM's in production for a particular solution.  Thinking we were being prudent, we asked to have the boxes provisioned with 4 Gigs of Memory since that's what the old 'hardware' boxes had. We deployed our solutions to the new boxes and tested them without a hitch.  So a week later a switch was flipped and the new VM's went live while the old boxes no longer took requests.  After a few days, we discovered (by looking at the task manager) that we were getting some conflicting and concerning readings for memory utilization. Check this out:

- the task manager graph said we were using 3.7 gigs of memory
- looking at the actual numbers below the graph, we only had 300Mb of memory available on the machine
- looking in the processes tab, none of the processes were using a significant amount of memory (several were using between 200-300MB, but that was within normal tolerances when compared with servers running the same system in other environments.  However these same processes where paging memory like they shouldn't have been!  Page faults were very high.
- Totaling the memory used in the individual processes, we were NOT using 3.7 Gigs of memory.

After doing some more investigation, we asked the team that built the server how the memory had been provisioned for these new boxes.  It turned out that they both had originally been given 1Gig of memory and then later (after they were running) 3 more Gigs were added.  At this point our hunch was that these  boxes weren't seeing the additional 3Gigs that had been subsequently added.  After doing a hard reboot on both boxes, this turned out to be the case.

Lesson learned: make sure you follow VM procedure (hard re-boots) for adding/removing resources to VM servers.  Short-cuts can create some pretty confusing statistics in the OS.

Debugging/Troubleshooting SQL performance/blocking issues

I picked up a good tip from one of our DBA's yesterday.  He uses sp_who2 to see which users have active threads/processes running against a particular database.  This stored proc returns the PIDs as well (Process ID number).  I had known about this previously.  What was new to me was he also used:

dbcc inputbuffer(pidNumberHere)

to get the database to tell him which stored proc was being executed by that PID.  That seems to me like a handy trick.  It certainly solved the problem I was having.

SMTP email issue with hosts file

I had an interesting issue last week.  I had a website with a contact page that uses SMTP sendmail to email a list of people whatever was entered on a form.  The curious thing about this issue was it would send emails to everyone in the list except email addresses that matched the domain name of the web site.  For example, it would send an email to perry.mckenzie@blah.com, but it wouldn't send an email to perry.mckenzie@continuouslylearning.blogspot.com, even though the server that was doing the sending was listed in the DNS as 'continuouslylearning.blogspot.com'.

I tried changing the email provider by pointing the Email Host to a server that I knew accepted anonymous email pushes, but it did the opposite - sending the emails successfully for users from blah@continuouslylearning.blogspot.com, but not sending them for blah@anyOtherAddress.com!

After looking through various logs (including /var/log/maillog), it seemed like sendmail was getting confused as the log was trying to associate the email address (someone@continuouslylearning.blogspot.com) with a user on the host and not finding the user.

sendmail failed error: localhost: someone@continuouslylearning.blogspot.com user not found

I tried removing an entry in the /etc/hosts file that pointed the IP address to continuouslylearning.blogspot.com, but that didn't seem to work.  What I didn't realize until later (with the help of my hosting provider) is that this was the correct change to make, I just needed to restart two services after I made the change:
service network restart
service sendmail restart
Then my email worked perfectly.

Windows OS Performance Guidelines

These are notes from a workshop I was able to be a part of a couple of weeks ago.  The workshop was on Windows Performance Monitor and monitoring vital signs on Windows Servers.  The windows performance monitor has a multitude of counters that a person could potentially monitor on a server.  I'm just going to point out a few critical ones that were highlighted in the workshop and what their tolerances are.

Some points to note before I get into these values:  

1. Don't always look at the graph first. The counter graphs can be scaled right out to lunch so they aren't necessarily a good first glance indicator.  For a particular counter, your focus should first be on the Minimum and Maximum values in perfmon just below the graph.
2. In the past, I would tend to keep an eye out for sympathetic counter relationships on the graphs.  However, keeping an eye out of inverse counter relationships is also a good idea (where one counter could be decreasing in value while another is increasing)
3. Counters can get corrupted.  Apparently this happens more often than one would think.  They can be rebuilt - directions are in this KB post.
4. You can attach a PID (process ID) to a perfmon counter in an OS older than Windows Server 2008 by modifying reg keys. Details are in this kb article. Versions of perfmon on 64bit OS's come with this already set up.

On to the objects/counters of note:

• Process Object
• Handle Count - greater than 500 handles may point to a problem.  
• Private Bytes - greater than 250MB could be a problem.  I've seens procs over a Gig and they definitely were a problem (they can get that high).               
• Working set - greater than 250MB could be a problem
• Thread Count - Greater than 500 threads needs to be watched to ensure they aren't increasing over time
•  Processor Object
• Processor Time - all core instances.  _Total can get you an overall trend.  Greater than 91% utilization is potentially an issue.
• Network Interface Object - you need to know what the spec is for your network interface to determine it's capacity.  Anything over 80% of capacity could point to a problem
• Current Bandwidth - help you determine the NIC's capacity
• Output Queue Length - greater than 2 is an problem
• Bytes Total - greater then 65% of capacity utilized is past the warning threashold (blinking read with siren)
•  Memory Object
• Free System Page Table Entries - the higher the better here.  Lower than 5000 is considered critical.  I've seen boxes 'run' (aka hobble) around 2500.
• Available Megabytes - again higher here is desirable.  Less than 100 MB or 5% free is very problematic
• Pool Non Paged Bytes - greater than 80% consumed is out of spec (not at all good)
• Pool Paged Bytes - Same as non-pages bytes.  Anything between 60-80% should be watched.
• Logical and Physical disk Objects - they have the same critical counters so I've put them together here
• Idle percentage - 19 to 0 percent is critical. Anything over 55 is warning.
• Current or Avg Disk Queue Length - 3 to 31 in the queue - you need to keep an eye on it.  greater than 32 is an issue
• %Avg Disk Sec Read or Write - 25ms and above is critical

 Other interesting points of note:
- Mark Russinovich was the original developer of Perfmon.  His blog is apparently pretty good and our facilitator was very impressed with him.
- Windows Server 2008  does processor/core parking.  This means the server will 'retire' (effectively turn off to save power) CPU's when there isn't a heavy load on the box.  Our facilitator told us that one ISP moved all their boxes to Windows Server 2008 for this reason and their power bills were 15% less a month. 

Another emergency software fix

A couple of weeks ago I had another client take advantage of my Emergency Software Troubleshooting services.  They had a web site maintained by a popular CMS system.  Unfortunately, their scheduled backups weren't as scheduled or robust as they thought and they were left without a website (or backend database) after their hard disk died.  I was called to help.

After some invesigation, we surmised we were left with:
- a bunch of .myd, .myi file backups for MySql (they were a couple of months old)
- another .sql file backup of the database 12 months old
- a file system backup of the site a couple of months old
- and another file system backup around 12 months old.

We rebuilt the database using a combination of the most recent myd and myi files.  This was a bit of a trick as the .frm files were missing in some cases.  What we did in those cases was create a dummy .frm file, rebuild the tables from the myd/myi files and when we did that we had mysql replace the frm file (it's an option you can enable during this operation).  We then used the data from the older database to populate the tables that didn't have the frm files.

With the file systems, it was a matter of patching together what worked between the two file system backups with the data we had in the database.  Since the old system was pretty much blown away, it was decided to upgrade the OS and the web server at the same time.  This meant reconfiguring the web server and ensuring the majority of the site worked.  In the end, the site was pretty much ready to go after 8.5 hours of work.

Licenses.Licx error with DevExpress in Visual Studio 2010

While refactoring a business solution at work to deploy using Visual Build rather than InstalShield, I ran into an issue with our DevExpress web components not seeing the correct licenses in the licenses.licx file.  After googling for a bit and discovering how the licenses.licx files are created and manged in VS2010, I started troubleshooting to see if I could resolve my issue - seeing the red text on every web page telling me that I'm using a 'trial' edition of the software when I clearly wasn't.
 
I deleted the file to get VS to recreate it.  That didn't seem to work for me.  I renamed it.  I checked permissions on it.  I investigated how it got deployed and whether I needed it on the server or not (if I remember correctly, you don't have to explicitly move it to the server).  In the end, my solution sort of knocked my socks off.  I deleted all of the content in the file and left it blank.  I then built my msi and deployed.  Imagine my surprise when the blank licenses.licx file got rid of my trial license message!  I don't know if this is a bug with VS2010 or it's integration with DevExpress 8.2.2, but my fix seemed like a total hack.  However we've tested this in multiple environments and it's solid.

Log4Net performance issue with WCF

We ran into an interesting production problem recently that had an interesting resolution. After upgrading our application server to use WCF calls, we found that performance was lagging in production. We were able to replicate this problem in our development environments and spent a good deal of time trying to resolve the issue.  Avenues we explored included profiling calls to the databases (using Sql Server Profiler and filtering on database name and login Id), mining activity logs in the database, pouring over configuration files, utilizing Wireshark,  SysInternals Process Explorer , SysInternals ProcessMonitor (and filtering by processName and PID), and the Windows PerfMon (performance Monitor - viewing the defaults and various performance objects).

Interesting things I learned/discovered were:
- IE browsers (IE 7 and older) default to only two connections to the web server See this link
- It seems IE processes javascript before rendering a page while Firefox process javascript after the page is rendered in some cases
- The default for WCF connections to the database is serializable rather the read committed!
- WCF connections default to some low threasholds for ConcurrentCalls, ConcurrentSessions, and ConcurrentInstances.  These can be changed using the serviceThrottling node and attributes of a service behavior you create.  See this link for more detail
- In dos, you can filter a netstat to only show specific ports and statuses like this:
netstat /n | find "ESTABLISHED" | find "8085"
- Use /n with netstat to not resolve dns names - without /n, dns name resolution can make netstat take a while.
- In dos, you can use the 'tasklist' command to show something akin to a process list in the Task Manager
- Retrieve IIS process names from the command line by executing this command:
cscript c:\WINDOWS\system32\iisapp.vbs

In the end, the resolution to our problem was to turn off a custom implementation of Log4Net that had running.  In the process monitor we saw that log4net was re-initializing, re-reading it's configs, and re-creating all the log files for EVERY page request.  We discovered this because it seemed that there was a several second lag time between when the request was executed on the browser and when the web server actually acted on it.  Disabling log4net entirely improved our performance by a significant amount. (10's of seconds)

Update - Nov 26, 2010.  One of the reasons we were running into the issue above is the log4net configuration loader was placing a lock (being very pessimistic) on the configuration file every time it was loading.  Agreed, this solution should likely have been using a static logger - one instance for the application.  However, given the way it was implemented, I'm thinking that we could have avoided a bunch of hurt if we had made the configuration loader a little more optimistic and used:
<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />

Common Obfuscated Error Messages with IIS

I've noticed that IIS tends to obfuscate error messages - displaying messages that are different than the actual issue that is happening to the application.  This may very likely be because we are running a plethora of web sites with different architectures and history on one instance of IIS.  I thought I document some of those 'tricky' error messages here just in case someone else runs into them (or I forget what they really mean).

1. 'An existing connection was forcibly closed by the remote host'  I ran into this error numerous times trying to configure a WCF web service to bring back a large result set (60k + rows).  Playing with the values for maxBuffer PoolSize and MaxReceivedMessageSize in the web.config, I came to discover that this error was really seemed to be masking was an out of memory error that forced the shutdown of the AppPool that was serving the web service.

2. 'BinaryFormatter Version Incompatibility errror'  Generally I've seen this error when you have IIS site on a web server trying to communicate with another IIS site on an application server, and there is no response from the IIS site on the application server - either that web site isn't running, or your url pointing to the web site on the application server is wrong.

Key Questions in Software Sustainment

Here are some of the questions I ask myself when a software system has stopped running properly...

The first big question is: What changed between when the application was running well, and the time the application stopped running well?  To get to the bottom of that question, there are a number of other questions that can help point the way.

Question isolation - Is this problem isolated in any way?  Is it only on a specific network, environment, or group of servers?  Does it only happen for a specific group of users or a specific client?  Is there a period of time it's isolated to?  Is it isolated to a particular 'item' in your data?  I've seen users try to make an application  use a different browser version than the documented supported versions.  Some times it takes a while to get to the bottom of simple issues like that.  Also, we've run into situations where an organization will not have given users the rights on their machines to install third party active X components for their browser that the application they are trying to use requires.

Question data integrity - Whether you are looking at legacy data or no logic to manage special characters in your data, you need to break the problem down. With legacy data, is the problem isolated to a particular user or group of users, or a particular item?  Or if it's an ETL function, does relational integrity from the first DB line up to the second DB correctly - are you missing or adding 'types' of data that the second DB either is or isn't expecting?  Sometimes you need to pinpoint the exact row or time when the issue occurred to determine what the problem was.

Question continuity - Did something stop running or listening? A service, appPool, web site, 3rd party server?  Are your cron jobs or scheduled tasks still there? Monitoring would quickly and easily answer this question for you.

Question communication -  Are the lines of communication open to all of the dependencies that your application has?  Has a network cable been severed by a backhoe down the street? (I've seen that before)  Is there too much communication going on?  Too many calls from one routine can take you system down.  We had an issue like that with some javascript that called a data access function to a GIS server.  As soon as one too many layers got added to the map, performance died as there were too many round trip calls to the server in one request. Internal systems that need to do identity verification or IP Geolocation are heavily dependent on external third parties to operate.  These external vendors can in turn can be dependent on other external services.  Know all your dependencies and have monitoring and SLA's in place for all of them or you could be sweating bullets.

Question dependencies - Here I'm thinking more in the context of internal dependencies - what internal 3rd party services are you dependent on?  Databases, reporting tools, monitoring systems, document management systems,  any internal system that your application depends on that you aren't responsible for fits into this category.  Are they up and running?  How do you know?  Are they running properly?

Question the un-questionable - Is your hvac system working at your Co-Lo?  As redundant as your service provider tries to be, there will always be something under the radar.  Always.  I've seen an external hvac system take down an entire enterprise.  Many of you have likely seen a McAfee or a Norton patch take down an enterprise.  How secure is your UPS (Uninterrupted Power Supply) management console?  I've logged into one I've found by accident using username:admin password:password!

Question known changes - software enhancements and patches, file or db permissions, config changes, etc. I've seen issues where a changed file path will take down a critical Ftp routine, a 3rd party software patch for a document management system crippled a production system by removing key indexes in the database, and an (unfortunately un-automated) database refresh will have missing roles or users or the db will still be restricted mode.  Doing anything manually can get you into trouble.  A fat finger can push a wrong dll/library or mis-type an entry in a config file.  Even fat fingering an automated deploy can get you into problems.  Automated deploys still depend on data that is manually entered.  We have pointer to app servers in our web.config files entered wrong and multiple entries in machine.config for a particular component.

Question security - Malicious changes are possible either internally or externally.  Unfortunately, the question of security is a larger one everyday.  And it has to be scrutinized at every level.  Does your system have a firewall and IP Sec rules in place?  Does you application provide you with an audit trail?  How secure is your production data? 

Resources at your disposal that can help in your investigation:
Log files - event logs, server logs, if you're logging to tables in the DB, don't forget to look there.
Users - it's like CSI - you need to get ALL the information you can about their problem.  You cannot be afraid to ask.
Thread dumps - killing a hurting server and ensuring that it does a thread dump when it terminates can be very effective in your problem search
Networking tools - WireShark, telnet, ping, netstat - these are great for checking your communication.
Monitoring tools - Nagios, SCOM, GroundworkOpenSource
Books -  Michael Nygard's Release It! and Luke Hohmann's Beyond Software Architecture.

More Software Emergency Fixing Experiences

I've recently helped two more clients with 'emergency fix' issues.

The first issue was a problem a national grocery chain had with an export from a SAS program that did a pull from multiple database tables into an excel spreadsheet, and subsequently imported this data into a MS Access database using a SSIS (Sql Server Integration Services) package.  The SAS export seemed to be fine - that is, it executed without errors, but the SSIS package would fail. From my understanding, the client would run this process about every month or so, so there were a fair number of records (1000's).  However, they started having this issue in Dec. and I didnt' get called about the issue until late April, so there was a backup of data.
The client didn't have the SQL BI studio installed, so we had to go through the idx file manually to see if there was anything obviously wrong.  There didn't appear to be.  The database and excel connections tested fine. There didn't appear to be anything riotously wrong with the data formatting or invalid characters in the excel spreadsheet.  So, what do you do?
Well,  for one thing, don't test with the full dataset if you think it might be a data related issue.  Keep making your dataset smaller until you are sure that none of the data works.  In this case, we cut the dataset in half, started from Feb to the end of Apr and the import worked!  This confirmed for us that there were no permissions issues, or software upgrade issues (we were considering that software upgrades might be part of the problem as more than one piece of the software had been upgraded in that time period).  The successful test confirmed for us that we were dealing with a data related issue.  From there it was just a matter of determining exactly which rows of data couldn't be imported, and what it was about the data in those rows that botched things.
Less than 5 hours of work to this point on this fix.

With the second emergency, a telecom company was having issues getting an automated FTP (file transfer protocol) process to work.  They had changed the paths of where the ftp files were residing and they were voip experts, not so much into the bash shell scripting thing.  Their first problem was pathing in the bash file.  That was an easy fix.  Then there was a problem with sending a notification email that apparently they never got to work.  This was a little bit more tricky.  They originally had the email coded similar to this example.

       #!/bin/bash
       telnet smtp.example.org 25 <<_EOF
       HELO relay.example.org       MAIL FROM:<joe@example.org>
       RCPT TO:<jane@example.org>
       DATA       From: Joe <joe@example.org>
       To: Jane <jane@example.org>
       Subject: Hello
       Hello, world!
       .
       QUIT       _EOF

However, after a bit of work I discovered that the server this script was running on was processing all the command too quickly for the smtp server to get the email off.  In the end, I had to do something more like this:

       #!/bin/sh
       ( echo HELO ccielogs.com       
       sleep 2
       echo MAIL FROM:<test@somedomain.com>
       sleep 2
       echo RCPT TO:<test@someotherdomain.com>
       sleep 2
       echo DATA
       sleep 2
       echo Subject:Test-Mail!
       sleep 2
       echo If you can read this, it Works!!!
       sleep 2
       echo .
       sleep 2
       echo QUIT
       ) | telnet XXX.XXX.XXX.XXX  25

Then to set the bash script to run in a windows scheduler (using cygwin) I had to do something like this:

       C:\cygwin\bin\bash.exe --login -c "/myfolder/myscript.sh" 

Less than 5 hours to fix.

Network isolation

For the second time this week, production has gone down for an extended period of time. This time it was because of this bug introduced by McAffee. It was proliferated through all of our environments (apparently) by SSCM. I'm not sure why we are putting McAffee fixes right into production without testing them in a staging environment first. Seems to me like there is a wrong process there somewhere - Either on our side, McAffee's side, or both.
Before we realized this was a McAffee bug, it seemed that this 'virus' got propagated to production from lower environments. Having the production environment on an isolated network from the rest of our development and test environments would have protected us to some degree from this kind of propagation. I have (officially) suggested this more than once, but resourcing and other priorities have gotten in the way of making it a reality.
For future reference though, here's just another reason to have a production environment on an isolated network.

Single Points of Failure

Within the last 8 months, all of the production servers for a client of mine were moved out of the building they've been in and into a 'bunker', or co-lo. There was a variety of reasons for doing this. As an architect choosing a Co-location provider, and planning the move, you would want to make sure as many systems there are as redundant as possible - power supply, UPS system, networking, etc.

Well, this week everyone painfully discovered a system that wasn't redundant (or if it was, it wasn't as redundant as it should have been). Apparently the HVac systems went down. This resulted in servers getting too hot and consequently having to be shut down. Production servers.

Everything was resolved in a couple of hours, but it just goes to show, it seems that there is always one system that is forgotten.

I've actually run into another situation like this before. It wasn't nearly as big an issue for our shop as it only affected development environments. However, the Calgary airport(!!) along with most of NE Calgary was without a network connection just as long as we were. Apparently a construction crew was using a backhoe doing some digging at an intersection and accidentally cut a main networking cable that supplied most of NE Calgary with it's network connection. Kind of makes you wonder if municipalities should be considering redundant underground networking, doesn't it?

Networking Architects  work hard to eliminate as many single points of failure in their systems as they can.  Some are hard to control though.  Recently McAfee released a virus library update that wasn't tested properly and as a result shut down a plethora of systems across North America.  It thought a Windows dll was a virus (false positive) and sent thousands of systems into a perpetual reboot.  It would be an interesting bit of process engineering to figure out the best way protect your production systems from the most current viruses AND protect them from bugs like this sent out by the 'big' virus scanning companies at the same time.

The importance of Process (adodb.dll and SSRS)

We have a number of different development environments. Deployments follow a natural progression through these environments before they get put into production. We were getting reading for a release and doing deploys and testing in our Staging environment (our env just before prod in our progression). One morning recently, our Nunit tests failed after a run, complaining about not finding adodb.dll registered.

We were surprised because adodb.dll is a Microsoft dll and we certainly weren't deploying any dll's like that - for that matter, we hadn't done any deploys in the last day or two. So why were our Nunits failing suddenly?

Quickly looking into the event logs, we discovered that SSRS (Sql Server Reporting Services) had been (unbeknowst to us) uninstalled on our transaction server during the evening. I went and spoke with the infrastructure group who did the uninstall and they got very concerned as they were ready to uninstall SSRS in production the next evening.

In the end, it appears that uninstalling SSRS did, in fact, remove adodb.dll from the Gac on our transaction server. Our infrastructure group cancelled the uninstall from production. We are trying to impress upon them and others the importance of communicating changes like this even if the assumption is there is no impact. Fortunately, our process of percolating changes up in different environments managed to catch this change before it blew us up in production. However, it wasn't caught in the lower environments because it was unadvertised. Communication is key!

Things learned from Website moves

I recently moved several client sites off of servers that they had been on for the better part of a decade to new VPS boxes. They had MySql backends and used Java/Velocity CMS system (InfoGlue) hosted on Tomcat with an AJP connector to apache web servers on the front end. One site uses htdig for it's search implementation.

Issues (with resolutions) that I ran into...

1. GZIP'ing a running Tomcat instance
Before one of the sites was moved, a client tried to make a backup of the Tomcat instance that they were running under. They executed a:
gzip -r filename.gz ./tomcat/
which ended up recursively zipping all files under the tomcat directory. This crashed the running instance of Tomcat. (Perhaps trying to tar or gzip a running tomcat instance isn't a great idea. Copy it first and then tar/gzip that) Everything was quickly unzipped and they tried to restart Tomcat but it wouldn't start. After an hour of search, I was asked to help. I poked and prodded for about 90 minutes and discovered that all the webapps were missing the
/WEB-INF/web.xml
file. I don't know how/why gzip and upzipping would make these files disappear, but they were there anymore. So I replaced them and tried restarting Tomcat and everything came up.

2. Problems with MysqlDump
I had a couple of problems backing up the MySql database. First MysqlDump would execute, but not put all of the tables into the backup file. I discovered that I had run out of disk space. Then it turned out that mysqldump doesn't care about relational integrity when it creates the backup script so when I tried to import it into the new database I ran into errors complaining about foreign keys. To resolve this, I had to add set foreign_key_checks=0; at the top of my backup script. This allowed me to import it successfully.

3. Mysql connection pool Exhausted
After getting the db imported and the web server moved over, the application seemed to be running fairly nicely on the new VM's. Then I left it alone for a couple of hours. When I came back it was throwing connection pool exhausted exceptions all over the place. I could resolve this by logging back into InfoGlue's cms instance, but that wasn't the right resolution for me. So I did some checking. Many people suggest to set the wait_timeout parameter in the my.cfg file for mysql higher or lower. I tried that and it didn't seem to work. What I ended up doing was adding a line in my database.xml file:
<param name="validation-query" value="select * from someTable" />
This helps keep the connection alive by pinging the db with the query every so often.

4. MySql Data Directory Change
While trying to resolve the previous issue I was changing/adding parameters into the mysql my.cfg file. I did a copy/paste which had a number of properties, one of which was the pointer to the data directory for mysql. This pointer was different than what I was using. As a result, when I restarted mysql, all of my databases, tables, users, and data was gone. I freaked. Then after thinking about things for a little bit and checking through files one more time I realized the mistake I had made and commented that line out. Restarting mysqld brought all my databases, users, and data back again. Whew.

5. Getting HtDig going again
The original installation of HtDig on the old server hadn't been indexed in over two years because cron had been broken. I didn't have access to the root password to fix this problem so the client was very interested in getting it running again on the new server. I had no previous experience with HtDig.
I copied over all the files I could find related to HtDig off of the old server and installed them on the new box. After a few tries, re-indexing worked, but I still had a problem with displaying the search results page in the web site. It turned out that I was missing a virtual directory configuration in my httpd.conf file for the directory where htsearch was running from (as a cgi script). The only reason I figured that out was by using lynx (linux CLI web browser). After fixing that, I got my newly indexed results displaying on the web site.

6. Using /etc/hosts helps
I've found that the /etc/hosts file is a big help in moving sites like this - whether you want to test the site while the live site is still running, or quickly configure pointers to a server dependency.

7. MySql sql queries are case sensitive
One of the website moves I did involved porting a MySql database from a Windows box to a CentOs box. The dump on the Windows box made all characters for tables lower case. I didn't pay too much attention to this at the time. They imported the same way.
When I started up my Tomcat server, it tried to start up, but threw errors related to jdbc connection pool and a ValidateObject. After a bit of googling I discovered that this is related to the validate query (the query I wrote about earlier that checks the connection every so often to make sure it hasn't gone stale). I tried running that query right on the mysql box and it would run because the table name was all lower case. So I changed all my table names to camel case and things worked.

8. Issues with unknownDefaultHost and serverNodeName in Tomcat
My inability to set the serverNodeName was resolved by adding a line in my hosts file to point the IP of my box to the Host variable found when I run the 'set' comment in CentOS. My issue with unknowDefaultHost was resolved by going into the server.xml file and editing the Engine element's defaultHost attribute - changing it from my old domain name to 'localhost'

9. MySql error 1153 - Got packet bigger than 'max_allowed_packet' bytes
Got this error a couple of times with different DB imports. To resolve I needed to go to the my.cnf file (found sometimes in /etc/) and set that property like this:
max_allowed_packet=16M
for the mysqld (mysql daemon). Then I had to restart the daemon (/etc/init.d/mysqld restart) and I could run my import (mysql -p -h localhost dbname < dbImportFile.sql)