IOException - Disk Quota Exceeded error - yet there's lots of disk space

I use a Mac for work.  I've run into an interesting issue with making backups of client web sites and zip files with my Mac...  It seems that whenever I've zipped a file and backed it up to my Mac, I get these wonderful __MACOSX folders - that's 2 underscores at the beginning of the folder name injected into the folder mix.

Restoring a site from one of those backups on a linux machine can have adverse consequences because of those 'injected' __MACOSX folders.  It seems that certain flavours of linux really don't like that folder name and as a result will not write a single file to the machine any more.... at all.  Instead, it throws an a 'Disk Quota Exceeded' error.  You can try touching a file, you can try unzipping some other files - whatever you do, you'll get that error.   The error didn't go away for me until I deleted all __MACOSX folders that got copied up to my machine in the zip file.  Once I did that - it was all good.

The interesting thing about this error was I could have tons of disk space still on the machine, and yet it would still throw that error until I removed those offending __MACOSX folders.

There are various commands out there for removing these folders from zip files created on a Mac... something like this:
zip -d your-archive.zip "__MACOSX*"

Another emergency software fix

A couple of weeks ago I had another client take advantage of my Emergency Software Troubleshooting services.  They had a web site maintained by a popular CMS system.  Unfortunately, their scheduled backups weren't as scheduled or robust as they thought and they were left without a website (or backend database) after their hard disk died.  I was called to help.

After some invesigation, we surmised we were left with:
- a bunch of .myd, .myi file backups for MySql (they were a couple of months old)
- another .sql file backup of the database 12 months old
- a file system backup of the site a couple of months old
- and another file system backup around 12 months old.

We rebuilt the database using a combination of the most recent myd and myi files.  This was a bit of a trick as the .frm files were missing in some cases.  What we did in those cases was create a dummy .frm file, rebuild the tables from the myd/myi files and when we did that we had mysql replace the frm file (it's an option you can enable during this operation).  We then used the data from the older database to populate the tables that didn't have the frm files.

With the file systems, it was a matter of patching together what worked between the two file system backups with the data we had in the database.  Since the old system was pretty much blown away, it was decided to upgrade the OS and the web server at the same time.  This meant reconfiguring the web server and ensuring the majority of the site worked.  In the end, the site was pretty much ready to go after 8.5 hours of work.

More Software Emergency Fixing Experiences

I've recently helped two more clients with 'emergency fix' issues.

The first issue was a problem a national grocery chain had with an export from a SAS program that did a pull from multiple database tables into an excel spreadsheet, and subsequently imported this data into a MS Access database using a SSIS (Sql Server Integration Services) package.  The SAS export seemed to be fine - that is, it executed without errors, but the SSIS package would fail. From my understanding, the client would run this process about every month or so, so there were a fair number of records (1000's).  However, they started having this issue in Dec. and I didnt' get called about the issue until late April, so there was a backup of data.
The client didn't have the SQL BI studio installed, so we had to go through the idx file manually to see if there was anything obviously wrong.  There didn't appear to be.  The database and excel connections tested fine. There didn't appear to be anything riotously wrong with the data formatting or invalid characters in the excel spreadsheet.  So, what do you do?
Well,  for one thing, don't test with the full dataset if you think it might be a data related issue.  Keep making your dataset smaller until you are sure that none of the data works.  In this case, we cut the dataset in half, started from Feb to the end of Apr and the import worked!  This confirmed for us that there were no permissions issues, or software upgrade issues (we were considering that software upgrades might be part of the problem as more than one piece of the software had been upgraded in that time period).  The successful test confirmed for us that we were dealing with a data related issue.  From there it was just a matter of determining exactly which rows of data couldn't be imported, and what it was about the data in those rows that botched things.
Less than 5 hours of work to this point on this fix.

With the second emergency, a telecom company was having issues getting an automated FTP (file transfer protocol) process to work.  They had changed the paths of where the ftp files were residing and they were voip experts, not so much into the bash shell scripting thing.  Their first problem was pathing in the bash file.  That was an easy fix.  Then there was a problem with sending a notification email that apparently they never got to work.  This was a little bit more tricky.  They originally had the email coded similar to this example.

       #!/bin/bash
       telnet smtp.example.org 25 <<_EOF
       HELO relay.example.org       MAIL FROM:<joe@example.org>
       RCPT TO:<jane@example.org>
       DATA       From: Joe <joe@example.org>
       To: Jane <jane@example.org>
       Subject: Hello
       Hello, world!
       .
       QUIT       _EOF

However, after a bit of work I discovered that the server this script was running on was processing all the command too quickly for the smtp server to get the email off.  In the end, I had to do something more like this:

       #!/bin/sh
       ( echo HELO ccielogs.com       
       sleep 2
       echo MAIL FROM:<test@somedomain.com>
       sleep 2
       echo RCPT TO:<test@someotherdomain.com>
       sleep 2
       echo DATA
       sleep 2
       echo Subject:Test-Mail!
       sleep 2
       echo If you can read this, it Works!!!
       sleep 2
       echo .
       sleep 2
       echo QUIT
       ) | telnet XXX.XXX.XXX.XXX  25

Then to set the bash script to run in a windows scheduler (using cygwin) I had to do something like this:

       C:\cygwin\bin\bash.exe --login -c "/myfolder/myscript.sh" 

Less than 5 hours to fix.

Replatforming an app in 21 hours....

I got a cold call a couple of weeks ago. A prominent institution in Calgary had a java application that ran on a Sun OS with an iPlanet web server that was key to their business that didn't work with a key upgrade from a 3rd party vendor. It needed to be upgraded an running in production 2 weeks. Could I help?
I agreed to help and showed up that afternoon to see what could be done. The current code in production ran fine. However, there was no guarantee it would run fine with the upgrade. Unfortunately, the code base was a mess. Multiple versions of the same file were all over the production server with .bak, .ver2, etc extension. Everything had been done on the fly in the past. The logs couldn't provide us with any useful information, and they had no test environment.
After seeing what could be done on the Sun box (which had been running this application non-stop for more than 10 years!), I decided to try and port the application to Tomcat - in this container I knew what I was doing and I could debug the problems more effectively. Within 14 hours we had the application running on Tomcat with some minor bugs.
Over the weekend, new production and test virtual machines were requisitioned and when I came back on my next visit, we got the last of the bugs out of the way and had everything working in the new test environment.
Some lessons learned....
- Don't be afraid to replatform in Java. I would've had a much harder time trying to do this with Microsoft technology. The clients were VERY happy to get off that old server. No one was around to support it anymore...etc. That was 10 year old java code that I didn't have to do a thing to when I moved it onto j2sdk1.4.2_XX.
- It's almost scary how many different ways you can configure things in Tomcat. We had outstanding issue that was 'bugging' me after 14 hours of work. The client required the application to be available from two separate url paths without using an apache web server to do url rewriting. I tried various different configurations in the app's web.xml, the global web.xml, the server.xml and found that I could get the url's to work, but then I had problems with sessions getting lost/corrupted. In the end, we wrote another servlet (sort of like this, but different) that we configured to 'catch' the url we needed it to and redirect the request to the controller servlet that was already mapped to the other url. This worked beautifully.
- We discovered that if your dealing with pages that have a lot of scriptlet code that does response.sendRedirect(someUrl), these method calls will sometimes throw IllegalState exceptions. The (quick) way to get around throwing the exception is to add a return; statement right after it. Or you could put that kind of logic into a servlet if you have time. I didn't (have time).
- Having good logging is critical to debugging an app in an emergency. Seeing my System.out.println() outputs in the tomcat console was like drinking hot chocolate after a day of skiing. So RIGHT! Of course I commented out those lines before we put the code into production.
- Using technologies that don't required registry keys makes configuration and multiple environment installations very easy. I found that I could write two short paragraphs of directions (documentation?) and that was enough for my client to install java and Tomcat into their new test environment by himself. (Pretty much copy - paste, and add the JAVA_HOME system variable and add the j2sdk...\bin onto the Path system variable). They were very happy about that too.
- Having hard coded path references to properties files in java code is so nasty! Every time we changed the path, we had to recompile all our classes. I know that's not the right way to do things, but the client just wanted to get things working and worry about refactoring later.