Things learned from Website moves

I recently moved several client sites off of servers that they had been on for the better part of a decade to new VPS boxes. They had MySql backends and used Java/Velocity CMS system (InfoGlue) hosted on Tomcat with an AJP connector to apache web servers on the front end. One site uses htdig for it's search implementation.

Issues (with resolutions) that I ran into...

1. GZIP'ing a running Tomcat instance
Before one of the sites was moved, a client tried to make a backup of the Tomcat instance that they were running under. They executed a:
gzip -r filename.gz ./tomcat/
which ended up recursively zipping all files under the tomcat directory. This crashed the running instance of Tomcat. (Perhaps trying to tar or gzip a running tomcat instance isn't a great idea. Copy it first and then tar/gzip that) Everything was quickly unzipped and they tried to restart Tomcat but it wouldn't start. After an hour of search, I was asked to help. I poked and prodded for about 90 minutes and discovered that all the webapps were missing the
/WEB-INF/web.xml
file. I don't know how/why gzip and upzipping would make these files disappear, but they were there anymore. So I replaced them and tried restarting Tomcat and everything came up.

2. Problems with MysqlDump
I had a couple of problems backing up the MySql database. First MysqlDump would execute, but not put all of the tables into the backup file. I discovered that I had run out of disk space. Then it turned out that mysqldump doesn't care about relational integrity when it creates the backup script so when I tried to import it into the new database I ran into errors complaining about foreign keys. To resolve this, I had to add set foreign_key_checks=0; at the top of my backup script. This allowed me to import it successfully.

3. Mysql connection pool Exhausted
After getting the db imported and the web server moved over, the application seemed to be running fairly nicely on the new VM's. Then I left it alone for a couple of hours. When I came back it was throwing connection pool exhausted exceptions all over the place. I could resolve this by logging back into InfoGlue's cms instance, but that wasn't the right resolution for me. So I did some checking. Many people suggest to set the wait_timeout parameter in the my.cfg file for mysql higher or lower. I tried that and it didn't seem to work. What I ended up doing was adding a line in my database.xml file:
<param name="validation-query" value="select * from someTable" />
This helps keep the connection alive by pinging the db with the query every so often.

4. MySql Data Directory Change
While trying to resolve the previous issue I was changing/adding parameters into the mysql my.cfg file. I did a copy/paste which had a number of properties, one of which was the pointer to the data directory for mysql. This pointer was different than what I was using. As a result, when I restarted mysql, all of my databases, tables, users, and data was gone. I freaked. Then after thinking about things for a little bit and checking through files one more time I realized the mistake I had made and commented that line out. Restarting mysqld brought all my databases, users, and data back again. Whew.

5. Getting HtDig going again
The original installation of HtDig on the old server hadn't been indexed in over two years because cron had been broken. I didn't have access to the root password to fix this problem so the client was very interested in getting it running again on the new server. I had no previous experience with HtDig.
I copied over all the files I could find related to HtDig off of the old server and installed them on the new box. After a few tries, re-indexing worked, but I still had a problem with displaying the search results page in the web site. It turned out that I was missing a virtual directory configuration in my httpd.conf file for the directory where htsearch was running from (as a cgi script). The only reason I figured that out was by using lynx (linux CLI web browser). After fixing that, I got my newly indexed results displaying on the web site.

6. Using /etc/hosts helps
I've found that the /etc/hosts file is a big help in moving sites like this - whether you want to test the site while the live site is still running, or quickly configure pointers to a server dependency.

7. MySql sql queries are case sensitive
One of the website moves I did involved porting a MySql database from a Windows box to a CentOs box. The dump on the Windows box made all characters for tables lower case. I didn't pay too much attention to this at the time. They imported the same way.
When I started up my Tomcat server, it tried to start up, but threw errors related to jdbc connection pool and a ValidateObject. After a bit of googling I discovered that this is related to the validate query (the query I wrote about earlier that checks the connection every so often to make sure it hasn't gone stale). I tried running that query right on the mysql box and it would run because the table name was all lower case. So I changed all my table names to camel case and things worked.

8. Issues with unknownDefaultHost and serverNodeName in Tomcat
My inability to set the serverNodeName was resolved by adding a line in my hosts file to point the IP of my box to the Host variable found when I run the 'set' comment in CentOS. My issue with unknowDefaultHost was resolved by going into the server.xml file and editing the Engine element's defaultHost attribute - changing it from my old domain name to 'localhost'

9. MySql error 1153 - Got packet bigger than 'max_allowed_packet' bytes
Got this error a couple of times with different DB imports. To resolve I needed to go to the my.cnf file (found sometimes in /etc/) and set that property like this:
max_allowed_packet=16M
for the mysqld (mysql daemon). Then I had to restart the daemon (/etc/init.d/mysqld restart) and I could run my import (mysql -p -h localhost dbname < dbImportFile.sql)

Tomcat Alias

As my previous post says, I went live with a new website on Friday. Well, within minutes of going live I realized we didn't have an alias set up for www.myNewSite.com and got my client's administrator to add it. Once it was made live, the alias still didn't work. I had it configured in my tomcat server.xml file, but no dice. What was going on?

When I installed tomcat, I got rid of all the default sites it comes with (and I thought I had gotten rid of all their configs as well. It turned out I hadn't. In the server.xml file the host was configured correctly, but the 'engine' configuration was still pointing to 'localhost' which wasn't in my host configuration anymore. So I pointed it to my new 'root' configuration for the server - so now it looks something like this:

Engine name="Catalina" defaultHost="www.myNewSite.com"

and the host lookes like this:

Host name="www.myNewSite.com" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false"
alias myNewSite.com alias

Replatforming an app in 21 hours....

I got a cold call a couple of weeks ago. A prominent institution in Calgary had a java application that ran on a Sun OS with an iPlanet web server that was key to their business that didn't work with a key upgrade from a 3rd party vendor. It needed to be upgraded an running in production 2 weeks. Could I help?
I agreed to help and showed up that afternoon to see what could be done. The current code in production ran fine. However, there was no guarantee it would run fine with the upgrade. Unfortunately, the code base was a mess. Multiple versions of the same file were all over the production server with .bak, .ver2, etc extension. Everything had been done on the fly in the past. The logs couldn't provide us with any useful information, and they had no test environment.
After seeing what could be done on the Sun box (which had been running this application non-stop for more than 10 years!), I decided to try and port the application to Tomcat - in this container I knew what I was doing and I could debug the problems more effectively. Within 14 hours we had the application running on Tomcat with some minor bugs.
Over the weekend, new production and test virtual machines were requisitioned and when I came back on my next visit, we got the last of the bugs out of the way and had everything working in the new test environment.
Some lessons learned....
- Don't be afraid to replatform in Java. I would've had a much harder time trying to do this with Microsoft technology. The clients were VERY happy to get off that old server. No one was around to support it anymore...etc. That was 10 year old java code that I didn't have to do a thing to when I moved it onto j2sdk1.4.2_XX.
- It's almost scary how many different ways you can configure things in Tomcat. We had outstanding issue that was 'bugging' me after 14 hours of work. The client required the application to be available from two separate url paths without using an apache web server to do url rewriting. I tried various different configurations in the app's web.xml, the global web.xml, the server.xml and found that I could get the url's to work, but then I had problems with sessions getting lost/corrupted. In the end, we wrote another servlet (sort of like this, but different) that we configured to 'catch' the url we needed it to and redirect the request to the controller servlet that was already mapped to the other url. This worked beautifully.
- We discovered that if your dealing with pages that have a lot of scriptlet code that does response.sendRedirect(someUrl), these method calls will sometimes throw IllegalState exceptions. The (quick) way to get around throwing the exception is to add a return; statement right after it. Or you could put that kind of logic into a servlet if you have time. I didn't (have time).
- Having good logging is critical to debugging an app in an emergency. Seeing my System.out.println() outputs in the tomcat console was like drinking hot chocolate after a day of skiing. So RIGHT! Of course I commented out those lines before we put the code into production.
- Using technologies that don't required registry keys makes configuration and multiple environment installations very easy. I found that I could write two short paragraphs of directions (documentation?) and that was enough for my client to install java and Tomcat into their new test environment by himself. (Pretty much copy - paste, and add the JAVA_HOME system variable and add the j2sdk...\bin onto the Path system variable). They were very happy about that too.
- Having hard coded path references to properties files in java code is so nasty! Every time we changed the path, we had to recompile all our classes. I know that's not the right way to do things, but the client just wanted to get things working and worry about refactoring later.