Continuously Learning

Dell Latitude Diagnostic Error Code 2000-0142 Hard Drive Failure

2011-07-18T22:40:00.000-07:00

'Apparently' the hard drive on my laptop failed yesterday. Out of the blue I got a couple of popup windows complaining about some dll not being able to load, and then the windows blue screen of death. I thought maybe my machine got too hot, but I tried rebooting after letting it cool off and it didn't do anything after the Dell splash page during the boot. So I ran a BIOS diagnostic and got the following error:
Msg: Error Code 2000-0142
Msg: Unit 1: Drive Self Test
Failed Status Byte=77
I called Dell and they basically confirmed what the diagnostic test said - my hard drive was fried.

Not wanting to give up on my hard drive that easily, I googled to see what I could do. I discovered I could try to repair windows using the Windows Recovery Console. So I found my Windows CD and booted from it and went to the Repair option. Once I had entered the Admin password, I got a command prompt and after a bit of searching around I discovered that most of my hard drive seemed to be intact. This, however, was still in the dos prompt of the Windows Recovery Console. I couldn't back up my files to a USB from there.

I tried chkdsk, chkdsk /p, and chkdsk /r - those all worked successfully. Yet my machine still wouldn't book. So I preused the list of commands available to the Windows Recovery Console (here) and tried a few more - bootcfg, fixboot, and fixmbr. In the end, it was fixmbr (which repairs the master boot record on the specified disk) that ended up working for me.

My laptop is 4 years old now. I removed the battery a couple of months ago, and with problems like this starting to happen, I going to start shopping for a new one. Hopefully this info helps someone out there though.

Business Idea - Automated Lawn Mowers

2011-06-12T18:34:00.000-07:00

Similar to my business idea regarding automated window washers, yet I think this one would potentially be even easier to implement. I was driving down our ring road here in Calgary and seeing all the grass growing very fast in the median and around the overpasses. I also saw a tractor/lawn mower left by the city over the weekend in one of the medians and it sparked a thought... Why couldn't lawn mowers be automated? It turns out somebody already has that idea. Check this link out: http://www.robomow.com/

These aren't industrial size/strength machines, though. I guess there are a couple of challenges with using an automated mower in an industrial/municipal context:
- Keeping them secure - they'd potentially be easy to steal. You could get around this by putting a GPS tracker on the unit something like this: http://www.gpspolice.com/
- More varied terrain - the mowers would have to content with steeper hills and soggy or muddy grass that they could get stuck in. The unit would have to be able to manage/mitigate these.
- Safety - The units would need to be well tested with emergency shut offs logic in a bunch of places to ensure the public's safety.

Two more business ideas

2011-05-26T22:31:00.000-07:00

I've got two more business ideas.

1. I've seen and heard of several companies here in Calgary that have tried to outsource their software development to India and have failed for one reason or another. I think it would be interesting to offer one's services as a liaison between a local client and an overseas outsourcing company to help with communication, expectation setting, and overall project management. It seems to me like there could be a big opportunity there - reasonable and talented software development, and clients looking for cheaper ways to build and implement their software. A boutique firm that acted as a 3rd party to help manage issues related to challenges with outsourcing projects like this overseas would be interesting to try.

2. On a totally different vein, (this is going to sound a bit silly, but...) it bothers me when I throw away a perfectly piece of good seran-wrap/cling-wrap. I doubt this would be a big money maker, but putting together a little utility/tool that stores the clean sheet of cling-wrap for re-use was my idea. :-)

Access Denied - Windows Service - StreamWriter- VB.Net

2011-05-22T22:24:00.000-07:00

I'm going to cut right to the chase on this one. I didn't realize that Windows obfuscates error messages for security. Without that gem of knowledge, I spent WAY more time than I should have last week trying to get a Windows Service to write a file to the local file system on a server. (I have another post related to obfuscated error messages with Windows here, but it focuses on IIS).

I was continually getting an access denied message trying to programatically create and write a file using a StreamWriter in VB.Net. My Windows Service was running as a particular service account, and I had given that particular account Modify access to the parent folder where the file was supposed to be created - but it would not write. I tried pushing to a different server to see if it would work there - still no dice. I tried adding security policies (run as a service, run as a batch job, run as part of the OS) to the service account running the service - again, the annoying message 'Access Denied'. For a while it was thought that maybe the file size was an issue - it was a little over a Mb. I deployed another version that wrote just one line. No luck.

In the end, I enlisted a colleague's help (thanks Dorina) and we discovered that the problem wasn't the user at all. It was in the path that I was trying to write to. I was using the Path.combine() method to put the path together which, in the debugger appeared to put the file name and path together correctly. However when we used Procmon to look at the output it complained on a createFile command that the 'file' was 'isDirectory'. Upon closer examination, it turned out that my Nunit test was calling one constructor that correctly used that Path.combine() method, and my Service called another constructor that did not use the Path.combine() method.

So in the end, my AccessDenied error, was really an incorrect path issue. Windows (apparently) doesn't display the correct error message so as not to advertise potentially sensitive paths. And I need to ensure that my Nunits exercise the same methods that my implementation classes use.

Hand-held application and smart board business ideas

2011-04-13T20:05:00.000-07:00

I had a couple of new technology ideas recently. In the last couple of weeks at work, all of our meeting rooms have been getting outfitted with Smart Boards - and quite a number of them with video conferencing features. While we were trying to get set up for a meeting we were impressed to see the camera for the video conference realign itself after our PM had pointed it a different direction. While he was continuing to get the display organized, I had the urge to help him from my seat using my fingers to zoom the image like it was an iPad or an iPhone. Then I thought 'why couldn't I do that, given the Kinect technology now with the xbox?' A touch-less touch screen? Maybe in the future.....

It wouldn't surprise me if there is an app already for the other idea I had. I was quickly going over a hard copy of a chart of geological formations looking for a particular name. I thought 'wouldn't it be great if I could just hit ctrl-F and find the word I'm looking for?' and then I realized that something like that would be a great app for an iPhone or similar hand held. Take a picture of the document, and write some logic to parse the text in the image and then search for the search term entered. The idea is similar to the 'Word Lens' app here, except that rather than translating text, your searching it.

IIS and setting NTAuthenticationProviders and authNTLM

2011-03-26T07:17:00.000-07:00

More than once now I've run into situations with the NTLM configuration in IIS where it seems that the GUI setting doesn't do all you need it to to properly configure integrated windows authentication for a web site. Earlier this week I was asked to look into a situation for a particular project team where Firefox wasn't prompting them to enter a username/password in one environment (they'd just get an http 401.2 error), and yet it was giving them the prompt in another environment. (IE was running fine in both environments, by the way).

I went through all the IIS GUI configs in IIS and the two boxes looked identical. Since this appeared to be an issue with integrated authentication, I ran this script:

cscript c:\inetpub\Adminscripts\adsutil.vbs GET W3SVC/WEB_SITE_Number/ROOT/NTAuthenticationProviders

which showed me that the working server had nothing set (wasn't set), while the server that was giving them problems returned with 'Negotiate,NTLM.' I subsequently set NTAuthenticationProviders to '' and it worked by using 'set' instead of 'get' in the above script and providing an empty parameter.

Note: I understand the above setting to be different than the one below. In more than one instance at work I've had this issue and proven that this is the case. The script below, from my understanding, the same as the Windows Integrated Authentication checkbox on the Directory Security tab in IIS.

cscript c:\inetpub\adminscripts\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/authNTLM TRUE

IT Professional - Know Thyself

2011-03-03T21:31:00.000-08:00

What are you good at? It pays to know - literally. It will also make you happier and more productive on the job.

The more I work in IT, the more I see where I'm most effective. I also see in comparison where others blow past me in their individual skills. It's important for multifunctional teams to have a good idea of different team member's areas of expertise. Leveraging (knowing when and who to ask for a specific kind of help) these team skills can greatly improve the effectiveness of a team.

Here's some real life examples:

I'm rather weak on writing complex sql statements. I can profile sql, and can get most sql work done. But if I've got a difficult query I'm working on, I'll go ask my team mate who is an expert on putting together complex sql. Later on, he'll ask me for help with a visual studio or server/service configuration related question. Its a good, productive working relationship that keeps everyone happy. The BA's are happy because the data looks right. My PM is happy because my task was finished on time. And my team mate and I our happy because we both learned something and did it faster than trying to google for it on our own.

I'm great at problem solving software related issues - particularly in a domain I'm familiar with. Whether it's missing permissions, configuration, network connectivity, everyone sitting around me will tell you I'm very, VERY persistent at solving problems that are brought to me. My neighbor one cube west is fantastic at IIS/MSTDC/COM related bug/configuration debugging. She also knows our software domain better than almost anyone else in the building - she seems to have it memorized. When I have an issue with our software that I don't know how to fix, or a business related problem in the software, I talk to her without hesitation. Conversely, when she is stuck on an issue that she doesn't know how to solve, she will buzz it past me. Again, a productive working relationship that makes everybody happy. The business is happy they got their patch as fast as they did. Once again our PM is happy, etc....

In meetings I will keep quiet and not say very much unless I feel there is an important point being missed. I'm not super great at arguing a point. I'm not the person that's always called on to go to a meeting to convince another group of people that 'our way is the right way'. Part of this is my personality, part of it is my belief that there is multiple ways to get 'a good enough' answer for the problem we are looking to resolve. However, we have another fellow in our group that will absolutely blow you away with his gift of logical persuasion. He's the guy that gets sent to those 'convince the group' meetings. On the other hand, I'm called on when diplomacy and a consensus is required.

Know what you are good at. Seek to know what your team mate's strengths are. Become a happier, more productive development team.

Excessive logging kills AppPool (log4net)

2011-02-14T08:14:00.000-08:00

We had an issue where one of our Application Pools (IIS 6) was 'dying'. The interesting thing about it was while the application itself was dead (error page, sometimes showing an outOfMemory exception, other times showing a meaningless exception), the appPool itself appeared in the GUI to be running. The more troubleshooting I do with Windows OS's, the more I do not trust what the GUI says. Appearances in Windows can be deceiving. For several weeks - until we understood the root cause of the issue - are resolution to the dead application was to restart the 'running' appPool.

A side note - this appPool was running on a Network Load-balanced server, on the dominant node. The appPool for this particular application only ever failed on the dominant node. It never failed on the other node of the cluster, so we had reason to believe that the 'load' had something to do with the root issue.

In the end, we discovered the root cause of our dying application by accident. We were cleaning up our files on the prod servers when it became apparent to us that log4net on another application (and another appPool) was set to log 'ALL'. This resulted in 1.5Mb of log created about every 5 minutes. I had a hunch that all this logging was having a memory impact on the server - we certainly didn't want to be logging at the 'ALL' level anyway, so we created a change request and fixed that issue ASAP. Then we crossed our fingers, hoping that our dying AppPool issue would go away as well.

It's been more than two weeks since we implemented this log4net change and our flaky appPool hasn't died once. Lessons learned:

- Watch for excessive logging on prod servers
- Excessive logging can apparently cause other appPools to through outOfMemory Exceptions!

Windows VM False Performance Monitor Stats

2011-02-09T22:20:00.000-08:00

We almost exclusively run VM's now in most of our environments. Recently, we added two new Windows VM's in production for a particular solution. Thinking we were being prudent, we asked to have the boxes provisioned with 4 Gigs of Memory since that's what the old 'hardware' boxes had. We deployed our solutions to the new boxes and tested them without a hitch. So a week later a switch was flipped and the new VM's went live while the old boxes no longer took requests. After a few days, we discovered (by looking at the task manager) that we were getting some conflicting and concerning readings for memory utilization. Check this out:

- the task manager graph said we were using 3.7 gigs of memory
- looking at the actual numbers below the graph, we only had 300Mb of memory available on the machine
- looking in the processes tab, none of the processes were using a significant amount of memory (several were using between 200-300MB, but that was within normal tolerances when compared with servers running the same system in other environments. However these same processes where paging memory like they shouldn't have been! Page faults were very high.
- Totaling the memory used in the individual processes, we were NOT using 3.7 Gigs of memory.

After doing some more investigation, we asked the team that built the server how the memory had been provisioned for these new boxes. It turned out that they both had originally been given 1Gig of memory and then later (after they were running) 3 more Gigs were added. At this point our hunch was that these boxes weren't seeing the additional 3Gigs that had been subsequently added. After doing a hard reboot on both boxes, this turned out to be the case.

Lesson learned: make sure you follow VM procedure (hard re-boots) for adding/removing resources to VM servers. Short-cuts can create some pretty confusing statistics in the OS.

Debugging/Troubleshooting SQL performance/blocking issues

2011-01-28T11:24:00.000-08:00

I picked up a good tip from one of our DBA's yesterday. He uses sp_who2 to see which users have active threads/processes running against a particular database. This stored proc returns the PIDs as well (Process ID number). I had known about this previously. What was new to me was he also used:

dbcc inputbuffer(pidNumberHere)

to get the database to tell him which stored proc was being executed by that PID. That seems to me like a handy trick. It certainly solved the problem I was having.

WCF and Large Result sets

2011-01-27T07:12:00.000-08:00

Dealing with oil resource data across several disparate systems and also trying to make our API's more generic and consistent, we use WCF a lot (web services with windows apps). We've had a couple of instances where we've had to tweak the config settings for either our publishers or our clients to return some large datasets over WCF.
For the most part, I have found the error messages relating to constrained WCF config settings refreshingly to the point. They pretty much pointed me directly to the attribute in the file that I needed to change in order to fix my issue.
For memory related errors on the client, I found that juggling with the maxBufferPoolSize and maxReceivedMessageSize in my binding to help. I moved the maxBufferPoolSize from 9999999 to 99999999, and changed the maxReceivedMessageSize to 350000000. Sometimes after that I would get a timeout error on the client machine (depending on what kind of load the server was under). So I'd have to make the timeout settings on the binding higher.
Obviously it helps when the publisher has the same values for the same attributes on its binding.

SMTP email issue with hosts file

2011-01-18T17:52:00.000-08:00

I had an interesting issue last week. I had a website with a contact page that uses SMTP sendmail to email a list of people whatever was entered on a form. The curious thing about this issue was it would send emails to everyone in the list except email addresses that matched the domain name of the web site. For example, it would send an email to perry.mckenzie@blah.com, but it wouldn't send an email to perry.mckenzie@continuouslylearning.blogspot.com, even though the server that was doing the sending was listed in the DNS as 'continuouslylearning.blogspot.com'.

I tried changing the email provider by pointing the Email Host to a server that I knew accepted anonymous email pushes, but it did the opposite - sending the emails successfully for users from blah@continuouslylearning.blogspot.com, but not sending them for blah@anyOtherAddress.com!

After looking through various logs (including /var/log/maillog), it seemed like sendmail was getting confused as the log was trying to associate the email address (someone@continuouslylearning.blogspot.com) with a user on the host and not finding the user.

sendmail failed error: localhost: someone@continuouslylearning.blogspot.com user not found

I tried removing an entry in the /etc/hosts file that pointed the IP address to continuouslylearning.blogspot.com, but that didn't seem to work. What I didn't realize until later (with the help of my hosting provider) is that this was the correct change to make, I just needed to restart two services after I made the change:
service network restart
service sendmail restart
Then my email worked perfectly.

Books I want to read soon...

2010-11-29T21:29:00.000-08:00

Here's two books I want to read soon:

The Nomadic Developer: Surviving and Thriving in the World of Technology Consulting

And

Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler))

Thoughts on Design

2010-11-25T12:04:00.000-08:00

I recently re-read The Last Lecture by Randy Pausch and was impressed with what he did with his classes that focused on User Interface Design. The first day of class he would bring in an old VCR, put it on a table in front of class, and without saying a word he'd pull a sledgehammer from behind his desk and destroy it. Needless to say it got the class's attention. The message Randy was trying to convey in this demonstration is that poor end user design is detrimental to the product. End User Design is critical.

There is another person who is a strong proponent of excellence in End User Design. He is still alive today and he's be a driving force in perfection in Design in the IT industry. He designs have been flying off of store shelves for the last 25 years or so. If you haven't guessed yet, I'm thinking of Steve Jobs from Apple. I ran across a very interesting article about him in Bloomsberg Businessweek. It's basically an interview with John Sculley, the CEO of Apple from the mid-80's to the mid-90's, and what it was like 'Being Steve's Boss'. He talks a lot of Steve's philosophy and background in design, and Steve's influence on Apple in this area. It's a great article.

It got me thinking... In software I have not been super stringent about software design on the applications I've written. Let me clarify - object oriented design is a must. The train of thought I'm seeking to follow here is 'is it a requirement to implement design patterns in your code everywhere possible?' A major reason for this is I've worked with a number of people who think their job is refactoring and renaming classes and packages in a code base - to the determent of the project. I often tell the students in my classes that there are numerous ways to 'skin a cat' as it were, when coming to a working solution in code. I wonder though, is there always one perfect way? Is this something to strive for? Does it matter as long as the application works the way the client wants it to? What do you think?

Personally, I think that the answers to these questions depends on a lot of things, not the least of which are your client, the project, and your team. Elegant solutions look and feel so good when your done - both for you and for you client. Perhaps the motivation and context for the project need to be analyzed before a decision is made whether or not you should 'shoot for the best designed, most elegant solution' possible? Obviously if your project has constraints like budget, resources, and time, there's a limit to how much effort you can put into evolving your project into the best design in can be. That is one big reason why design patterns and frameworks are so prevalent (and pretty much a requirement) in software engineering.

I like to read/sign out Fast Company from the Library - I'm actually thinking about getting a subscription. Anyway, they have an annual 'Masters of Design' issue that is a lot of fun to read. In it they highlight different world class designs and their designers. Here's a link to their MOD tag cloud.

Windows OS Performance Guidelines

2010-11-21T09:12:00.000-08:00

These are notes from a workshop I was able to be a part of a couple of weeks ago. The workshop was on Windows Performance Monitor and monitoring vital signs on Windows Servers. The windows performance monitor has a multitude of counters that a person could potentially monitor on a server. I'm just going to point out a few critical ones that were highlighted in the workshop and what their tolerances are.

Some points to note before I get into these values:

Don't always look at the graph first. The counter graphs can be scaled right out to lunch so they aren't necessarily a good first glance indicator. For a particular counter, your focus should first be on the Minimum and Maximum values in perfmon just below the graph.
In the past, I would tend to keep an eye out for sympathetic counter relationships on the graphs. However, keeping an eye out of inverse counter relationships is also a good idea (where one counter could be decreasing in value while another is increasing)
Counters can get corrupted. Apparently this happens more often than one would think. They can be rebuilt - directions are in this KB post.
You can attach a PID (process ID) to a perfmon counter in an OS older than Windows Server 2008 by modifying reg keys. Details are in this kb article. Versions of perfmon on 64bit OS's come with this already set up.

On to the objects/counters of note:

Process Object

Handle Count - greater than 500 handles may point to a problem.
Private Bytes - greater than 250MB could be a problem. I've seens procs over a Gig and they definitely were a problem (they can get that high).
Working set - greater than 250MB could be a problem
Thread Count - Greater than 500 threads needs to be watched to ensure they aren't increasing over time

Processor Object

Processor Time - all core instances. _Total can get you an overall trend. Greater than 91% utilization is potentially an issue.

Network Interface Object - you need to know what the spec is for your network interface to determine it's capacity. Anything over 80% of capacity could point to a problem

Current Bandwidth - help you determine the NIC's capacity
Output Queue Length - greater than 2 is an problem
Bytes Total - greater then 65% of capacity utilized is past the warning threashold (blinking read with siren)

Memory Object

Free System Page Table Entries - the higher the better here. Lower than 5000 is considered critical. I've seen boxes 'run' (aka hobble) around 2500.
Available Megabytes - again higher here is desirable. Less than 100 MB or 5% free is very problematic
Pool Non Paged Bytes - greater than 80% consumed is out of spec (not at all good)
Pool Paged Bytes - Same as non-pages bytes. Anything between 60-80% should be watched.

Logical and Physical disk Objects - they have the same critical counters so I've put them together here

Idle percentage - 19 to 0 percent is critical. Anything over 55 is warning.
Current or Avg Disk Queue Length - 3 to 31 in the queue - you need to keep an eye on it. greater than 32 is an issue
%Avg Disk Sec Read or Write - 25ms and above is critical

Other interesting points of note:
- Mark Russinovich was the original developer of Perfmon. His blog is apparently pretty good and our facilitator was very impressed with him.
- Windows Server 2008 does processor/core parking. This means the server will 'retire' (effectively turn off to save power) CPU's when there isn't a heavy load on the box. Our facilitator told us that one ISP moved all their boxes to Windows Server 2008 for this reason and their power bills were 15% less a month.

Newbie Advice

2010-11-19T07:06:00.000-08:00

Having gone through an IT technical school, I realize they push a lot of information and technology at you. There's a lot to assimilate in less than a year (depending on the school). On this page I'd like to point out some technologies that perhaps weren't covered much in classes that are widely used by good shops in the industry, and also offer some career advice for the future. Before I delve deeper, let me say that this information is based on my experience, which is mostly on the Java technology side of the fence.

Technical Heads Up

1. Design Patterns. One thing many of the technical schools don't have time to cover in their curriculum is code design patterns. It doesn't matter if you doing .NET development or Java development, design patterns can really make a difference in organizing, maintaining, and the reusability of your code. A great book to pick up for an introduction on design patterns is published by O'Reilly and called Head First Design Patterns. If you don't have access to the book, a good place to start is to google some popular patterns that are being used in code now. Dependancy Injection is a pattern that is really big right now. Two frameworks that use it a lot are Spring and Google's Guice. Other pattern I've seen used frequently are the decorator pattern, the abstract factory pattern, factory pattern, the singleton pattern, and the proxy pattern.

2. Using a Repository. Using a repository, or a version control system, should be required by every development shop. Frankly, if you're in a company where they aren't using one, either push to get one installed, or start looking for another place to code. A repository is basically an application that functions as the premier source/bank/datahub/repository for all code. This application should be on a server with lots of disk space and accessable to all developers in the company. Repositories allow developers to manage code changes at many different levels allowing many developers to work on the same code base at the same time. Some repositories I've used in the past are Subversion, CVS, StarTeam, and Perforce. Googling any of these will give you more information.

3. Unit Testing. Unit testing is the development practice of coding 'tests' that validate individual classes or 'units' of code, to ensure they are functioning as required. This practice has developed over the past 4-5 years or so and has been found to be very effective in giving developers the confidence to refactor their code and ensure that it's functionality stays consistent over time. Some unit testing technologies are JUnit, JWebUnit, HttpUnit, DBUnit, and NUnit .

4. Continuous Integration. Continuous integration is a process that automatically rebuilds and tests (with unit tests) an application or code base. Continuous integration employs an application like CruiseControl or Anthill Pro which is configured to automatically check out the code base from the repository, compile it, run it's unit tests, and sometime even deploy it to a test environment. It allows developers to check for integration problems in the code, and warns developers of broken code, allowing and reminding them to fix it ASAP.

Soft Skills and Career Advice

1. Change is GOOD. Once you've gotten your feet wet in the industry, being a Java developer in one company can mean something quite different that being a Java developer in another company. There is such a vast variety of Java technology that two Java developers can have quite different experiences in developing applications. Whole development processes and methodologies can be very different in differenct companies. One company could be flying by the seat of their pants, developing with not backups or repository, and manually pushing out code, while another is totally immersed in Agile methodology and outsourcing everything but development.... etc. Keeping this in mind, my advice is to not get comfortable in any one company but move around a bit - that is, after 18 months or two years at one place, get a different job. You'll be put into a position where you'll either learn a lot of new stuff, or you'll be mentoring. Either way, it'll be an experience that will likely teach you more then if you had stayed put. It'll also give you some motivation to brush up on your resume writing, interview skills, and expand you network.

2. Walk & Talk While software developers, for the most part, end up working in cube farms, it should not mean that they need to stay in their cubby hole all day and do work. This may be just my style, but I find when I'm working on a big integration project or on a project with a lot of departments or teams involved, getting up and talking to people really helps things move along. Whether it's getting socializing a new means of managing your server configuration in the code base, or talking to a DBA about a script that you'll need to release with you new code, talking face to face is far more effective than emails, I've found. I generally use emails if I can't find the person, or I want to make sure there is hard documented proof about a decision that is being made, or if I want to include a bigger audience in my communication. Successful software development is more about relationships that a lot a people realize. Relationships are more apt to be cultivated face to face... plus it's good exercise.

Career Paths in IT

2010-11-17T23:10:00.000-08:00

I've been teaching at SAIT again this week. Generally before every class gets going in the morning I like to do a little 'blurb' on concepts or thoughts that aren't covered in the course the students get. One of the things I find is that the vast majority of students (and the population at large for that matter) don't realize how many different career paths there are in just the software side of IT. Guaranteed there are some students in the course that I teach who, by the time I get the chance to teach them, are very concerned they made the wrong decision by getting into software programming. They don't feel like they are cut out for programming. I try to put their fears at rest by suggesting that there are other opportunities for them - careers that can utilize the coding experience that they are getting, but allow them to use talents that they are more comfortable with. Here's a list (by no means exhaustive) of some other opportunities in the software field.

- Project Management. Project managers who have coding experience or at least a technical background are a step ahead of PM's who aren't techies. I myself have considered taking some project management courses or getting my PMP, but thought the better of it after seriously thinking about what I like to do at work. PMs need to have a great soft-skill set. While I don't think I'd have a problem with that, I would missing troubleshooting/solving problems. They also have to be politically savvy and have patience for meetings.

- Business Analyst. BA's also need to have a great soft-skill set, as they need to be talking with everyone (the business, the developers, and the PM) and writing down the requirements. It also helps if they are technically competent, and they enjoy documenting/writing. For some reason in Calgary right now, there have been a lot of postings for BAs over the past 6 months. I just found out today that a variety of institutions offer certificates/diplomas for a Business Analyst career path. (NAIT and Mount Royal University are two local options)

- Quality Assurance. This field has been growing a lot over the past 8 - 10 years, and frankly I don't think projects that I've been on over the past 5 years would have been successful without our QA teams. QA experts need to have a penchant for detail and process. They need to be able to understand the business and relevant business rules well. Most of the QA people I've worked with haven't had a certification, but you apparently can get more info on a software QA certification here.

- Release Management. This is what I (like to think I) specialize in. We ensure that teams are using a continuous integration server and QA has the builds they need for testing where and when then need them. We have two main goals: 1. Keep downtime in production to a minimum and 2. Keep lost man days in QA to a minimum. I like release engineering because I get to see the whole picture - I have to understand the code and it's dependencies, I need to understand the infrastructure, and I'm responsible for the product getting through all the environments to production. For me (being a bit proactive), this has also lead to automating deployments of database objects and datascripts, server configurations, and code, as well as installing monitoring tools to ensure that every environment is ready to 'rock'.

- Security Analyst. These dudes make sure the good people have access to everything they need access to, and the bad people are locked out. They spend a lot of time combing though logs and configuring IPSec rules. They also perform software audits on code, all kinds of different tests on web sites, and network audits. I wouldn't mind moving my career this direction in the future as they also need to have a pretty good idea of the whole system/infrastructure/architecture. Great web sites for more resources on these types of positions are here and here.

Even more specialized positions:
- GIS (Geographic Information System) developer or architect or analyst. Working with digital 2d and 3d mapping systems. Having some limited experience with this at the ERCB (deploying mapping systems that display reservoirs,etc in Alberta) I think this is pretty cool stuff.
- Search Engineer (Search Engine Optimization specialist) I've dabbled in this off and on for my clients. You definitely need to be on top of your game and know how the search algorithms are changing.
- Business Process Specialist - there's lots of software to specialize in here. Tibco, Agile Point, SAP, JDEdwards, etc.
- Document Management Specialist - Sharepoint, Edocs, Documentum, etc. This involves the automated management of documents - not the writing of them, but their storage. Government bodies and law firms definitely have a need for these kinds of systems.

Another emergency software fix

2010-11-16T19:36:00.000-08:00

A couple of weeks ago I had another client take advantage of my Emergency Software Troubleshooting services. They had a web site maintained by a popular CMS system. Unfortunately, their scheduled backups weren't as scheduled or robust as they thought and they were left without a website (or backend database) after their hard disk died. I was called to help.

After some invesigation, we surmised we were left with:
- a bunch of .myd, .myi file backups for MySql (they were a couple of months old)
- another .sql file backup of the database 12 months old
- a file system backup of the site a couple of months old
- and another file system backup around 12 months old.

We rebuilt the database using a combination of the most recent myd and myi files. This was a bit of a trick as the .frm files were missing in some cases. What we did in those cases was create a dummy .frm file, rebuild the tables from the myd/myi files and when we did that we had mysql replace the frm file (it's an option you can enable during this operation). We then used the data from the older database to populate the tables that didn't have the frm files.

With the file systems, it was a matter of patching together what worked between the two file system backups with the data we had in the database. Since the old system was pretty much blown away, it was decided to upgrade the OS and the web server at the same time. This meant reconfiguring the web server and ensuring the majority of the site worked. In the end, the site was pretty much ready to go after 8.5 hours of work.

Licenses.Licx error with DevExpress in Visual Studio 2010

2010-11-14T13:40:00.000-08:00

While refactoring a business solution at work to deploy using Visual Build rather than InstalShield, I ran into an issue with our DevExpress web components not seeing the correct licenses in the licenses.licx file. After googling for a bit and discovering how the licenses.licx files are created and manged in VS2010, I started troubleshooting to see if I could resolve my issue - seeing the red text on every web page telling me that I'm using a 'trial' edition of the software when I clearly wasn't.

I deleted the file to get VS to recreate it. That didn't seem to work for me. I renamed it. I checked permissions on it. I investigated how it got deployed and whether I needed it on the server or not (if I remember correctly, you don't have to explicitly move it to the server). In the end, my solution sort of knocked my socks off. I deleted all of the content in the file and left it blank. I then built my msi and deployed. Imagine my surprise when the blank licenses.licx file got rid of my trial license message! I don't know if this is a bug with VS2010 or it's integration with DevExpress 8.2.2, but my fix seemed like a total hack. However we've tested this in multiple environments and it's solid.

The Phoenix and Projectors

2010-11-04T21:29:00.000-07:00

Two business ideas in this post. Actually one is a business model, the other is an idea.

The Phoenix
The business model that I ran across I'd like to call 'The Phoenix'. Basically the idea is to take a dead business or business idea, and resurrect it. The tale of the Twin Otter, a Canadian built Short Takeoff and Landing Utility plane is a classic example of this business model. Details of it's life, death, and resurrection can be read on Wikipedia here.

In its first life, production ended on the Twin Otter in 1988. The aircraft's versatility made it a bush pilot favorite and demand for the planes have increased. Viking Air of Victoria, BC, picked up on this and bought the rights to build the plane again. After nearly 20 years of being out of production, new Twin Otters are now being built in Calgary and the company has over 50 orders from all over the world.

Nov 14 update - I just ran into an article in The Globe and Mail about another brand that did something similar. In fact, some work colleagues and I had been musing about this turn-around in the summer as we found some Pop-Shoppe bottles in a nearby convenience store. Here's the link to the article in the Globe and Mail.

Projectors
The second idea isn't nearly as sexy. Watching my project manager try juggle between a spreadsheet of software development stories and another of schedules and tasks, I got to thinking why can't there be a single projector with the ability to support and display like two monitors? Or even right a button into the software/firmware of the projector so it can flip between screen instances. For that matter, why hasn't windows come up with the ability to manage screen instances like Linux has? Maybe that functionality is there and I just haven't been paying attention...

The importance of soft skills in IT

2010-10-15T11:38:00.000-07:00

Soft skills are crucial to being a successful developer in the IT industry. Some of the soft skills I've found beneficial in my career are:

Being a good communicator. It's not just about talking, but listening as well. Dirareah of the mouth is never cool, especially when a job needs to get done. Strive for mutual understanding in a conversation. Communicating in the IT world isn't just limited to talking and listening though.

Having the discernment to know when to write an email versus actually talking to a person is important as well. I do not recommend doing all your communication through email. Communication in emails can easily be mis-understood or mis-construed and take on an entirely different meaning and tone than you originally intended. Emails are great for documenting decisions.
Make sure that the content of your email deals with the subject you've put in your subject line. For example, having a subject line of 'Quote for project XYZ' but then talking about quotes for that project AND other projects in that email is bad news. Send you quotes for the other projects in a different email with a more relevent subject line (I know this from experience).

Developing relationships. Extending communication a bit further and actually developing relationship with people in the office is huge as well. Questions like "How was your vacation?", "Where are you going on vacation?", or "What did you do over the weekend?" are great for this. When you've got a relationship in place, there's trust. Many times I've been granted exceptional privileges and become more productive in my work because I've had a relationship with a key person who trusts me. These relationships can also help in the future as well when you are looking for that next gig.
Having patience. Whether it's letting somebody else go to the front of the 'line' for access to production, or having forbearance when dealing with a difficult client, patience can go a long way to strengthening that 'trust' in your relationships. Sometimes work (or working with clients) in IT can get frustrating. Things I do to help me step back and get some perspective are:

go for a walk
find something else to do for a while
talk to someone - use them as a sounding board
take a day off or go on vacation

Positive Attitude. I like to keep my workplace fun, interesting, and upbeat. To promote this around me I'll sometimes:

Bring baking from home to share
Audibly rejoice in small victories (getting a solution migrated and compiling from VS2005 to VS2010 for example)
Write thank-you notes (either hard or soft copy) to others who help me
Encourage people who are down. We had a large project I was helping with and the lead was concerned that between the time and the technical challenges he had, he wasn't going to be able to finish on time. I believe he could and I told him so (and backed it up with my help). In the end, he did finish.

Being Flexible. Generally, you are always working for a client. Invariably, they will change their mind or want an enhancement. Microsoft will upgrade their software, the US president will change daylight savings time one more time, the CTO will be convinced that Tibco will solve all the business problems again. I've found that being able to take these changes in stride is crucial to maintaining my mental health (avoid being overcome with frustration). It's all part of the territory - and frankly, job security.

Business Idea - Jam Joint or Practice Pad

2010-10-14T21:38:00.000-07:00

I was getting into some music on the commute home the other day, lamenting the fact that I don't have a drum set to practice on anymore when I was struck with this business idea. Rent a smaller stand-alone building and renovate it to have reasonably sound-proof practice rooms. Furnish smaller rooms with drum kits, leave some empty (for electric guitar/bagpipe/tuba practice), and make some bigger for bands to jam in.

Many drummers, musicians in general have no where to practice in cities because it disturbs the peace. Renting these practice/jam rooms out might be an interesting and franchise-able business opportunity. My wife could only listen to me play the piano at home for so long (and I don't think I'm that bad - 11 years Royal Conservatory). As soon as I started practicing technique, her patience got thin. :-)

Log4Net performance issue with WCF

2010-10-13T12:25:00.000-07:00

We ran into an interesting production problem recently that had an interesting resolution. After upgrading our application server to use WCF calls, we found that performance was lagging in production. We were able to replicate this problem in our development environments and spent a good deal of time trying to resolve the issue. Avenues we explored included profiling calls to the databases (using Sql Server Profiler and filtering on database name and login Id), mining activity logs in the database, pouring over configuration files, utilizing Wireshark, SysInternals Process Explorer , SysInternals ProcessMonitor (and filtering by processName and PID), and the Windows PerfMon (performance Monitor - viewing the defaults and various performance objects).

Interesting things I learned/discovered were:
- IE browsers (IE 7 and older) default to only two connections to the web server See this link
- It seems IE processes javascript before rendering a page while Firefox process javascript after the page is rendered in some cases
- The default for WCF connections to the database is serializable rather the read committed!
- WCF connections default to some low threasholds for ConcurrentCalls, ConcurrentSessions, and ConcurrentInstances. These can be changed using the serviceThrottling node and attributes of a service behavior you create. See this link for more detail
- In dos, you can filter a netstat to only show specific ports and statuses like this:
netstat /n | find "ESTABLISHED" | find "8085"
- Use /n with netstat to not resolve dns names - without /n, dns name resolution can make netstat take a while.
- In dos, you can use the 'tasklist' command to show something akin to a process list in the Task Manager
- Retrieve IIS process names from the command line by executing this command:
cscript c:\WINDOWS\system32\iisapp.vbs

In the end, the resolution to our problem was to turn off a custom implementation of Log4Net that had running. In the process monitor we saw that log4net was re-initializing, re-reading it's configs, and re-creating all the log files for EVERY page request. We discovered this because it seemed that there was a several second lag time between when the request was executed on the browser and when the web server actually acted on it. Disabling log4net entirely improved our performance by a significant amount. (10's of seconds)

Update - Nov 26, 2010. One of the reasons we were running into the issue above is the log4net configuration loader was placing a lock (being very pessimistic) on the configuration file every time it was loading. Agreed, this solution should likely have been using a static logger - one instance for the application. However, given the way it was implemented, I'm thinking that we could have avoided a bunch of hurt if we had made the configuration loader a little more optimistic and used:
<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />

Ms Sql Server Grant Execute against sub-set of stored procs

2010-09-22T11:20:00.000-07:00

Thanks to Karlson for this -
Here's a script that can grant execute (or whatever you wish to grant) to a set of stored procs based on a prefix in the stored proc name:

Set nocount on
Declare @proc_name as nvarchar(max)

DECLARE procs CURSOR
     FOR Select [name] FROM    sysobjects WHERE [name] like '[sp prefix here]%' AND xtype = 'P'

OPEN procs

FETCH NEXT FROM procs
INTO @proc_name

WHILE @@FETCH_STATUS = 0
BEGIN

    exec('Grant Execute ON ' + @proc_name + ' TO [roleName here]')

    FETCH NEXT FROM procs
    INTO @proc_name
END
CLOSE procs;
DEALLOCATE procs;
Set nocount off

Another business idea

2010-09-20T20:01:00.000-07:00

One of my clients just moved into a brand new office tower. Everyone was all pumped about the new digs - lights that have movement sensors that turn off automatically when no one is in the room and turn on when someone comes in, new desks and chairs, new faster elevators,etc. However, upon using the 'facilities' for the first time, I couldn't help but notice a common problem I see in many public bathrooms around the city. Paper towel fills up the garbage cans too fast. Nobody wants to compact it with their hands. Inevitably, it overflows before the janitors make their rounds.
My business idea this go round is to import/sell garbage can compactors to these public buildings in the city that have this problem. Other places where I've seen this issue (besides busy office tower washrooms) are airports and universities.

Open Source Bug/Issue Tracking System

2010-09-09T10:55:00.000-07:00

In the last couple of weeks I've been helping facilitate the development process (through build & deployment automation, etc) on a new project, and I was asked to spin up a new instance of Trac for the project team. It didn't take me long to get it going. I was amazed at the rate of adoption of this tool by the team as a whole. The business was very positive about the tool and had no issues with learning it and beginning to log tickets. I will definitely keep this tool in mind in the future as it is relatively easy to install, straight forward to use, integrates with Subversion and Active Directory, and fulfills all the basic requirements for a bug/issue tracking system that my project team was looking for.

Common Obfuscated Error Messages with IIS

2010-08-31T07:23:00.000-07:00

I've noticed that IIS tends to obfuscate error messages - displaying messages that are different than the actual issue that is happening to the application. This may very likely be because we are running a plethora of web sites with different architectures and history on one instance of IIS. I thought I document some of those 'tricky' error messages here just in case someone else runs into them (or I forget what they really mean).

1. 'An existing connection was forcibly closed by the remote host' I ran into this error numerous times trying to configure a WCF web service to bring back a large result set (60k + rows). Playing with the values for maxBuffer PoolSize and MaxReceivedMessageSize in the web.config, I came to discover that this error was really seemed to be masking was an out of memory error that forced the shutdown of the AppPool that was serving the web service.

2. 'BinaryFormatter Version Incompatibility errror' Generally I've seen this error when you have IIS site on a web server trying to communicate with another IIS site on an application server, and there is no response from the IIS site on the application server - either that web site isn't running, or your url pointing to the web site on the application server is wrong.

Some good links (to blog articles) on Configuration Management

2010-08-23T06:59:00.000-07:00

Here's some good links to blog articles on Configuration Management. The first is about the importance of a CM person in a company. I would tend to agree with one of the comments that talks about the size of the company making a difference. Obviously a smaller startup doesn't need a dedicated CM person. However, larger enterprises like the one I'm at now with numerous development environments, multiple servers, a GIS infrastructure, a document management infrastructure, and a beomoth of a mainframe legacy system needs a dedicated CM team. Here's the link:
http://evgeny-goldin.com/blog/2010/08/21/configuration-management-position/

The other link considers adding database automated deployment to you CM system. I would highly recommend when I contrast my experience over the last two years with my previous release management experience. Previously we automatically deployed database schema and data changes. In my more recent experience the DBA group has flatly declined any encouragement and offer to do this. The difference in the amount of time to do database refreshes and debug issues with them is stark - like hours vs. days. Here's the link:

http://www.simple-talk.com/sql/t-sql-programming/when-database-source-control-goes-bad/

Testing with IE 6 (on Linux!)

2010-08-21T10:23:00.000-07:00

I have a requirement from a client to ensure that their website looks good in IE6. None of the machines I normally work on have IE6 installed. So I googled IE6 emulators as I had used one in the past that was pretty good, and came up with:
http://tredosoft.com/Multiple_IE
and
http://www.browsercam.com/
However you have to sign up to get a free time trial and I thought maybe there was something better for me out there. Some of the forums talked about having an old laptop that they keep around just for testing with IE6. I do have an old desktop running XP, so I checked it. It turned out I have been keeping up with all the updates for the most part and one of the service packs (I don't remember if it was 2 or 3) installed IE7. Bummer.
Googling a bit more, I discovered that you can install IE6 on Linux. Pushing against this thread a bit more, I found this site:
http://www.tatanka.com.br/ies4linux/page/Main_Page
and went to this page:
http://www.tatanka.com.br/ies4linux/page/Installation:Fedora
followed the directions and IE6 was installed and running, properly displaying my client's website within 10 minutes. The Tatanka gives you the option to install IE5.5 and IE5 as well, but when I tried them, the installations were corrupt. Since IE6 was all I really needed, I was happy with leaving things the way they stood.

Active Directory SPOF

2010-08-20T10:04:00.000-07:00

We had a great example of an AD SPOF (single point of failure) recently. A vbscript had been written by someone with no malicious intent to (for some reason) hook into AD and check service accounts. What this script ended up doing though was locking out said service accounts. Within the span of 30 minutes we were well on our way to locking out 400+ service accounts in the enterprise, irrespective of environment. This took production out of commission.
Seems like a bit of a hole to me.

Grep features

2010-07-22T07:32:00.000-07:00

I was asked to search several teams' code-bases recently based on a list of over 270 different search strings. We are a Windows OS organization and doing a search like that would take some time. By the way, has anyone noticed that the Windows explorer search tool is flaky? Even if I used a tool like this it would still take a while.
I have cygwin installed, and I know grep works well - when I search for something I know is in the file, I find it (unlike Windows sometimes). What I didn't know was that there is a -f option on grep that allows you to provide a file with all you search terms separated by newlines. An astute worker made me aware of this and I tried it out this morning. It worked nicely and saved me some time. Here's how I implemented it:

grep -r -i -f file_of_search_strings.txt * | grep -v .svn | grep -v Assert | grep -v .sql >> resultFile.txt

-r is a recursive search based on where I'm current sitting on the command line
-i is a case insensitive search
-f allows me to pass in my file of search strings
* tells grep to search every folder/file at this level. Used with -r it tells grep to search every single folder and file that is currently at my current cursor position, or a child of the folder I've cd'ed to.
The | grep -v .svn tells grep to ignore/not display files with *.svn in the results
and then >> allows me to pipe my results to a file for inspection.

You may have to run the dos2unix command on the text file with your list of search items to get grep to see the new lines in the file correctly.

MS Sql Server Stored Proc Search Query

2010-07-21T12:31:00.000-07:00

If you work in an org where there's a lot of business logic put into (MS Sql Server) stored procedures or you have an application that is stored proc intensive, you might find this code snipped helpful. It searches all the stored procs in the database for a specific string

select distinct o.name from syscomments c
inner join sysobjects o
on c.id = o.id
where o.type = 'p'
and c.text like '%string to search for%'
order by o.name

.Net Self Serve Automated deploy

2010-07-12T11:56:00.000-07:00

There are a couple of prerequisites that need to be in place in order for a self serve automated deployment system to work.
1. You should have a Nant, MsBuild, Visual Build, or a similar type of scripting file that is doing deploys for you already - see my previous posts on doing that here
2. Your folder structure for every project you want to deploy should be similar. To have a good Self-Serve Deploy tool, it needs to work on the assumption that directory structures are consistent for certain types of projects. This could be set up using your repository tags directory, or automatically copying deployment files to the required folder structure after the build is successful. A folder structure I've used in the past is b:\deployment\projectname\version\ My build file(s). Any other required folders and files reside in the version folder with a consistent folder structure.
3. You need either a properties file or a DB table to manage which tag/version has been deployed to which environment. We've used a symlink in the past to denote the current 'tip' build. This example will use a file. The file is structured like:

</serverType>

</blahboxType>

</fooboxType>

</versions>

Once the deploy is finished to a particular environment, logic in the deployment file automatically updates the respective project in the appropriate version file.

Here's a bit of sample code for a client side asp page for a Self Serve app. It passes some values to a server side asp page (deploy.asp) so that page will know which version of which project to deploy into which environment. The onchange event of the env select control called an AJAX function that retrieves the currently deployed version of the project from the versions.xml file.

<option value="">--Select--</option>

<option value="" & version & "">" & version & "</option>

</select>

<option value="">--Select--</option>

<option value="SB">Sandbox</option>

<option value="STG">Staging</option>

<option value="PRD">Production</option>

</select>

-</div>

</form>

Then on my deploy.asp page, the request parameters are set to variables which are then used to dynamically build the path to the deploy file (in this case a visual build file called Deploy-<serverName>-<component>.bld) This could be a nant, msbuild, or ant file as well, you'd just have to change which executable you're using in the executeString. The command is then executed in a Windows Shell and if it has a return code of '0' it was successful. Successful or not, we do some very simple logging to show who tried to deploy what, where, and when.
Here's the code:

' need a longer server timeout setting so our deploy can finish

Server.ScriptTimeout=1200

'instantiate some vars

env = ""

version = ""

project = ""

severType = ""

'set the vars

set version = Request.queryString("version")

set env = Request.queryString("env")

set project = Request.queryString("project")

set serverType = Request.queryString("serverType")

'gets the user

Set WSHNetwork = CreateObject("WScript.Network")

user = WSHNetwork.UserName

if version="" then

Response.Write("Please ensure you chose a version <br/>")

elseif env="" then

Response.Write("Please ensure you chose an environment")

elseif project="" then

Response.Write("error occured - project required")

elseif LCase(env)="PRD" then

Response.Write("We don't want to deploy to Production from this app")

else

Response.Write(version & " " & env & " " & project)

'set up a dynamic path here to the deployment file (nant or ant or visual build file) This string assumes the

'build file is taking a parameter called Env - the environment to deploy to

execString = "<pathToVisualBuildExecutable>\VisBuildCmd.exe /b " & Chr(34) & version & "\visual build\Deploy-" & serverType & "-" & component & ".bld" & Chr(34) & " Env=" & env

Response.Write("<br/>" & executeString)

set oFs = server.createobject("Scripting.FileSystemObject")

set oTextFile = oFs.OpenTextFile("<pathOfFileToLogTo>\deploylog.txt", 8, True)

Dim WshShell

Set WshShell = CreateObject("WScript.Shell")

proc = WshShell.Run(executeString,6,true)

if proc="0" then

response.write("<br/><br/><div style='color:green;font-size:16px;'>Deploy of build " & component & " " & version & " to " & env & " as " & user & " was successful")

dt = now()

logtext = dt & ": Deploy of build " & component & " " & version & " to " & env & " was successful" & vbCrLf

oTextFile.Write logtext

oTextFile.Close

else

response.write("<br/><br/><div style='color:red;font-size:16px;'>Deploy of build " + component + " " + version + " to " + env + " failed using user " + user)

dt = now()

logtext = dt & ": Deploy of build " & component & " " & version & " to " & env & " failed as user " & user & vbCrLf

oTextFile.Write logtext

oTextFile.Close

end if

response.write("<br/>Deploy times and results are logged at <pathOfFileToLogTo>\deploylog.txt")

set oTextFile = nothing

set oFS = nothing

set WshShell=nothing

end if

response.write("<br/><a href='javascript:history.back()'>Back to deploy page<a/>")

Key Questions in Software Sustainment

2010-07-10T22:16:00.000-07:00

Here are some of the questions I ask myself when a software system has stopped running properly...

The first big question is: What changed between when the application was running well, and the time the application stopped running well? To get to the bottom of that question, there are a number of other questions that can help point the way.

Question isolation - Is this problem isolated in any way? Is it only on a specific network, environment, or group of servers? Does it only happen for a specific group of users or a specific client? Is there a period of time it's isolated to? Is it isolated to a particular 'item' in your data? I've seen users try to make an application use a different browser version than the documented supported versions. Some times it takes a while to get to the bottom of simple issues like that. Also, we've run into situations where an organization will not have given users the rights on their machines to install third party active X components for their browser that the application they are trying to use requires.

Question data integrity - Whether you are looking at legacy data or no logic to manage special characters in your data, you need to break the problem down. With legacy data, is the problem isolated to a particular user or group of users, or a particular item? Or if it's an ETL function, does relational integrity from the first DB line up to the second DB correctly - are you missing or adding 'types' of data that the second DB either is or isn't expecting? Sometimes you need to pinpoint the exact row or time when the issue occurred to determine what the problem was.

Question continuity - Did something stop running or listening? A service, appPool, web site, 3rd party server? Are your cron jobs or scheduled tasks still there? Monitoring would quickly and easily answer this question for you.

Question communication - Are the lines of communication open to all of the dependencies that your application has? Has a network cable been severed by a backhoe down the street? (I've seen that before) Is there too much communication going on? Too many calls from one routine can take you system down. We had an issue like that with some javascript that called a data access function to a GIS server. As soon as one too many layers got added to the map, performance died as there were too many round trip calls to the server in one request. Internal systems that need to do identity verification or IP Geolocation are heavily dependent on external third parties to operate. These external vendors can in turn can be dependent on other external services. Know all your dependencies and have monitoring and SLA's in place for all of them or you could be sweating bullets.

Question dependencies - Here I'm thinking more in the context of internal dependencies - what internal 3rd party services are you dependent on? Databases, reporting tools, monitoring systems, document management systems, any internal system that your application depends on that you aren't responsible for fits into this category. Are they up and running? How do you know? Are they running properly?

Question the un-questionable - Is your hvac system working at your Co-Lo? As redundant as your service provider tries to be, there will always be something under the radar. Always. I've seen an external hvac system take down an entire enterprise. Many of you have likely seen a McAfee or a Norton patch take down an enterprise. How secure is your UPS (Uninterrupted Power Supply) management console? I've logged into one I've found by accident using username:admin password:password!

Question known changes - software enhancements and patches, file or db permissions, config changes, etc. I've seen issues where a changed file path will take down a critical Ftp routine, a 3rd party software patch for a document management system crippled a production system by removing key indexes in the database, and an (unfortunately un-automated) database refresh will have missing roles or users or the db will still be restricted mode. Doing anything manually can get you into trouble. A fat finger can push a wrong dll/library or mis-type an entry in a config file. Even fat fingering an automated deploy can get you into problems. Automated deploys still depend on data that is manually entered. We have pointer to app servers in our web.config files entered wrong and multiple entries in machine.config for a particular component.

Question security - Malicious changes are possible either internally or externally. Unfortunately, the question of security is a larger one everyday. And it has to be scrutinized at every level. Does your system have a firewall and IP Sec rules in place? Does you application provide you with an audit trail? How secure is your production data?

Resources at your disposal that can help in your investigation:
Log files - event logs, server logs, if you're logging to tables in the DB, don't forget to look there.
Users - it's like CSI - you need to get ALL the information you can about their problem. You cannot be afraid to ask.
Thread dumps - killing a hurting server and ensuring that it does a thread dump when it terminates can be very effective in your problem search
Networking tools - WireShark, telnet, ping, netstat - these are great for checking your communication.
Monitoring tools - Nagios, SCOM, GroundworkOpenSource
Books - Michael Nygard's Release It! and Luke Hohmann's Beyond Software Architecture.

More ideas

2010-07-10T19:12:00.000-07:00

Business idea: Create a computer screen with removable sides so there's not space when you put it against another computer screen - they'll almost look like one screen instead of having the 1-2 inch separation between the two.

Business idea: If I could create a machine/robot that would automatically wash windows I think I could make a lot of money. Every window I can think of besides one has to be washed manually by hand. All the high rises downtown still have all their windows washed manually. The only place they use a different technique as far as I'm aware is with race cars where they use layers of thin plastic over the windshield. They rip off a layer every time the car is in the pit and instantly the driver has a clean windshield.

Business idea: A new service industry helping people/seniors navigate their way around the health maze we have created for ourselves. You need a Personal Medical Manager! Here's why: There are significant disconnects between specialists who analyze medical tests, the lap techs who perform/analyze the tests, and the doctors who originally asked for the tests. Then there's another layer of indirection at the pharmacy. How does an older person manage all their prescriptions and make sure they aren't 'colliding' with each other, or even just remember to take them all and follow all the directions? I've had my wife 2nd guess a doctor's prescription, ask the pharmacist about it, and have him adamantly agree with her that the prescription was wrong.

Political idea: Why are we still running democracy the way we are? The reason we elected people as 'representatives' long ago was so they could represent us in parliament (or the house, or congress) - or at least represent our vote. Lobbyists have done away with our representation in the vote. But why do we need a representative now anyway? Technology could allow us to instantly vote on every single issue before parliament either using a computer or a phone. We wouldn't need a representative to vote for us anymore. Don't get me wrong, I'm all for democracy! I just wonder if it can be done better now with technology and removing the 'middle man' and the lobbyists.

Code Repositories

2010-06-04T10:05:00.000-07:00

Over lunch the other day, some collegues and I were jawing about different code repositories that we've used, and how clients/projects that we've worked with have evolved to use different VC products. At one point, all six of us had worked together on a big (successful) online wallet system. It was originally on Perforce. We migrated the whole system to Subversion because we were having problems managing code/library dependencies between various projects. Things worked great in Subversion for what we were doing at the time (this was 4 years ago or so). One of the developers I worked with there was on the team for Monotone - a precurser of Mercurial - in that they both followed the distributed versioning system model.
In the last year we made the change from ClearCase to Subversion, and for what we were doing it just felt (and continues to feel) so much better.
A couple of guys that were eating with us at lunch are working with a client that switched from Subversion to Mercurial earlier this year. They said it was a bit of a paradigm shift for all the developers in the organization as you have to think about how you manage your changes a little bit differently. One of the guys also forwarded me this article that is really good at explaining the differences (and has a link to a tutorial the author wrote).

CMS systems

2010-06-03T07:52:00.000-07:00

I found this good post http://designtutorials4u.com/15-great-content-management-systems-for-designers/ on some of the more popular systems out there. Unfortunately, it doesn't document specifically the technology they use or differentiate very well which ones are 'free' vs. which ones you have to pay to use. I've also run into a couple others lately:
Umbraco - an asp.net based cms
and
Alfresco - a java based cms

More Software Emergency Fixing Experiences

2010-05-13T08:25:00.000-07:00

I've recently helped two more clients with 'emergency fix' issues.

The first issue was a problem a national grocery chain had with an export from a SAS program that did a pull from multiple database tables into an excel spreadsheet, and subsequently imported this data into a MS Access database using a SSIS (Sql Server Integration Services) package. The SAS export seemed to be fine - that is, it executed without errors, but the SSIS package would fail. From my understanding, the client would run this process about every month or so, so there were a fair number of records (1000's). However, they started having this issue in Dec. and I didnt' get called about the issue until late April, so there was a backup of data.
The client didn't have the SQL BI studio installed, so we had to go through the idx file manually to see if there was anything obviously wrong. There didn't appear to be. The database and excel connections tested fine. There didn't appear to be anything riotously wrong with the data formatting or invalid characters in the excel spreadsheet. So, what do you do?
Well, for one thing, don't test with the full dataset if you think it might be a data related issue. Keep making your dataset smaller until you are sure that none of the data works. In this case, we cut the dataset in half, started from Feb to the end of Apr and the import worked! This confirmed for us that there were no permissions issues, or software upgrade issues (we were considering that software upgrades might be part of the problem as more than one piece of the software had been upgraded in that time period). The successful test confirmed for us that we were dealing with a data related issue. From there it was just a matter of determining exactly which rows of data couldn't be imported, and what it was about the data in those rows that botched things.
Less than 5 hours of work to this point on this fix.

With the second emergency, a telecom company was having issues getting an automated FTP (file transfer protocol) process to work. They had changed the paths of where the ftp files were residing and they were voip experts, not so much into the bash shell scripting thing. Their first problem was pathing in the bash file. That was an easy fix. Then there was a problem with sending a notification email that apparently they never got to work. This was a little bit more tricky. They originally had the email coded similar to this example.

       #!/bin/bash
       telnet smtp.example.org 25 <<_EOF
       HELO relay.example.org       MAIL FROM:<joe@example.org>
       RCPT TO:<jane@example.org>
       DATA       From: Joe <joe@example.org>
       To: Jane <jane@example.org>
       Subject: Hello
       Hello, world!
       .
       QUIT       _EOF

However, after a bit of work I discovered that the server this script was running on was processing all the command too quickly for the smtp server to get the email off. In the end, I had to do something more like this:

       #!/bin/sh
       ( echo HELO ccielogs.com       
       sleep 2
       echo MAIL FROM:<test@somedomain.com>
       sleep 2
       echo RCPT TO:<test@someotherdomain.com>
       sleep 2
       echo DATA
       sleep 2
       echo Subject:Test-Mail!
       sleep 2
       echo If you can read this, it Works!!!
       sleep 2
       echo .
       sleep 2
       echo QUIT
       ) | telnet XXX.XXX.XXX.XXX  25

Then to set the bash script to run in a windows scheduler (using cygwin) I had to do something like this:

       C:\cygwin\bin\bash.exe --login -c "/myfolder/myscript.sh"

Less than 5 hours to fix.

Param datatypes with Sql Server Stored Procs

2010-04-26T20:48:00.000-07:00

I tried several times to save and post this blog entry earlier today and it would NOT save. I have no idea what is wrong as I tried to email it to myself so I could do it later and I couldn't email it either. I'm thinking there was a bad character in my original text, but I pasted in into (and copied it out of notepad) so... I'm not sure what is going on. Sometimes these tools in the 'cloud' seem to be in a 'fog'.

Anyway, what I wanted to say was....
I've been working with stored procs lately in Sql Server. My stored procs take search parameters that I wanted to be very flexible - strings with wild characters on either end using a LIKE in the SQL. Originally in my declaration for the parameters I used a char datatype. What I didn't realize was if the param passed in is shorter than the length you specified, a char datatype will automatically add placeholder characters to fill up the rest of the space. So, if you declared a char with 9 characters and passed the SP a 3 character string, SQL Server would automatically add 6 more characters to it. I was then trying to prefix and append the percentage sign (wild character) on either end of my param, but it was getting ignored.

In the end (with the help of my more DB inclined colleague, Karlson) I changed the datatype of my param to varchar and I could happily prefix and append wild characters to it.

Got another interesting business idea over the weekend - it's based on a few assumptions, and it's for the investment/financial industry. The assumptions are:
- history repeats itself.
- the stock market, like history, is cyclical in nature
- investment funds are just as much marketing as they are statistical science
- people will buy anything that is marketed well, whether it provides them with tangible value or not - (the value of stock is all about perception of value)
The idea is simple - create a fund that is based on the cyclical nature of the stock market. Predictable things happen when interest rates go up, or when commodities or high. Convince people that you can harness the 'value' in the history of what happens to the stock market with certain indicators are up, and you'll have a winner fund.

Network isolation

2010-04-21T15:51:00.000-07:00

For the second time this week, production has gone down for an extended period of time. This time it was because of this bug introduced by McAffee. It was proliferated through all of our environments (apparently) by SSCM. I'm not sure why we are putting McAffee fixes right into production without testing them in a staging environment first. Seems to me like there is a wrong process there somewhere - Either on our side, McAffee's side, or both.
Before we realized this was a McAffee bug, it seemed that this 'virus' got propagated to production from lower environments. Having the production environment on an isolated network from the rest of our development and test environments would have protected us to some degree from this kind of propagation. I have (officially) suggested this more than once, but resourcing and other priorities have gotten in the way of making it a reality.
For future reference though, here's just another reason to have a production environment on an isolated network.

Single Points of Failure

2010-04-20T21:50:00.000-07:00

Within the last 8 months, all of the production servers for a client of mine were moved out of the building they've been in and into a 'bunker', or co-lo. There was a variety of reasons for doing this. As an architect choosing a Co-location provider, and planning the move, you would want to make sure as many systems there are as redundant as possible - power supply, UPS system, networking, etc.

Well, this week everyone painfully discovered a system that wasn't redundant (or if it was, it wasn't as redundant as it should have been). Apparently the HVac systems went down. This resulted in servers getting too hot and consequently having to be shut down. Production servers.

Everything was resolved in a couple of hours, but it just goes to show, it seems that there is always one system that is forgotten.

I've actually run into another situation like this before. It wasn't nearly as big an issue for our shop as it only affected development environments. However, the Calgary airport(!!) along with most of NE Calgary was without a network connection just as long as we were. Apparently a construction crew was using a backhoe doing some digging at an intersection and accidentally cut a main networking cable that supplied most of NE Calgary with it's network connection. Kind of makes you wonder if municipalities should be considering redundant underground networking, doesn't it?

Networking Architects work hard to eliminate as many single points of failure in their systems as they can. Some are hard to control though. Recently McAfee released a virus library update that wasn't tested properly and as a result shut down a plethora of systems across North America. It thought a Windows dll was a virus (false positive) and sent thousands of systems into a perpetual reboot. It would be an interesting bit of process engineering to figure out the best way protect your production systems from the most current viruses AND protect them from bugs like this sent out by the 'big' virus scanning companies at the same time.

Google Tools post

2010-04-19T09:36:00.000-07:00

Good link here that lists the top Google Tools for website developers:
http://sixrevisions.com/tools/the-top-15-google-products-for-people-who-build-websites/
Personally, I start to get lost in the plethora of tools out there. Just getting Adwords properly integrated with Analytics is a headache I don't really have time to deal with (when I have so many other higher priority things to manage). I wish some of these tools where a little more straight forward to use.
My hesitation in making the investment to learning one well is that it will change too soon and too much for me to keep up with (I do have a 9-5 job along with at least half a dozen clients on the side). I need to focus my attention on where I get the most return on my time investment - and learning how every single Google Tool works doesn't rate that high.
It is a good post for reference, though. I like knowing what all the different tools are out there.

More Ideas

2010-04-14T19:09:00.000-07:00

Here's some more business ideas that I've had lately. I want to write them down so I don't forget them as they are fun to discuss with people.
- An automatic window washer for houses. Living in Calgary with all the construction and wind, windows don't stay clean very long. Wouldn't it be nice to have a little 'thing' (like the robot vacuum cleaner) that you could run on your windows?
- A job website specifically targeted to the baby boomers and finding them good part-time work.
- Any kind of inventions/devices that will help baby-boomers stay in the workplace longer. I'm sure their feet would get sore after serving customers for a couple of hours in Tim Horton's. Wouldn't it be interesting if you could install 'chairs' that ran on a 'track' behind the counter in fast food restaurants for the servers to sit on?
- I learned at a conference this past weekend that apparently there is no Canadian company offering servers on demand in 'the cloud' like Amazon's EC2. There is a potential business opportunity here for a smart Canadian company with a bit of infrastructure in place. Eucalytpus is basically a freeware version of the same software that Amazon uses to run it's cloud. Set a few servers up with that and you're ready for Canadian customers. The advantage here is other Canadian businesses are hesitant to use American cloud solutions because of the American Federal government's attitude toward privacy. Host your cloud in Canada and don't worry about the American 'Fed's' keeping an eye on your business data. :-)

More lessons learned (not mine though)

2010-04-06T06:45:00.000-07:00

Here's another good link to a lessons learned article: http://java.dzone.com/articles/lessons-learned-taking-project It talks about how to communicate, what to have in place, and how to manage client expectations on an 'emergency fix' project. I have a special interest in this type of thing as I'm testing a new business idea here: http://www.netfocusconsulting.com/Experts.jsp

Considerations with a site re-design

2010-03-27T10:39:00.000-07:00

I just completed a site redesign for the Alberta Roofing Contractors Association. They wanted to refresh the pages a bit, simplify the navigation and add Google Custom Search. Here's some things I learned...

We switched from static html pages to php pages that used simple includes for the header, footer, and navigation. We also reorganized the folder structure a bit to clean things up. Consequently all the pages/pdf's that were previously indexed on search engines 'disappeared'. I wasn't sure what to do about this at first, but then I found an .htaccess file in the root of the ftp directory on the server and I made 301 redirects for all the old html pages and pdf files. This worked great. Lesson Learned: a simple site redesign on an established site isn't as simple as it would seem on first glance.

I found re-indexing Google custom search to be a pain. I had to add the site into Google's webmaster tools (both www.arcaonline.ca and arcaonline.ca), set the old pages up to be removed from the cache, create a sitemap.xml file and upload it, create a robots.txt file and upload it, and add all the new url's for pages and pdf's in the 'explicit' indexer. It was rather cumbersome, but I got it to work in the end.

There were a number of issues with compatibility related to the search engine form and the flash on the home page that we had to work around using CSS hacks for IE ( *+html ).

Using a wrapper in CSS is a big help in some cases to make background images display properly in all situations. Something similar to this:
#wrapper {
margin: 0px;
padding: 0px;
width: 100%;
background-image: url(../images/bg.jpg);
background-repeat: no-repeat;
background-position: center top;
margin-left: auto;
margin-right: auto;
}

Automatic Deployments taken to a new level

2010-03-17T12:44:00.000-07:00

I was doing some research for an article I was thinking about writing on Automated Deployment and came across this fantastic blog post by Grig Gheorghiu. He talks about how OpenX uses Amazon EC2 (virtual machine cloud) to automatically provision and terminate servers of different types, different monitoring solutions they use, and in another post he talks about the different automated deployment technologies that can be used (push vs. pull, ruby, python, etc).
I personally liked the comments about 'never having to log onto a production server', and "There's nothing like real-world network traffic, and I mean massive traffic -- we're talking hundreds of millions of hits/day -- to exercise your carefully crafted system infrastructure."
All the comments at the bottom are really good as well.

CPU's and Sql Server

2010-02-23T06:57:00.000-08:00

Again in our staging environment getting ready for a release in production, we were seeing unexplained slow response times for our AgilePoint application when compared against production or development envs. After some digging we discovered that our production DB server that serviced the AgilePoint DB had twice as many CPU's as the dev DB server, and 4 times as many CPU's as the staging server.

What I didn't know is that in Sql Server you can assigned/limit how many CPU's the database can use. This setting is in Server Properties on the Advanced tab, and is called 'Max Degree of Parallelism'. Like many of the other settings with Server Properties in Sql Server, 0 equates to 'unlimited' or in this case 'all processors'.

The importance of Process (adodb.dll and SSRS)

2010-02-18T10:13:00.000-08:00

We have a number of different development environments. Deployments follow a natural progression through these environments before they get put into production. We were getting reading for a release and doing deploys and testing in our Staging environment (our env just before prod in our progression). One morning recently, our Nunit tests failed after a run, complaining about not finding adodb.dll registered.

We were surprised because adodb.dll is a Microsoft dll and we certainly weren't deploying any dll's like that - for that matter, we hadn't done any deploys in the last day or two. So why were our Nunits failing suddenly?

Quickly looking into the event logs, we discovered that SSRS (Sql Server Reporting Services) had been (unbeknowst to us) uninstalled on our transaction server during the evening. I went and spoke with the infrastructure group who did the uninstall and they got very concerned as they were ready to uninstall SSRS in production the next evening.

In the end, it appears that uninstalling SSRS did, in fact, remove adodb.dll from the Gac on our transaction server. Our infrastructure group cancelled the uninstall from production. We are trying to impress upon them and others the importance of communicating changes like this even if the assumption is there is no impact. Fortunately, our process of percolating changes up in different environments managed to catch this change before it blew us up in production. However, it wasn't caught in the lower environments because it was unadvertised. Communication is key!

Things learned from Website moves

2009-11-24T12:44:00.000-08:00

I recently moved several client sites off of servers that they had been on for the better part of a decade to new VPS boxes. They had MySql backends and used Java/Velocity CMS system (InfoGlue) hosted on Tomcat with an AJP connector to apache web servers on the front end. One site uses htdig for it's search implementation.

Issues (with resolutions) that I ran into...

1. GZIP'ing a running Tomcat instance
Before one of the sites was moved, a client tried to make a backup of the Tomcat instance that they were running under. They executed a:
gzip -r filename.gz ./tomcat/
which ended up recursively zipping all files under the tomcat directory. This crashed the running instance of Tomcat. (Perhaps trying to tar or gzip a running tomcat instance isn't a great idea. Copy it first and then tar/gzip that) Everything was quickly unzipped and they tried to restart Tomcat but it wouldn't start. After an hour of search, I was asked to help. I poked and prodded for about 90 minutes and discovered that all the webapps were missing the
/WEB-INF/web.xml
file. I don't know how/why gzip and upzipping would make these files disappear, but they were there anymore. So I replaced them and tried restarting Tomcat and everything came up.

2. Problems with MysqlDump
I had a couple of problems backing up the MySql database. First MysqlDump would execute, but not put all of the tables into the backup file. I discovered that I had run out of disk space. Then it turned out that mysqldump doesn't care about relational integrity when it creates the backup script so when I tried to import it into the new database I ran into errors complaining about foreign keys. To resolve this, I had to add set foreign_key_checks=0; at the top of my backup script. This allowed me to import it successfully.

3. Mysql connection pool Exhausted
After getting the db imported and the web server moved over, the application seemed to be running fairly nicely on the new VM's. Then I left it alone for a couple of hours. When I came back it was throwing connection pool exhausted exceptions all over the place. I could resolve this by logging back into InfoGlue's cms instance, but that wasn't the right resolution for me. So I did some checking. Many people suggest to set the wait_timeout parameter in the my.cfg file for mysql higher or lower. I tried that and it didn't seem to work. What I ended up doing was adding a line in my database.xml file:
<param name="validation-query" value="select * from someTable" />
This helps keep the connection alive by pinging the db with the query every so often.

4. MySql Data Directory Change
While trying to resolve the previous issue I was changing/adding parameters into the mysql my.cfg file. I did a copy/paste which had a number of properties, one of which was the pointer to the data directory for mysql. This pointer was different than what I was using. As a result, when I restarted mysql, all of my databases, tables, users, and data was gone. I freaked. Then after thinking about things for a little bit and checking through files one more time I realized the mistake I had made and commented that line out. Restarting mysqld brought all my databases, users, and data back again. Whew.

5. Getting HtDig going again
The original installation of HtDig on the old server hadn't been indexed in over two years because cron had been broken. I didn't have access to the root password to fix this problem so the client was very interested in getting it running again on the new server. I had no previous experience with HtDig.
I copied over all the files I could find related to HtDig off of the old server and installed them on the new box. After a few tries, re-indexing worked, but I still had a problem with displaying the search results page in the web site. It turned out that I was missing a virtual directory configuration in my httpd.conf file for the directory where htsearch was running from (as a cgi script). The only reason I figured that out was by using lynx (linux CLI web browser). After fixing that, I got my newly indexed results displaying on the web site.

6. Using /etc/hosts helps
I've found that the /etc/hosts file is a big help in moving sites like this - whether you want to test the site while the live site is still running, or quickly configure pointers to a server dependency.

7. MySql sql queries are case sensitive
One of the website moves I did involved porting a MySql database from a Windows box to a CentOs box. The dump on the Windows box made all characters for tables lower case. I didn't pay too much attention to this at the time. They imported the same way.
When I started up my Tomcat server, it tried to start up, but threw errors related to jdbc connection pool and a ValidateObject. After a bit of googling I discovered that this is related to the validate query (the query I wrote about earlier that checks the connection every so often to make sure it hasn't gone stale). I tried running that query right on the mysql box and it would run because the table name was all lower case. So I changed all my table names to camel case and things worked.

8. Issues with unknownDefaultHost and serverNodeName in Tomcat
My inability to set the serverNodeName was resolved by adding a line in my hosts file to point the IP of my box to the Host variable found when I run the 'set' comment in CentOS. My issue with unknowDefaultHost was resolved by going into the server.xml file and editing the Engine element's defaultHost attribute - changing it from my old domain name to 'localhost'

9. MySql error 1153 - Got packet bigger than 'max_allowed_packet' bytes
Got this error a couple of times with different DB imports. To resolve I needed to go to the my.cnf file (found sometimes in /etc/) and set that property like this:
max_allowed_packet=16M
for the mysqld (mysql daemon). Then I had to restart the daemon (/etc/init.d/mysqld restart) and I could run my import (mysql -p -h localhost dbname < dbImportFile.sql)

Musing on Automated Deployments

2009-11-11T10:41:00.000-08:00

I have been a key player in big automated deployment strategies in two significantly sized organizations now. One used ant with a java code base, the other used Visual Build with a VB code base. Both of these implementations deployed multiple dependent projects onto a variety of server types into development, testing, staging, and production environments. With the exception of prod, each environment had more than one instance of the environment running.

Some of my earlier musings on automated builds and deploys can be found by clicking here.

One would think that an automated deployment would be deterministic. In other words, given the logic in the deployment file(s), it should deploy the same every time. Surprisingly, we have found this not always true. Since many of these deploys are pushed to remote boxes, hiccups in the network end up throwing a proverbial wrench into things. And (again) surprisingly, these can occur more often than I would've though. We actually blamed these hiccups on increased solar activity for a while. I have no solutions to getting around these network hiccups, except to say that if you see your deployments failing consistenly at a certain time during the day, schedule them for another time. Our Sunday evening deploys lately have always been failing. Yet when we kick them off Monday morning (with no changes to deployment logic) everything this fine. We're thinking that there's possibly a weekly batch job or two that are running during our Sunday deploy that is bogging the network down....
I've also seen automated deploys act inconsistently (only with Windows) with registering dll's in the assembly. We can deploy and Gac things fine onto our bare metal, VM servers with no problem. Yet, when we deploy the same software onto a legacy hardware server where the dll's are already gac'ed (our deployment logic un-gac'ing and re-gac'ing the dll's) they fail to register it seems. I've wondered if perhaps the deployment moves through all the logic too fast? The command is definitely correct. Sometimes we'll even see the dll's in the assembly folder in the GUI, but the application can't. Manually registering them from the command line fixes the problem, but we shouldn't have to do that.

Something else to consider when implementing automated deploys - do you want to deploy everything from scratch (bare metal deploy) or do you want to deploy onto an already working image or server (overlay deploy)? I have tossed this question around a number of times. I think the correct answer for you depends on how you answer the following questions:
Are you thinking about deploying to a system that's already running in production? Are all the configurations that make that production system work documented? Are you confident that you could rebuild the production server and getting it running without any major problems? If you answer 'yes' to all of these questions, then you could probably save some time and implement overlaying automated builds. If you are starting work on a greenfield (new) application or you aren't confident that you could rebuild you production server, then you should probably consider bare metal deploys. Bare metal deploys done properly essentially become self documenting DRP's.

Cookies and Perl

2009-11-11T09:53:00.000-08:00

I have a client who is using and old (outdated and unsupported) php/perl CMS as an intranet site. In implementing a DRP (Disaster Recovery Plan) they switched all the server references in the code and on the server from the server IP address to a DNS name. This was done using a search and replace :-(
This effectively broke the intranet site. They were able to get most of it back up and running with the exception of logins and how sessions were managed. For some reason, once this CMS saw a domain name instead of an IP address, it changed the path in the cookie to something like this:
mywebserver/http://mywebserver/somesite
from what worked before which was:
mywebserver/somesite
We searched through config files and didn't find anything conclusive to begin with. Then we found some code where cookies were being created and modified it so that after the cookie string was created we did some string manipulation on it like this:
$setcookie =~ s/http:\/\/mywebserver//m
This resets the setcookie var by searching through the setcookie var (it's a multiple line string so we needed to use /m) for the string 'http://mywebserver', replacing it with nothing.
It turned out that the CMS was creating more than one cookie, so we found this bit of code in two more files, added our hack there, and sessions worked again!
I have to confess, I was surprised that this little hack worked.

IIS default app pool proc terminated

2009-10-27T14:26:00.000-07:00

Here's a great link to error codes for w3wp (event id 1009) and what they likely mean:
http://blogs.iis.net/brian-murphy-booth/archive/2007/03/22/how-to-troubleshoot-an-iis-event-id-1009-error.aspx
My current issue is error 0x0 which is documented in this link but fairly sparse. I checked the debugger flags with gflags and everything was unchecked. I'm still looking for a resolution to my issue.

***

I did find a resolution to this issue. I went through all the configs and our DRP (Disaster Recovery Procedure) document to see if a could figure it out. No dice there. I downloaded and hooked up procmon to see if it would tell me which registry key or file was causing the issue. No dice there either. It turned out that since our app had a dependency on SoftArtisan's FileUpEe, the fileUpEe dll's were registered in the wrong order! I had registered them 'com' folder first, 'dotnet' folder second (per the DRP). They needed to be registered 'dotnet' first and 'com' second. That made my IIS appPool stop cratering (it was definitely the issue).

Tools for working with SVN & Visual Studio

2009-10-14T08:18:00.000-07:00

We're moving to Subversion as our code repo. Tools that we are using are:
- VisualSVN Server on the server box for maintaining the svn server (managing users, creating/importing new repos, etc)
- TortoiseSVN on client boxes. This works in conjunction with windows explorer to tell you the status of files in your local copy of the repo. We've found that the icons don't change status immediately - you need to be patient with them.
- Collabnet AnkhSVN - Subversion plug-in for visual studio. Allows you to see the status and check files in and out inside of VS.
- Collabnet SVN command line client. We found we needed this (particularly svn.exe in the PATH environment variable) if we wanted to run the SVN steps in a Visual Build file.

Monitor the right things

2009-10-07T06:19:00.000-07:00

I started re-reading Release It! by Michael Nygard this morning on the commute into work. In his first chapter he talks about a (very small) issue that turns into a colossus and takes down an airline's check-in system. The system had a monitor configured and performing checks on it, but it turned out that it wasn't checking the right things (it was looking at the http port on transactional servers when it should have been looking at the RMI port).
It totally reminded me of something that happened about a month ago. We have a bunch of web applications that run on our production server. After fine tuning our monitoring to look at pages that the application has to apply logic to to server up (rather than a static home page) we found that our monitoring corresponded much closer to complaints from users.
Think twice about what you want to monitor and where to point it.

MS Sql Server bug

2009-10-05T13:53:00.000-07:00

We ran into an Sql Server bug today that was rather interesting. We were implementing synonyms across a number of views, tables, and stored procs in a couple of DB's. Everything was working fine until another team did a deploy and changed the index on a view that was referenced by one of our synonyms. It turns out that there is a documented bug which requires that any time the DDL has changed on a view that is referenced by a synonym, that synonym looses it's connection to the view. This includes just updating the index to the view.

Handy Utilities for Windows

2009-09-03T07:19:00.000-07:00

I don't want to forget about these handy software utilities for doing troubleshooting and debugging on the windows platform.

SpaceMonger is a tool that graphically displays how data is utilized in you hard drive.

SysInternals software...

TcpView essentially provides a graphical view of a netstat on a windows box with a few extra handy features (counting the number of sockets in time_wait, for example).

Procmon allows you to get information about what is happening with a specific process id.

windows account security

2009-09-02T10:13:00.000-07:00

The domain (active directory) policy will override any local security setup for a particular account on a server. If the domain security policy says that an account will lock after 5 failed attempts and the local security policy says the account will lock after 3 attempts, the account will lock out after 5 attempts.
Here's the interesting thing:
The account lockout counter is reset every 24 hours or with every passed login attempt. So if you have two services using the same account, one with a correct password and one with an incorrect password, you can likely run indefinitely before the account will lock out. One service will never work and you should get a lot of errors in the security event log.

More IIS automated deploys

2009-08-26T07:39:00.000-07:00

Been working more with IIS automated deploys and found a couple of good posts in David Wang's Blog along with some other good blog posts.

This one is related to how to manipulate IIS list data. He provides a handy vbs script to do this that worked nicely for me in adding a new application mapping to IIS.

This one talks about IIS App pool crashes - fatal communication errors between the Application Pool and IIS.

This one talks about app pool recycling and IIS availability. There's some great conversations past David's article further down the page.

This one (not David Wang) shows how to change COM+ MSDTC settings programmatically. This script worked great for me as well.

This one (Ian Morrish) shows how to manipulate COM+ Security launch and activation permissions using DComPerm from the windows SDK.

With all of these scripts, I found I had to carefully read the directions and make sure I had the parameters that were being passed in correct. I was often passing in erroneous parameters on my first couple of tries.

Windows bug with adding users to a group

2009-08-21T13:11:00.000-07:00

I was using a script to try and add users to a local group today in an automated build script. I was sent a script that worked, but when I plugged in my values, it didn't. I always got a script syntax error, like one of my parameters was not in the right place, or had spaces or something. Well the group name did have spaces, but that wasn't my issue.

It turns out that there is a documented Windows bug (http://support.microsoft.com/kb/324639) that limits the username to be added to only 20 characters. Any more and it won't run.

Log parsing/network security tools

2009-08-14T10:38:00.000-07:00

I discovered some new tools today that are useful for network security. QRadar from Q1 labs (http://www.q1labs.com/) is a really slick log parsing tool for organizations that are looking to implement a distributed log management offering to collect, archive, and analyze network and security event logs. It then parses this information into graphs and data that you can tune to alert you when things go awry. You can configure it to look at firewall logs, web server access logs, event logs, etc.
Splunk (http://www.splunk.com/) seems like it might be a competitor. At a glance, I'd say that QRadar has a lot more features and might be a lot more expensive.

XML manipulation in Visual Build (and vbscript)

2009-08-14T10:29:00.000-07:00

We've been working on basically turning our Visual Build deployment files into DRP (Disaster Recovery Plan) scripts. Essentially, everything required to get our applications running on a 'blank' server box is documented as a config, permission, or push in our Visual Build files. This has required updating then machine.config and web.config files that reside under windows/microsoft.net/framework/v1.1.4322 or v2.0.50727/config at times. Sometimes we're adding attributes to existing element, sometimes we're adding entirely new elements.

It seems to me that adding entirely new elements in Visual Build (using the Write XML Action) has a bug in it. It will add new attributes to existing elements no problem. But it won't add new elements. So we hacked around that and used the 'Run Script' Action with vbscript to add elements to our files. To do this in vbscript it looks something like this:

Dim objXMLDoc, objNewNode, objText, strXPath, objParentNode, objChildNode, objCaseExist

Set objXMLDoc = CreateObject("Microsoft.XMLDOM")
objXMLDoc.async = False

' load the XML file - make sure to include the fully qualified path
fSuccess = objXMLDoc.load("\\%TARGET_SERVER%\c$\WINDOWS\Microsoft.NET\framework\v2.0.50727\config\web.config")
If Not fSuccess Then
wscript.echo ("error loading XML file")
Else
wscript.echo ("XML file loaded")
End If

' set to proper node
Set nodeList = objXMLDoc.getElementsByTagName("configuration/system.web/compilation/buildProviders")
If nodeList.length > 0 Then

Set objParentNode = nodeList(0)

' add the SGAS SecurityDisabled element/node
Set objNewNode = objXMLDoc.createElement("add")
objNewNode.setAttribute "extension", ".uplx"
objNewNode.setAttribute "type", "System.Web.Compilation.PageBuildProvider"
objParentNode.appendChild(objNewNode)
objParentNode.appendChild objXMLDoc.createTextNode (vbCrLf)

set objNewNode = Nothing

else

wscript.echo ("node list empty")

end if

' save the XML file
objXMLDoc.save("\\%TARGET_SERVER%\c$\WINDOWS\Microsoft.NET\framework\v2.0.50727\config\web.config")

Error 8510 and MSDTC...

2009-08-06T07:11:00.000-07:00

We had the whole development environment down for a day. After spending a good bit of time debugging, we discovered that we were getting a significant number of errors in our event logs saying:
Inner: The transaction has aborted.
Inner: Failure while attempting to promote transaction.
Inner: Fatal error 8510 occurred at Aug 5 2009 1:09PM. Note the error and time, and contact your system administrator.
A severe error occurred on the current command. The results, if any, should be discarded.
They were thrown against the running of several different stored procs and because of the architecture of our system, all of our COM+ components were rendered useless.

I found a post that talks about 'fatal error 8510' that was interesting:
http://blogs.msdn.com/asiatech/default.aspx - go to the bottom of the page (it's a ways down). However that turned out to NOT be the resolution to our problem

The resolution to our problem resided in the fact that SQL Server could not initiate a distributed transaction. We had thought that there was a problem with the MSDTC cluster, but that turned out to not be an issue as other sql servers in the cluster could initiate distributed transactions. Restarting services on the sql server that could not initiate distributed transactions resolved the problem

linux reboot and chkconfig

2009-07-31T21:35:00.000-07:00

Setting up a new linux server pair for clients, I've did some things I haven't done in a while, so I thought I'd document them here so I'd remember in the future.

To set up a service in linux, use chkconfig. Running the chkconfig --list command shows all the potential daemons that are sitting in /etc/init.d/ and whether they are set to on|off|etc for each linux run level. The main run levels to set are 2,3,4,5. You can set the to on for http like this:
chkconfig --level 2345 httpd on

It's also a good idea to go to the script for that service in /etc/init.d (httpd in this case) and modify the chkconfig line at the top of the file to add those changed params.

To reboot the server you can run reboot. It actually calls shutdown -r or shutdown -h depending on the linux implementation.

Automated Windows Deploy scripts and tricks

2009-07-20T13:25:00.000-07:00

Permissions (domain is optional):
cacls drive:\folder /T /E /G "domain\username:C" - :C modify permissions
cacls drive:\folder /T /E /G "domain\username:R" - :R read permissions
cacls drive:\folder /T /E /G "username:F" - :F full permissions

net localgroup "IIS_WPG" "domain\username" /add - adds user to IIS_WPG localgroup

IIS configs:
Set the friendly name for an IIS website:
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/AppFriendlyName IIS_WEB_SITE_Name
Set the port of a website:
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ServerBindings ":8080:"
Install an IIS Application pool:
cscript drive:\path-to\adsutil.vbs CREATE w3svc/AppPools/AppPoolName MyApplicationPool
Set identity type on AppPool
cscript drive:\path-to\adsutil.vbs SET W3SVC/AppPools/WebSiteName/AppPoolIdentityType 3 (2 is the predefined network service account)
Set appPool username
cscript drive:\path-to\adsutil.vbs SET W3SVC/AppPools/WebSiteName/WAMUserName username
Set appPool password
cscript drive:\path-to\adsutil.vbs SET W3SVC/AppPools/WebSiteName/WAMUserPass password
Delete a web site:
cscript c:\windows\system32\iisweb.vbs /delete "WebSite name"\
Create a web site:
cscript c:\windows\system32\iisweb.vbs /create "IIS File Path" "WebSite name" /dontstart
Set server bindings
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ServerBindings :80:myWebSite :80:myWebSite.com
Set virtual directory permissions
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/AccessFlags 513
Set enable anonymous user for vDir
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/AuthAnonymous TRUE
Set vDir anonymous username
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/AnonymousUserName username
Set vDir anonymous password
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/AnonymousUserPass password
Set vDir to use integrated windows authentication
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/authNTLM TRUE
Set vDir ASP.NET version
C:\WINDOWS\Microsoft.NET\Framework\version_number\aspnet_regiis -s W3SVC/WEB_SITE_Number/root/vDir_Name

Troubleshooting

2009-06-09T09:44:00.000-07:00

Things to always watch out for when troubleshooting:
- locked out users - users who have been locked out of the system because the wrong password has been entered 3 consecutive times
- out of disk space (proper monitoring would fix this problem)
- GUI display that is contrary to what is actually happening on the system. Gac'ed dll's that really aren't gac'ed. In other words, assume nothing.
- inconsistent application of permissions across environments.
- duplicate libraries/dll's on machines.
- closed ports or ports/services not listening/running. Also port conflicts. More than one service wanting to use the same port.
- services, IO connections, db connections not getting closed/terminated properly.
- logic between functions or classes that creates infinite loops

IIS binary formatting error

2009-06-05T07:09:00.000-07:00

Oh man, did I get a slap in the face yesterday. I was helping/coordinating a patch into production for our team and was asked if we needed to 'drainstop' (read turn off load balancing) to deploy to our app servers. I said no, I had never done that before here. Later I realized that I probably haven't done a prod patch to the app servers here period.
Anyway, we pushed our change onto running servers and everything appeared to deploy fine until we actually tried to use them. It seemed that even though we pushed new components and gac'ed them, that was ignored because the server still had a running request on the old one. Once that request was gone, our app servers were down. When our app servers go down and our web servers try to access them in that state, we get this wonderfully intuitive error called a 'binary formatter error' with a bunch of junk that doesn't make any sense.
So, in the end, two lessons learned:
1. Always drainstop servers when you're deploying onto a live system
2. Binary Formatter errors can mean that you communication with a remote servers is down.

Tomcat Alias

2009-05-24T14:32:00.000-07:00

As my previous post says, I went live with a new website on Friday. Well, within minutes of going live I realized we didn't have an alias set up for www.myNewSite.com and got my client's administrator to add it. Once it was made live, the alias still didn't work. I had it configured in my tomcat server.xml file, but no dice. What was going on?

When I installed tomcat, I got rid of all the default sites it comes with (and I thought I had gotten rid of all their configs as well. It turned out I hadn't. In the server.xml file the host was configured correctly, but the 'engine' configuration was still pointing to 'localhost' which wasn't in my host configuration anymore. So I pointed it to my new 'root' configuration for the server - so now it looks something like this:

Engine name="Catalina" defaultHost="www.myNewSite.com"

and the host lookes like this:

Host name="www.myNewSite.com" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false"
alias myNewSite.com alias

MySQL service restart, InfoGlue

2009-05-24T14:23:00.000-07:00

I have MySQL running behind Infoglue for a new web app one of my clients has. We went into production on Friday with that web site. Saturday evening the hosting providing had a scheduled update and so all the boxes got rebooted. Unfortunately, the MySQL service which was configured to autorestart didn't come back up. In the application event logs (I've got it running on a Windows server) it said:

...MySQL Server 5.1\bin\mysqld: unknown option '--enable-pstack'

I searched the MySQL forums and Google for this error and nothing. I reinstalled the service a couple of times.. and nothing - same error. I was sweating by this time.

Then I started looking at the my.ini file for Mysql in the installation folder (sibling of the bin folder). I saw these lines at the very bottom:

#Print a symbolic stack trace on failure.
enable-pstack

Ah ha. So here's where that 'unknown option' was coming from. I had no idea what was making a failure happen on startup to make it print out this 'stack trace' so I just commented out that line. ...And low and behold, MySQL started up happily and things worked.

Whew.

Custom header in Cruise Control

2009-05-15T11:53:00.001-07:00

We have multiple CruiseControl (CCNET) instances running on our build boxes and we've been wondering how to differentiate which one is which so people can know at a glance which one they are pointing at.
This turns out to be relatively easy. In a typical CCNET installation, there's a \cruiseinstalldir\webdashboard\templates directory. Inside it you'll find a file called TopMenu.vm. Open the file up and after the last '#end' put some literal spaces in and type the title of your CCNET site with some plain html formatting to make it obvious (make the font bigger and bold, for example)

...save it and refresh your dashboard page.

Windows and Automated builds

2009-05-15T08:39:00.000-07:00

We've run into a couple of interesting situations with Windows with our automated build scripts that I wanted to document. Both of them have to do with inconsistencies with what the Windows GUI is saying about a configuration versus what the operating system is actually using for a configuration.
Both of these situations were discovered while we were trying to do an expansive upgrade of our code to use the new .NET framework. In the first instance we were manually trying to install/uninstall dll's from the gac on a particular box. We manually dragged and dropped a particular dll into the c:\windows\assembly folder and the GUI would say everything was successful with registering the dll in the gac. However, check it from the command line, or try to run something in the application against the newly gac'ed dll and the OS didn't seem to see it. The only way we could get the OS to successfully see the change was to gac the dll from the command line.
In the other instance (which is sort of similar in nature), our automated deploy scripts are creating a Virtual Directory underneath an IIS web site on a remote server. After the Vdir is created, we had a 'step' that automatically updated the ASP.NET version on the Virtual Directory fo 2.0.50727 (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis -s W3SVC/%WEB_SITE_NUMBER%/Root/%WebSiteName%). This script ran successfully, and the ASP.NET tab in the VDir on the web site displayed the change. However, IIS didn't actually see the change and continued to run the web site on the old 1 .1.4322 version - even though the GUI in IIS said it was. We knew this because the C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files folder didn't have the new corresponding virtual directory folder (this is where IIS keeps the compiled pages - it's like the 'working' folder in Tomcat). It seems the IIS metabase was updated, but the registry was not. I have a hunch that this might be because during that particular deploy, we left IIS running while we did the deploy. Perhaps shutting it down would make the update work properly through to the registry.

Don't assume anything

2009-05-12T10:29:00.000-07:00

We are doing a big upgrade of our components in our upcoming release. The security structure of our organization is fairly complex with users, roles, user groups assigned to roles, boxes in the domain, boxes outside of the domain, and a whole other layer of users and roles at the database level. Managing security for this release has been a hassle. It seems in every environment there are differences with who is assigned to what group, which users and which groups have 'these' permissions on 'this' folder, etc.
Thinking we had most of the issues documented, automated and swept under the rug, we deployed into our staging environment. Surprise, surprise, there were permissions issues there. The really interesting thing was these turned out to be new issues. Well, we spend the good part of the day trying to see if we missed a permission on a folder somewhere. In the end, it turned out that our architecture group (who is responsible for the shared components code and the overall enterprise architecture) had some code for applying special permissions that was hard-coded to be applied against ONLY our staging and production environments.
Shock.
I guess it goes to show, don't assume anything - no matter who is responsible for code or where the error seems to be, or how stable you think your configuration are.... We also discovered that that particular componenet had a different security setting for the COM+ in the Staging env compared to the Testing env.

InfoGlue installation

2009-05-07T06:21:00.000-07:00

I've been doing a lot of work with InfoGlue lately. I downloaded the new 2.9.6.X release and went to install it and got a 49.0 version error. It seems that the new version of InfoGlue doesn't run as good on Java 1.5 as it's documented to. It worked when I finally upgraded to 1.6. I also had issues getting it to work with MySql because my install of MySql was done with a production box in mind. It seems that the MySql configuration engine clamps down on security when you specify this, and as a result, I had to allow temporary access for root to all schematas, and then have to specify permissions for each user by schemata. While I'm very impressed and happy about this, it made for a little more grief than I was expecting in the installation process.
Also in the InfoGlue installation process, you do want to change the first username/password it asks you for to be the specific one that InfoGlue will use to call the DB with. Not realizing that, I had to go through all my web contexts and change the references to those properties.
Also, deleting a repo in InfoGlue using it's 'force' feature is not a great idea. Better to rename it instead. Sigh. I had to go through and insert data back into a couple of tables to get the repo working properly again.

IIS debugging

2009-04-07T06:35:00.001-07:00

Been working lately on getting a new IIS web application into production. We've been continuing to run into some more issues with anonymous/integrated windows authentication as the build is progressing through different environments. However, I found this cool tool yesterday that has some neat utilities for working/debugging IIS. It's called the Internet Information Services (IIS) Resource Kit Tools and you can download it here. Some of the cool utilities in that package are:

WFetch - allows you to fully customize an HTTP request and send it to a Web server so that you can see the raw HTTP request and response data. Sort of like Ethereal for IIS.
SelfSSL - allow you to create and sign you own ssl certificates for testing with ssl in dev environments
A Metabase Explorer - allows you to view all the configuration properties IIS and it's web sites, app pools, virtual directories, and more
And an Apache to IIS migration tool which would apparently allow me to migrate an apache web server (and it's configs) to an IIS6.0 server.

IIS and Nagios

2009-03-27T15:22:00.000-07:00

Working with IIS on a particular app, we had a situation where the behavior of the web server seemed to be inconsistent with how we had configured it. We had configured it to allow Authenticated access based on Integrated Windows authentication only, no anonymous access. When we browsed to the page we wanted to view on the web server itself, the page came up fine. But when we tried to browse to the page from another box, we were required to authenticate. This was not expected since it should have been using the Integrated Windows authentication to log in transparently.
After comparing it with another server that worked and seeing that most of the GUI configuration looked identical, I started googling and found this article that answered the problem for the most part. The only difference we found was even the order of the different NTAuthentication providers matters - the working server had "NTLM,Negotiate"; the server that didn't work was set up as "Negotiate,NTLM". Changing the broken server to look like the other server fixed our problem.

Our infrastructure teams has been doing some changes and moved the configurations for the email servers a bit. Unfortunately, this wasn't transparent to my Nagios/GroundworkOpensource installation (since I was lazy and hadn't reconfigured the email 'from' address for host). What ended up happening was all the Nagios emails ended up getting sent outside the internal network (because the domain name was still 'localhost.localdomain') and getting caught by the public antispamm appliance we have. I ended up switching notifications to use 'service-notify-by-sendemail' instead of 'service-notify-by-email' and then overriding the host in the command to point to our local email server. And that worked. Similar to this post on the groundwork forum.

Automatically deploying IIS web apps

2009-03-18T15:24:00.000-07:00

We've been working on automated builds and deploys at work. All of our web apps run on IIS and configuring 'good' automated deploys for these applications has been challenging. But I think we're seeing the light at the end of the tunnel now.

Microsoft provides a good number of support scripts (in vbs) to configure IIS, App Pools, and Virtual Directories from the command line. Some of them come with the IIS installation, some of them we had to download - if I remember correctly.

There are two 'sets' of these CLI vbs scripts that I'm aware of that work with IIS. One is found in the ../inetpub/adminscript folder. There's a bunch of vbs scripts in there. The other set is found in c:/windows/system32/iis*

Here's some examples of what you can do with these scripts:

create a virtual directory: cscript c:\windows\system32\iisvdir.vbs /create webSiteName virDirName virDirPath /s serverName
set perms on a virtual directory: c:\inetpub\adminscripts\adsutil.vbs SET w3svc/webSiteNumber/Root/virDirName/AccessFlags permNum (like 513)
set ASP.NET version for the virtual Directory c:\windows\Microsoft.net\framework\v2.0.50727\aspnet_regiis -s w3svc/webSiteNumber/root/virDirName
and tons more...

You can view all the website numbers in the IIS manager. They show up in a column.

IT Security

2009-03-16T20:06:00.000-07:00

Over the past few years I've seen sensitive information exposed in some very interesting places on enterprise networks and servers. Sometimes this leftover information can be super helpful if you're trying to debug problems or get an idea of what happened on the box in the past. In other cases, it just plain bad. Here's some of what I've seen:

Shared drives mapped all over the enterprise. Shared drives mapped on production boxes with access to files that contain sensitive info like passwords for production users.
Kickstart configuration files with username and passwords for domain users in clear text forgotten on servers
Passwords and sensitive information exposed in .bash_history files. Bash_History files are a treasure trove of information. They'll show you all kinds of things - where the db server is located, what the connection string is, where http servers are installed, how to shut them down and start them up....etc.
*.udl files - Microsoft specific. They store connection information in clear text for db servers. Don't leave them lying around and exposed.
Installations for UPS (Universal Power Supply) systems left with their default configured administrator username and password. I happened to find a login page for a UPS console one day and logged in on the first try using the first password I could think of. The dashboard I subsequently found myself on gave me the power to shut down the entire enterprise.

Here's some simple ways to make your network/enterprise more secure:

Don't allow a plethora of undocumented mapped drives.
Do searches for text like 'password' on any boxes, drives, etc that you might be concerned about. If you get results, take steps to either encrypt or delete those files or references.
Change default installation passwords

CMS experiences

2009-03-15T20:26:00.000-07:00

In the past year or so I have gotten a bit of experience with different CMS systems. I was given two clients(www.mross.com, www.auma.ca) that run on InfoGlue - a java/velocity based cms system. I am also messing around with Joomla, and I have one client (www.brooks.ca) that runs on it. I have yet to do anything with Droopal, but from what I've heard, it sounds more like InfoGlue than Joomla - that is, it's more geared towards an enterprise/portal centric CMS than what Joomla appears to be.

InfoGlue is very configurable and supports internationalization. This makes is somewhat cumbersome to configure to start out with. I've found that the important files to know about when doing a configuration on the fly are:
- WEB-INF/classes/cms.properties
- WEB-INF/classes/hibernate.properties
- conf/localhost/live.properties, etc

Some issues that I've noticed with the InfoGlue instances that I work on are:
- on logging into the CMS, I have to refresh the page 3 times before I acutally get the GUI. Only one of my instances does this, so I think it's a configuration thing.
- sometimes Tomcat seems to get it's knickers in a knot and needs to be rebooted in order for content changes to be saved. This happens periodically.
- deleting content (cleaning things up) can be a huge pain - especially if there are a lot of references to the content in other parts of the site. You end up having to delete the references first before you can delete the content.

So far I've been pretty impressed with Joomla. I was able to figure out how things were put together fairly quickly. I bought this book which helped some - it has some good chapters on SEO and working with Joomla templates. I was able to manage a complex upgrade to the site I maintain within a week of getting bootstrapped on the Joomla CMS. This impressed me.

Visual Build

2009-03-10T11:38:00.000-07:00

I've been working with Visual Build in a .NET environment off and on for almost the past year. When we decided to go forward with Visual Build, I thought it would be a more painful process than it turned out to be. Our technology stack includes ClearCase, Subversion, Ms SQL, PSExec, VMWare, VB.NET, IIS, CruiseControl, Nant, Nunit, and a bunch of 3rd party tools and servers related to document management.
Visual Build provides a bunch of example build files that are quite helpful in getting all kinds of different functionality going. We automatically compile, unittest, deploy (to remote boxes), and run verification tests using Visual Build. There a numerous other interesting smaller tasks that we've got Visual Build to manage for us like setting perms on remote boxes, stopping and starting servers, services and com+ objects, performing baselines, checkins/outs, and updates of clearcase streams and views, etc.
Some of the gotchas we've discovered in our work with Visual Build:
- managing windows (child build files that get spawned in a complex build process) and logs is not so trivial. If they aren't managed correctly, your build will stop without much notification as to why. We found that piping the output from the build to a separate file worked for us with logging. With windows, we found that matching the build context, waiting for completion, running the GUI app in silent mode, and not closing the GUI app on failure were what made things tick.
- psexec needs to be on the path on the box you're running Visual Build on.
- CruiseControl integration was relatively easy - we just used the command line with options:
<tasks>
<executable>C:\PathToVisualBuildInstallation\VisBuildCmd.exe</executable>
<baseDirectory>D:\pathToBuildFile</baseDirectory>
<buildArgs>/b "XXX.bld" -other args to pass to buildfile go here</buildArgs>
<buildTimeoutSeconds>1000</buildTimeoutSeconds>
</tasks>

Replatforming an app in 21 hours....

2009-03-06T12:40:00.000-08:00

I got a cold call a couple of weeks ago. A prominent institution in Calgary had a java application that ran on a Sun OS with an iPlanet web server that was key to their business that didn't work with a key upgrade from a 3rd party vendor. It needed to be upgraded an running in production 2 weeks. Could I help?
I agreed to help and showed up that afternoon to see what could be done. The current code in production ran fine. However, there was no guarantee it would run fine with the upgrade. Unfortunately, the code base was a mess. Multiple versions of the same file were all over the production server with .bak, .ver2, etc extension. Everything had been done on the fly in the past. The logs couldn't provide us with any useful information, and they had no test environment.
After seeing what could be done on the Sun box (which had been running this application non-stop for more than 10 years!), I decided to try and port the application to Tomcat - in this container I knew what I was doing and I could debug the problems more effectively. Within 14 hours we had the application running on Tomcat with some minor bugs.
Over the weekend, new production and test virtual machines were requisitioned and when I came back on my next visit, we got the last of the bugs out of the way and had everything working in the new test environment.
Some lessons learned....
- Don't be afraid to replatform in Java. I would've had a much harder time trying to do this with Microsoft technology. The clients were VERY happy to get off that old server. No one was around to support it anymore...etc. That was 10 year old java code that I didn't have to do a thing to when I moved it onto j2sdk1.4.2_XX.
- It's almost scary how many different ways you can configure things in Tomcat. We had outstanding issue that was 'bugging' me after 14 hours of work. The client required the application to be available from two separate url paths without using an apache web server to do url rewriting. I tried various different configurations in the app's web.xml, the global web.xml, the server.xml and found that I could get the url's to work, but then I had problems with sessions getting lost/corrupted. In the end, we wrote another servlet (sort of like this, but different) that we configured to 'catch' the url we needed it to and redirect the request to the controller servlet that was already mapped to the other url. This worked beautifully.
- We discovered that if your dealing with pages that have a lot of scriptlet code that does response.sendRedirect(someUrl), these method calls will sometimes throw IllegalState exceptions. The (quick) way to get around throwing the exception is to add a return; statement right after it. Or you could put that kind of logic into a servlet if you have time. I didn't (have time).
- Having good logging is critical to debugging an app in an emergency. Seeing my System.out.println() outputs in the tomcat console was like drinking hot chocolate after a day of skiing. So RIGHT! Of course I commented out those lines before we put the code into production.
- Using technologies that don't required registry keys makes configuration and multiple environment installations very easy. I found that I could write two short paragraphs of directions (documentation?) and that was enough for my client to install java and Tomcat into their new test environment by himself. (Pretty much copy - paste, and add the JAVA_HOME system variable and add the j2sdk...\bin onto the Path system variable). They were very happy about that too.
- Having hard coded path references to properties files in java code is so nasty! Every time we changed the path, we had to recompile all our classes. I know that's not the right way to do things, but the client just wanted to get things working and worry about refactoring later.

Groundwork/Nagios and Rockstars

2009-03-05T07:17:00.000-08:00

I've implemented Nagios at a few different places. In my current position, I've implemented Nagios with Groundwork, and I'm really happy with it. Groundwork provides a 'community' (free) version of their tool on a VM imaged instance that you can just plunk into a VM client. We use the npre_nt agents to monitor specific services and metrics on a number of windows servers. I reconfigured some of the vbscripts in the remote agents to call different methods in the WMI api which I've detailed in this forum. I've also customized the check_mssql to call specific tables looking for issues in production data (querying for held up business processes and corrupted documents). In order to do this, I had to download freetds and install it on the VM, overriding the default installation target with /usr/local/groundwork/. Then the freetds.conf file gets put into /usr/local/groundwork/etc and you can configure the servers you want to call in there. I put two configs for each server I wanted to call in that .conf file as I wanted to be able to test from the command line. I found that groundwork ends up calling using the IP of the box - so each server conf has a proper domain name config, and a IP name config.
One other thing about configuring the Groundwork/Nagios installation - I had to configure the CentOS to point to our exchange server so the email notifications would work.

I've read a couple of of good tech books lately (I now own them both). Release It! by Michael Nygard, and Secrets of Rockstar Programmers by Ed Burns.

Schema Crawler

2009-03-04T13:03:00.000-08:00

I got back to using schema crawler this week, trying to make sure we can explain the inconsistencies we see in the metadata between our different environments. I found this cool little script to do port scanning so I could discover which ports our DB's were using.

HOST=127.0.0.1;
for((port=1;port<=65535;++port)); do echo -en "$port \n"; if echo -en "open $HOST $port\nlogout\quit" | telnet 2>/dev/null | grep 'Connected to' > /dev/null;
then echo -en "\n\nport $port/tcp is open\n\n";
fi;
done

Tibco and Certificates

2007-10-17T18:50:00.000-07:00

I've been messing around for the past two weeks with certificate chains, different certificate libraries, and keystores in Tibco with a third party provider. Man, what a mess. To come into a project where nothing is documented about which certs are used and have more than half a dozen of them (legacy) littered here there and everywhere around the module, along with another 7 or so new ones isn't happiness.
Interesting things I've learned through this experience are:

It turns out that there are compatibility issues with .p12 certificates between java 1.4 and java 1.5. I had a pkcs12 certificate that I could read and sign fine in java 1.5, and yet java 1.4 could not find the alias in the same certificate at all. Apparently, there are a couple of ways around this issue - I only got one of them to work. One way is to convert the .p12 certificate into a .pem cert, and then convert it back to .p12 (all of this with openssl). The .p12 certificate is supposed to be readable after the conversion from .pem, but that didn't work for me. I got some kind of invalid tag error. What did work for me was to use openssl to convert the .p12 cert into a .pem cert and change my java 1.4 code to deal with the .pem file. To do this I had to use a couple of jars in the globus library. I remember something about a BouncyCastle class. :-) Also, here's a link I found helpful to discover what kind of certs you are actually dealing with.
Getting those new external jars to work in Tibco Designer's Tester was a bit of a trick. First I added them in tibco's jre/1.4.2/lib/ext directory. That enabled me to compile the java code in the palette java item. But when I went to run the tester, it said I had code I needed to compile !?! I then I found I also had to add them into the path property in the tibco/designer/5.2/bin/designer.tra file so that Tibco's Tester would compile the code.
Trying to load the module into production, I ran into two exceptions in the Tibco adminstrator tool. One was an OutOfMemoryError. This was resolved by doubling the java heap property found in the tibco/administrator/5.2/bin/tibcoadmin-.tra file, and also doubling the values for the min/max heap size for the module, found under the module itself in the administrator GUI. The second exception was a InputStream cut off IO error. This was because my module was too big. I was referencing java libraries that we already had referenced in a 'SharedResources' module we have. Getting rid of those drastically cut down the size of my module and it loaded fine after that.

Looking for a Pragmatic Development Process

2007-10-05T21:32:00.000-07:00

Consider for a moment, if you will, the delicate balance between robust, thoroughly quality assured code and the amount of time it takes to put that code into production. I ran hard into the brick wall of 'development process' this week - an obstacle numerous people in our business had been complaining about, but I'm not sure how many people on the IT side of our organization understood.

My story went like this. We have some feedback from one of our applications that gets displayed in another in office application that was getting truncated. This truncated information was deemed valuable by the SME's and so a fix was requested. I looked into the fix, and it turned out to be a one character change in a cold fusion file. Getting this change implemented in production would not even require a server restart. After getting the changed entered in our bug ticket system and socializing the change with the appropriate stakeholders, it seems that our development process requires that I wait a full month before my change is actually implemented in production... and we're (supposedly) an Agile shop. What (on earth) is wrong with this picture?

This isn't the first time the organization has run into this issue. There have been other similar issues in the past - an email address needed to be changed. Could you please wait 2 weeks? If fact, this has turned into a major issue with upper level management (read execs), to the point that an entire system rewrite is being considered using a totally different technology with the main purpose being to cut in half (or more) the time it takes to get from requirements to production.

Personally, I do not think that the business and management is going to find their silver bullet in implementing a different technology for development. I think the problem lies in our development process being too rigid. 'We have one release a month - your changes go in that release and if you miss getting your change in by the code complete date, too bad'. Different technology will expose other problems. For example, a lack of appropriate hooks (or a complete absence thereof) for proper unit testing. Or, an inability to version and automate the configuration and deployment of the new technology stack.

Save a few million bucks on 'new technology' (it would really only be new to the company). Seems to me that it wouldn't be that difficult to set up a small, very part time team of a senior developer, a senior qa analyst, a senior ba, and an rm dude to assess these 'changes' that don't fit into agile's bug definition, but should be made in production quickly. If the change is deemed relatively innocuous by the team, put on your cowboy hat's and let's have five minutes of fun. Maybe that's going a bit too far - you want to capture the change in your build system. Communicate the risk to your client (very likely the same person or close to them) whose asking for the change, and then make them happy.

Please, don't make them wait a month for their change.

Stealing the Network

2007-09-24T19:35:00.000-07:00

Last week a friend at work handed me a book to check out. It was called Stealing the Network. Boy did I enjoy it. It's basically a collection of 'fictional' hacker/security consultant stories. The stories cover the whole gammit of hacking, from social engineering to deconstructing worms, to stealing a 0-day exploit. I found it super interesting.

In their stories, the authors go into detail with the kinds of tools they use. Netcat, Snort, John the Ripper, and Ethereal all make an appearance, as well as many others. It's interesting to compare the list of tools used in this book with the list at sectools.org.

I'd highly recommend the book if you're familiar with any of these tools or have an interest in secure networks.

A Different Perspective...

2007-09-13T18:38:00.000-07:00

I've been working with Tibco for the last two months. Before that I was involved in a significant development effort building an automated build system for a large java development team. What a difference in projects.

Before this gig, I didn't have a lot of experience with big SOA (Service Oriented Architecture) solutions like Tibco. Approaching Tibco's 'solution' from a java perspective is a bit of a shock. Why? Isn't Tibco coded in Java? It certainly looks like it, however it's end users (I don't think) were supposed to be experience enterprise Java developers. It appears to be geared more towards business analysts and 'light' coders. ...Grab this component, drop it there, draw an arrow to it from that component.. etc. I was somewhat aware of that going into the project.

Probably the biggest surprise and 'Aha!' for me is Tibco's business model. What I see is Tibco markets their product to the big decision makers (read: people who sign the cheques). Generally, these people have a high level view of most things, including the development process and what actually happens behind the GUI of an application. Tibco very successfully markets their products to these people, telling them that Tibco applications will 'take your system from greedy (as in a money sink) to speedy'. It looks great on paper and running in a demonstration. It looks like a Godsend to businesses that are fed up with waiting a month for the release cycle to finish so they can get their little text change posted in production.

Who knows.. it might be. It seems to come down to who really owns the business. Who calls the shots about what the business direction is and what should be put into production? Who's responsible if revenue doesn't meet expected targets? Should the business be limited and reigned in by IT's processes because 'everything needs to be unit tested, smoke tested, functionally tested, regression tested, performance tested, and then tested in a staging environment'? Sometimes it's fun to wear a cowboy hat.

The Tibco tool we work with doesn't seem to have much in the way of a testing harness... at all. Please feel free to set me straight on this - I want to know how to do some QA against a Tibco app. Also, our Tibco application makes close to 2 dozen separate calls (between itself and over the wire with a significant amount of data) for 1 single request - it's VERY chatty. Apparently, Tibco's solution to increase performance and make chatty applications run faster... throw more hardware at it.

For me the jury is out. I don't have enough experience yet to make an informed decision either way: Java enterprise or Tibco SOA. You want granular control of how your application works at the lowest level and you don't need to turn production on a dime - I'd suggest Java. If you want quick turnaround and a business oriented IT application, and you're not concerned about what's happening underneath the covers, maybe Tibco is the option for you.

Flash boot, more cron, and caching.

2007-09-12T18:47:00.000-07:00

I learned an interesting thing today. Depending on the bios of your system, you can boot your box from a flash drive (memory stick). The flash drive needs to be in the usb port when you start your box up though, otherwise the bios won't know that's it's available.

I learned one more thing about cron today. To get cron to run once daily, don't do this: '* 0 *** /opt/myscript.sh'. What will happen is cron will wait until midnight to run, and then it'll run every minute because you've used a '*' instead of a '0' in the minutes space. The correct format is '0 0 *** /opt/myscript.sh'.

I've pretty much finished up my little project with FusionCharts. However, I've noticed there a bit of an issue with caching. It's actually not very consistent. I can have 5 charts displaying 5 different data sets on one page, and 4 of them will update with new data, one of them will absolutely refuse to until I shut down the browser and open a new one. (I'm working with Firefox, by the way).

Firefox seems to really be in love with pages it's seen before. I haven't just seen this issue with FusionCharts. Some time ago we ran into a situation at work where clients were actually having to log in twice because the first login seemed to be ignored. What was happening was Firefox was overwriting the cookie for the login page with the favicon cookie. We had to examine every packet for each request to figure that one out. We used an obscure setting in the ajp connector in tomcat's server.xml file to resolve that one. 'emptySession=true' or something like that - I can't remember it off the top of my head right now - it was a couple of months ago.

Oracle & perl

2007-09-11T18:18:00.000-07:00

Continuing to work with SqlPlus and Oracle 10g. Today I resolved two issues I had with an sql query.

With the first problem, I was passing two parameters into the .sql file and when the sql was executed in the file, along with the selected information, it would output 'old ... my sql here with the params not replaced' and then 'new... my sql here with the params replaced'. Thanks for that, but I know which params I'm replace, I don't need more notification of it. It turns out getting rid of this notification is fairly easy. At the top of the script, just put 'set verify off' and no more 'old...new' junk.

In the second problem (same script) I was doing some division and there were instances where my divisor could be (and was) 0. Sql doesn't like to divide by 0 at all (like pretty much every other language I've worked with). There is a neat little fix to get around this as well. My divisor looked like 'select count(a.blah) from ....' and if you add 'select decode(count(a.blah),0,1,count(a.blah)) from ...' it will substitute a 1 if the selected count is a 0. It worked super.

Messing around with Perl some more, I discovered some good functions. @myArray = split(/~/,$var); will take make an array of strings, seperating them by every '~' found in the string. Working with arrays, push(@myArray, someValue) pushes a new value onto 'myArray'. Pretty basic stuff, I know. I like to celebrate small victories (and remember them for next time).

Can you believe it, I hit a dog riding my bike on the way home from work today. A big doberman. He ran right infront of me. I slammed on my brakes and my rear end came up. I didn't hit him hard - but enough to put my handle bars out of alignment with my front tire a bit. I fixed that at the first red light I stopped at. I also just about hit a snake warming himself on the pavement.

Random assimilations and Business ideas

2007-09-07T22:01:00.000-07:00

Today I learned that you can pass in parameters to .sql scripts that are destined to be run with sqlplus. It looks like this: 'sqlplus -s dbuser/dbpasswd@dbname @scriptname.sql param1 param2' and then in the sql file you output the params using &1 and &2. I was finding though that for the replacement sqlplus was being verbose and telling me about every replacement it made. I didn't like that so much, but couldn't figure out how to turn it off. I've already got 'set headers off' and 'set feedback off' set at the beginning of the script. I tried 'set logging off' but it didn't like that (because logging is set to off as a default I believe). Anyhow, the replacements were working.

With Oracle SQL, if you want to select back by a time interval, do something like this: 'select count(*) from my_log_table where logdatetime > current_date - interval '15' minute;' or you can use '- interval '6' hour' ... or '- interval '2' day' etc. Single quotes around the number!

Back to cron: to set cron to run every minute go '*/1 * * * * /scriptToBeRun.sh' Every 3 minutes: '*/3 * * * * /script2BRun.sh'

One problem I've run into with FusionCharts is the limitation of the Flash plugin to handle a big data dump. Put more than a few graphs with, say, 50 datapoints each on a page, and the browser hangs and the plugin throws our of memory errors.

Interesting IT Business Ideas:
- Reusing software development tools for other industries/applications. For example: other than the obvious issue with binary file types, why don't architects and lawyers use a repository like SVN, or Monotone? ( I left out the ones you have to pay for :-) We've tossed this idea around wondering if anybody has tried to market repositories this way....
- Becoming a freeware software consultant for small businesses. You can get everything from a good OS to all the 'openOffice' applications for free, not to mention what Google and others are putting on line (word processors and spreadsheets, etc), and universally accessible too! Seems to me if you're good at training, you can really help small businesses save a buck on software by showing them how to use all the free stuff out there. It wouldn't surprise me if people are doing this one already.
- write a GOOD, scalable, secure, extensible front end framework for web applications. There's a tonne of them out there now, but it seems like every one of them (Ruby, JSF, Struts, Facelets, Tapestry, SpringMVC, etc) has a downside. I've had conversations with a number of knowledgeable architect types and they agree that there is no good front end web solution out there right now. So, one could code a good freeware framework, consult using it, and then sell it to Red Hat (like the JBoss consultants did).

Got Something done today!

2007-09-05T19:05:00.000-07:00

...or at least, it felt like it. After doing housekeeping with our production application yesterday at work, today was a 'sit down and code' day. Boy did it feel good. I had told myself that I had to get this charting solution finished today or it would turn into a waste of time. It turns out that FusionCharts are super easy to implement, and boy, do they look sharp.

In my 'solution', I'm using Nagios to provide the data, with its 'service-perfdata' process. That file gets big quickly. I just enabled it in the Nagios configuration on Friday last week, and I already had 12 Megs of data. Going through the Nagios documentation and forums, there doesn't appear to be any mechanism to 'roll' this file or archive it or buffer it.... It had me stumped for a while. Then I realized I could just do a simple 'tail -1000 service-perfdata.out > service-perfdata.out' in a shell script with a cron job, and presto - a simple fix.

So, mental note to self.... whenever you're coding a shell script (or any kind of script) that gets run by cron, make all the file paths in the script(s) absolute. I keep on running into this problem. Cron is ignorant about everybody's profile and position.

I just started coding with Perl this summer on this project, and I really like it. Working with arrays is tricky and not very intuitive. Otherwise, though, Perl is fantastic for whipping up something quick. Today I found out that you can use && and || in conditional 'if' statements. That was good. Also, something I've noticed in the past and ran into again today - the 'chop' and 'chomp' functions in perl don't seem to work like the documentation says they should. I ended up having to something like this to get rid of spaces '$var =~ s| ||g' - basically using sed (or the like) to get rid of spaces I didn't want. Perl's localtime() function to convert unix to to something readable worked slick for me today. Another thing I ran into not too long ago - the proper way to concatenate strings in Perl is with '.'

Rode my bike into work today. Just had the pedal axle fixed, and wow, what a difference. The aphids are all over the place right now. After a ride they are just plastered all over me. It's hard to not breathe them in while one rides.

Assumptions and Critical Thinking

2007-08-31T21:46:00.000-07:00

A couple of weeks ago I programmed a DB load process that ran off a cron job. It ftp-ed data files from a public box to a local one, and then loaded the data based on .dat files that were extracted from the download, and .CTL files already on hand. I assumed this process worked correctly because every time I ran it manually it worked, and when I started it using cron, it 'looked' like it workded. It turned out that when it ran off the cron, it didn't do everything I wanted it to do. I set about debugging this today, and it turned out that my problem was very similar to my PLSQL problem from earlier this week.

Part of the Load process used a java program to created create the .dat files. The java worked fine when I ran it manually because the user I was running it with had JAVA_HOME set in it's profile. Cron runs differently. Even though my cron job su's to the proper user, doing the same thing manually and then checking with a 'who am i' command (which, by the way, gets different results than a 'whoami' command) returns a surprise. It's not my su'ed user. Adding JAVA_HOME into the script that gets run solved the problem.

Been looking at graphing solutions for a little thing I'm working on. GDchart seems usable. I might do some more work with it on Tuesday next week. FusionCharts seems very interesting as well. I've worked with Cewolf and JFreeCharts in the past, but I need something quicker right now.

Picked up a book called Th!nk off the discount shelf at Cole's today. I'm liking it so far. It's something of a rebuttal to another book called Blink!.

Nagios and things part 2

2007-08-30T20:09:00.000-07:00

So I got to work this morning (with a fresh mind) and after a bit of googling figured out my problem in 30 minutes. Basically it boiled down to adding a reference to ORACLE_HOME in the bash script itself, and the full path of SQLPlus. After that, things ran fine even with changing the permissions of the scripts and related files back to normal and running Nagios with the Nagios user.

One of the 'nice to haves' for this little monitoring things I'm working on in Nagios is graphing capability - so we can see trends of the performance data at a glance. Just before I left today, I discovered that Nagios might install with this capability commented out. I let you know for sure in my next post.

While I was working on solving the graphing problem a different way, I got focused on some graphing addins for Nagios that required perl modules I didn't have. I learned that cspan is basically like 'yum' for perl. Didn't know what it was before today. In fact, I haven't had any exposure to perl before this project. It's all good - learning new stuff every day.

I also learned a big way to correct my golf swing today. I need to grip hard with my left hand and the right hand is just for support. I was driving much better and more consistently after I made that adjustment.

Nagios and things

2007-08-29T22:07:00.000-07:00

I've done a couple of installations of Nagios now, and I quite like as a monitoring tool. Today I was working on coding a custom plug-in so we could keep an eye on a particular KPI. This involved running something of a complex query in Oracle and then sending the results to Nagios.
Writing the shell script wasn't difficult. The Nagios plug-in developer documentation was clear enough for me to follow. It took me a few minutes to figure out how to return the codes that Nagios is looking for - 'exit 1, or exit 2, etc'. However, to try and get Nagios to automatically run SQLPlus is a totally bear. I'm going to sleep on it tonite. I've tried changing perms on all the files, even tried changing the ownership to oracle and setting the SUID - no dice. Tried running Nagios as root (not recommended, but I had to try), and set all the Oracle environment variables in the .bash_profile for all the users I could think of that might possibly be running that code. I tried doing a su -c 'call my script here' oracle, and that didn't work either.

Oracle posts online regarding SQL*Plus 43 initiation errors etc. are pretty sparse. In my opinion, it's just more fodder for people who advocate using 3rd party free ware. At least if it's broken, somebody has likely run into the same issue and posted it online. With Oracle and Tibco, it seems like they guard their support forums with their life (maybe Oracle not so much so).

Oracle, Tibco, and tcpdump

2007-08-27T22:11:00.000-07:00

I've started a new project recently that uses Tibco with an Oracle backend. It still amazes me the little nuances of domain knowledge that are so key to being a productive developer on some of these 'platforms'.
My issue today was a query I was trying to run using the jdbc 'component' in Tibco. It's pretty finicky about SQL syntax. I needed aliases for column names and it turned out the only way to get those to work is to use double quotes. Single quotes work if you want to hardcode what you're returning in your select statement. Double quotes (I was told) are effectively ignored by Oracle unless you're trying to alias a column. I'm starting this blog mainly so I can keep these little nuances documented somewhere - I hope it'll help me remember in the future.

I ran into a different configuration problem with Tibco last week which I had to use tcpdump to help me solve. I used the command something like this: tcpdump -nnXi eth0 -port 8080. This helped me realized that Tibco was responding to http GET and not http POST requests.