Tibco and Certificates

I've been messing around for the past two weeks with certificate chains, different certificate libraries, and keystores in Tibco with a third party provider. Man, what a mess. To come into a project where nothing is documented about which certs are used and have more than half a dozen of them (legacy) littered here there and everywhere around the module, along with another 7 or so new ones isn't happiness.
Interesting things I've learned through this experience are:

• It turns out that there are compatibility issues with .p12 certificates between java 1.4 and java 1.5. I had a pkcs12 certificate that I could read and sign fine in java 1.5, and yet java 1.4 could not find the alias in the same certificate at all. Apparently, there are a couple of ways around this issue - I only got one of them to work. One way is to convert the .p12 certificate into a .pem cert, and then convert it back to .p12 (all of this with openssl). The .p12 certificate is supposed to be readable after the conversion from .pem, but that didn't work for me. I got some kind of invalid tag error. What did work for me was to use openssl to convert the .p12 cert into a .pem cert and change my java 1.4 code to deal with the .pem file. To do this I had to use a couple of jars in the globus library. I remember something about a BouncyCastle class. :-) Also, here's a link I found helpful to discover what kind of certs you are actually dealing with.
• Getting those new external jars to work in Tibco Designer's Tester was a bit of a trick. First I added them in tibco's jre/1.4.2/lib/ext directory. That enabled me to compile the java code in the palette java item. But when I went to run the tester, it said I had code I needed to compile !?! I then I found I also had to add them into the path property in the tibco/designer/5.2/bin/designer.tra file so that Tibco's Tester would compile the code.
  
• Trying to load the module into production, I ran into two exceptions in the Tibco adminstrator tool. One was an OutOfMemoryError. This was resolved by doubling the java heap property found in the tibco/administrator/5.2/bin/tibcoadmin-.tra file, and also doubling the values for the min/max heap size for the module, found under the module itself in the administrator GUI. The second exception was a InputStream cut off IO error. This was because my module was too big. I was referencing java libraries that we already had referenced in a 'SharedResources' module we have. Getting rid of those drastically cut down the size of my module and it loaded fine after that.
  

Looking for a Pragmatic Development Process

Consider for a moment, if you will, the delicate balance between robust, thoroughly quality assured code and the amount of time it takes to put that code into production. I ran hard into the brick wall of 'development process' this week - an obstacle numerous people in our business had been complaining about, but I'm not sure how many people on the IT side of our organization understood.

My story went like this. We have some feedback from one of our applications that gets displayed in another in office application that was getting truncated. This truncated information was deemed valuable by the SME's and so a fix was requested. I looked into the fix, and it turned out to be a one character change in a cold fusion file. Getting this change implemented in production would not even require a server restart. After getting the changed entered in our bug ticket system and socializing the change with the appropriate stakeholders, it seems that our development process requires that I wait a full month before my change is actually implemented in production... and we're (supposedly) an Agile shop. What (on earth) is wrong with this picture?

This isn't the first time the organization has run into this issue. There have been other similar issues in the past - an email address needed to be changed. Could you please wait 2 weeks? If fact, this has turned into a major issue with upper level management (read execs), to the point that an entire system rewrite is being considered using a totally different technology with the main purpose being to cut in half (or more) the time it takes to get from requirements to production.

Personally, I do not think that the business and management is going to find their silver bullet in implementing a different technology for development. I think the problem lies in our development process being too rigid. 'We have one release a month - your changes go in that release and if you miss getting your change in by the code complete date, too bad'. Different technology will expose other problems. For example, a lack of appropriate hooks (or a complete absence thereof) for proper unit testing. Or, an inability to version and automate the configuration and deployment of the new technology stack.

Save a few million bucks on 'new technology' (it would really only be new to the company). Seems to me that it wouldn't be that difficult to set up a small, very part time team of a senior developer, a senior qa analyst, a senior ba, and an rm dude to assess these 'changes' that don't fit into agile's bug definition, but should be made in production quickly. If the change is deemed relatively innocuous by the team, put on your cowboy hat's and let's have five minutes of fun. Maybe that's going a bit too far - you want to capture the change in your build system. Communicate the risk to your client (very likely the same person or close to them) whose asking for the change, and then make them happy.

Please, don't make them wait a month for their change.

Stealing the Network

Last week a friend at work handed me a book to check out. It was called Stealing the Network. Boy did I enjoy it. It's basically a collection of 'fictional' hacker/security consultant stories. The stories cover the whole gammit of hacking, from social engineering to deconstructing worms, to stealing a 0-day exploit. I found it super interesting.

In their stories, the authors go into detail with the kinds of tools they use. Netcat, Snort, John the Ripper, and Ethereal all make an appearance, as well as many others. It's interesting to compare the list of tools used in this book with the list at sectools.org.

I'd highly recommend the book if you're familiar with any of these tools or have an interest in secure networks.

A Different Perspective...

I've been working with Tibco for the last two months. Before that I was involved in a significant development effort building an automated build system for a large java development team. What a difference in projects.

Before this gig, I didn't have a lot of experience with big SOA (Service Oriented Architecture) solutions like Tibco. Approaching Tibco's 'solution' from a java perspective is a bit of a shock. Why? Isn't Tibco coded in Java? It certainly looks like it, however it's end users (I don't think) were supposed to be experience enterprise Java developers. It appears to be geared more towards business analysts and 'light' coders. ...Grab this component, drop it there, draw an arrow to it from that component.. etc. I was somewhat aware of that going into the project.

Probably the biggest surprise and 'Aha!' for me is Tibco's business model. What I see is Tibco markets their product to the big decision makers (read: people who sign the cheques). Generally, these people have a high level view of most things, including the development process and what actually happens behind the GUI of an application. Tibco very successfully markets their products to these people, telling them that Tibco applications will 'take your system from greedy (as in a money sink) to speedy'. It looks great on paper and running in a demonstration. It looks like a Godsend to businesses that are fed up with waiting a month for the release cycle to finish so they can get their little text change posted in production.

Who knows.. it might be. It seems to come down to who really owns the business. Who calls the shots about what the business direction is and what should be put into production? Who's responsible if revenue doesn't meet expected targets? Should the business be limited and reigned in by IT's processes because 'everything needs to be unit tested, smoke tested, functionally tested, regression tested, performance tested, and then tested in a staging environment'? Sometimes it's fun to wear a cowboy hat.

The Tibco tool we work with doesn't seem to have much in the way of a testing harness... at all. Please feel free to set me straight on this - I want to know how to do some QA against a Tibco app. Also, our Tibco application makes close to 2 dozen separate calls (between itself and over the wire with a significant amount of data) for 1 single request - it's VERY chatty. Apparently, Tibco's solution to increase performance and make chatty applications run faster... throw more hardware at it.

For me the jury is out. I don't have enough experience yet to make an informed decision either way: Java enterprise or Tibco SOA. You want granular control of how your application works at the lowest level and you don't need to turn production on a dime - I'd suggest Java. If you want quick turnaround and a business oriented IT application, and you're not concerned about what's happening underneath the covers, maybe Tibco is the option for you.

Flash boot, more cron, and caching.

I learned an interesting thing today. Depending on the bios of your system, you can boot your box from a flash drive (memory stick). The flash drive needs to be in the usb port when you start your box up though, otherwise the bios won't know that's it's available.

I learned one more thing about cron today. To get cron to run once daily, don't do this: '* 0 *** /opt/myscript.sh'. What will happen is cron will wait until midnight to run, and then it'll run every minute because you've used a '*' instead of a '0' in the minutes space. The correct format is '0 0 *** /opt/myscript.sh'.

I've pretty much finished up my little project with FusionCharts. However, I've noticed there a bit of an issue with caching. It's actually not very consistent. I can have 5 charts displaying 5 different data sets on one page, and 4 of them will update with new data, one of them will absolutely refuse to until I shut down the browser and open a new one. (I'm working with Firefox, by the way).

Firefox seems to really be in love with pages it's seen before. I haven't just seen this issue with FusionCharts. Some time ago we ran into a situation at work where clients were actually having to log in twice because the first login seemed to be ignored. What was happening was Firefox was overwriting the cookie for the login page with the favicon cookie. We had to examine every packet for each request to figure that one out. We used an obscure setting in the ajp connector in tomcat's server.xml file to resolve that one. 'emptySession=true' or something like that - I can't remember it off the top of my head right now - it was a couple of months ago.

Oracle & perl

Continuing to work with SqlPlus and Oracle 10g. Today I resolved two issues I had with an sql query.

With the first problem, I was passing two parameters into the .sql file and when the sql was executed in the file, along with the selected information, it would output 'old ... my sql here with the params not replaced' and then 'new... my sql here with the params replaced'. Thanks for that, but I know which params I'm replace, I don't need more notification of it. It turns out getting rid of this notification is fairly easy. At the top of the script, just put 'set verify off' and no more 'old...new' junk.

In the second problem (same script) I was doing some division and there were instances where my divisor could be (and was) 0. Sql doesn't like to divide by 0 at all (like pretty much every other language I've worked with). There is a neat little fix to get around this as well. My divisor looked like 'select count(a.blah) from ....' and if you add 'select decode(count(a.blah),0,1,count(a.blah)) from ...' it will substitute a 1 if the selected count is a 0. It worked super.

Messing around with Perl some more, I discovered some good functions. @myArray = split(/~/,$var); will take make an array of strings, seperating them by every '~' found in the string. Working with arrays, push(@myArray, someValue) pushes a new value onto 'myArray'. Pretty basic stuff, I know. I like to celebrate small victories (and remember them for next time).

Can you believe it, I hit a dog riding my bike on the way home from work today. A big doberman. He ran right infront of me. I slammed on my brakes and my rear end came up. I didn't hit him hard - but enough to put my handle bars out of alignment with my front tire a bit. I fixed that at the first red light I stopped at. I also just about hit a snake warming himself on the pavement.

Random assimilations and Business ideas

Today I learned that you can pass in parameters to .sql scripts that are destined to be run with sqlplus. It looks like this: 'sqlplus -s dbuser/dbpasswd@dbname @scriptname.sql param1 param2' and then in the sql file you output the params using &1 and &2. I was finding though that for the replacement sqlplus was being verbose and telling me about every replacement it made. I didn't like that so much, but couldn't figure out how to turn it off. I've already got 'set headers off' and 'set feedback off' set at the beginning of the script. I tried 'set logging off' but it didn't like that (because logging is set to off as a default I believe). Anyhow, the replacements were working.

With Oracle SQL, if you want to select back by a time interval, do something like this: 'select count(*) from my_log_table where logdatetime > current_date - interval '15' minute;' or you can use '- interval '6' hour' ... or '- interval '2' day' etc. Single quotes around the number!

Back to cron: to set cron to run every minute go '*/1 * * * * /scriptToBeRun.sh' Every 3 minutes: '*/3 * * * * /script2BRun.sh'

One problem I've run into with FusionCharts is the limitation of the Flash plugin to handle a big data dump. Put more than a few graphs with, say, 50 datapoints each on a page, and the browser hangs and the plugin throws our of memory errors.

Interesting IT Business Ideas:
- Reusing software development tools for other industries/applications. For example: other than the obvious issue with binary file types, why don't architects and lawyers use a repository like SVN, or Monotone? ( I left out the ones you have to pay for :-) We've tossed this idea around wondering if anybody has tried to market repositories this way....
- Becoming a freeware software consultant for small businesses. You can get everything from a good OS to all the 'openOffice' applications for free, not to mention what Google and others are putting on line (word processors and spreadsheets, etc), and universally accessible too! Seems to me if you're good at training, you can really help small businesses save a buck on software by showing them how to use all the free stuff out there. It wouldn't surprise me if people are doing this one already.
- write a GOOD, scalable, secure, extensible front end framework for web applications. There's a tonne of them out there now, but it seems like every one of them (Ruby, JSF, Struts, Facelets, Tapestry, SpringMVC, etc) has a downside. I've had conversations with a number of knowledgeable architect types and they agree that there is no good front end web solution out there right now. So, one could code a good freeware framework, consult using it, and then sell it to Red Hat (like the JBoss consultants did).

Got Something done today!

...or at least, it felt like it. After doing housekeeping with our production application yesterday at work, today was a 'sit down and code' day. Boy did it feel good. I had told myself that I had to get this charting solution finished today or it would turn into a waste of time. It turns out that FusionCharts are super easy to implement, and boy, do they look sharp.

In my 'solution', I'm using Nagios to provide the data, with its 'service-perfdata' process. That file gets big quickly. I just enabled it in the Nagios configuration on Friday last week, and I already had 12 Megs of data. Going through the Nagios documentation and forums, there doesn't appear to be any mechanism to 'roll' this file or archive it or buffer it.... It had me stumped for a while. Then I realized I could just do a simple 'tail -1000 service-perfdata.out > service-perfdata.out' in a shell script with a cron job, and presto - a simple fix.

So, mental note to self.... whenever you're coding a shell script (or any kind of script) that gets run by cron, make all the file paths in the script(s) absolute. I keep on running into this problem. Cron is ignorant about everybody's profile and position.

I just started coding with Perl this summer on this project, and I really like it. Working with arrays is tricky and not very intuitive. Otherwise, though, Perl is fantastic for whipping up something quick. Today I found out that you can use && and || in conditional 'if' statements. That was good. Also, something I've noticed in the past and ran into again today - the 'chop' and 'chomp' functions in perl don't seem to work like the documentation says they should. I ended up having to something like this to get rid of spaces '$var =~ s| ||g' - basically using sed (or the like) to get rid of spaces I didn't want. Perl's localtime() function to convert unix to to something readable worked slick for me today. Another thing I ran into not too long ago - the proper way to concatenate strings in Perl is with '.'

Rode my bike into work today. Just had the pedal axle fixed, and wow, what a difference. The aphids are all over the place right now. After a ride they are just plastered all over me. It's hard to not breathe them in while one rides.

Assumptions and Critical Thinking

A couple of weeks ago I programmed a DB load process that ran off a cron job. It ftp-ed data files from a public box to a local one, and then loaded the data based on .dat files that were extracted from the download, and .CTL files already on hand. I assumed this process worked correctly because every time I ran it manually it worked, and when I started it using cron, it 'looked' like it workded. It turned out that when it ran off the cron, it didn't do everything I wanted it to do. I set about debugging this today, and it turned out that my problem was very similar to my PLSQL problem from earlier this week.

Part of the Load process used a java program to created create the .dat files. The java worked fine when I ran it manually because the user I was running it with had JAVA_HOME set in it's profile. Cron runs differently. Even though my cron job su's to the proper user, doing the same thing manually and then checking with a 'who am i' command (which, by the way, gets different results than a 'whoami' command) returns a surprise. It's not my su'ed user. Adding JAVA_HOME into the script that gets run solved the problem.

Been looking at graphing solutions for a little thing I'm working on. GDchart seems usable. I might do some more work with it on Tuesday next week. FusionCharts seems very interesting as well. I've worked with Cewolf and JFreeCharts in the past, but I need something quicker right now.

Picked up a book called Th!nk off the discount shelf at Cole's today. I'm liking it so far. It's something of a rebuttal to another book called Blink!.

Nagios and things part 2

So I got to work this morning (with a fresh mind) and after a bit of googling figured out my problem in 30 minutes. Basically it boiled down to adding a reference to ORACLE_HOME in the bash script itself, and the full path of SQLPlus. After that, things ran fine even with changing the permissions of the scripts and related files back to normal and running Nagios with the Nagios user.

One of the 'nice to haves' for this little monitoring things I'm working on in Nagios is graphing capability - so we can see trends of the performance data at a glance. Just before I left today, I discovered that Nagios might install with this capability commented out. I let you know for sure in my next post.

While I was working on solving the graphing problem a different way, I got focused on some graphing addins for Nagios that required perl modules I didn't have. I learned that cspan is basically like 'yum' for perl. Didn't know what it was before today. In fact, I haven't had any exposure to perl before this project. It's all good - learning new stuff every day.

I also learned a big way to correct my golf swing today. I need to grip hard with my left hand and the right hand is just for support. I was driving much better and more consistently after I made that adjustment.

Nagios and things

I've done a couple of installations of Nagios now, and I quite like as a monitoring tool. Today I was working on coding a custom plug-in so we could keep an eye on a particular KPI. This involved running something of a complex query in Oracle and then sending the results to Nagios.
Writing the shell script wasn't difficult. The Nagios plug-in developer documentation was clear enough for me to follow. It took me a few minutes to figure out how to return the codes that Nagios is looking for - 'exit 1, or exit 2, etc'. However, to try and get Nagios to automatically run SQLPlus is a totally bear. I'm going to sleep on it tonite. I've tried changing perms on all the files, even tried changing the ownership to oracle and setting the SUID - no dice. Tried running Nagios as root (not recommended, but I had to try), and set all the Oracle environment variables in the .bash_profile for all the users I could think of that might possibly be running that code. I tried doing a su -c 'call my script here' oracle, and that didn't work either.

Oracle posts online regarding SQL*Plus 43 initiation errors etc. are pretty sparse. In my opinion, it's just more fodder for people who advocate using 3rd party free ware. At least if it's broken, somebody has likely run into the same issue and posted it online. With Oracle and Tibco, it seems like they guard their support forums with their life (maybe Oracle not so much so).

Oracle, Tibco, and tcpdump

I've started a new project recently that uses Tibco with an Oracle backend. It still amazes me the little nuances of domain knowledge that are so key to being a productive developer on some of these 'platforms'.
My issue today was a query I was trying to run using the jdbc 'component' in Tibco. It's pretty finicky about SQL syntax. I needed aliases for column names and it turned out the only way to get those to work is to use double quotes. Single quotes work if you want to hardcode what you're returning in your select statement. Double quotes (I was told) are effectively ignored by Oracle unless you're trying to alias a column. I'm starting this blog mainly so I can keep these little nuances documented somewhere - I hope it'll help me remember in the future.

I ran into a different configuration problem with Tibco last week which I had to use tcpdump to help me solve. I used the command something like this: tcpdump -nnXi eth0 -port 8080. This helped me realized that Tibco was responding to http GET and not http POST requests.