The Job the Wasn't There - A Lesson in Applying for Jobs

Hermione (made up name to protect her identity) was a student in my fast-track Web Developer class at SAIT in early 2017.  Like a lot of students I get in that course, she was anxious about obtaining work after receiving her diploma.

One day in between exercises, I showed the class several 'Careers' web pages of good, local web design companies.  One of those companies was Critical Mass, a company I had actually consulted with before.  I often recommend this company to students because they have a world class client list, they do internships, and I have experience with them.  That particular day, they happened to have an opening at the time for a Junior Web Designer, but no posted opportunities for internships.  I encouraged the students to apply for the Junior Web Designer opportunity and Hermione challenged me...

"How can we do that when we don't have all the qualifications in their list of requirements?"

I often get this question, and I had an answer. "You need to understand how a company creates a job description.  Many put it together as a list of qualifications for the perfect candidate.  Others will build the job description based on an existing successful employee in the company. They realize that most of the applicants won't match all of the qualifications - and this is particularly true in the IT industry. "

Hermione digested my answer, and piped up again. "But we're still in school and we have several more weeks before we'd be available to start working!  Does it really make sense to apply now for a position like this?"

"Absolutely!" I replied. "You never know what might come out of an application.  The hiring process for many companies takes several weeks.  There's usually a bunch of interviews for them to schedule and have, and then some planning and logistics around actually bringing the successful applicant aboard.  You never know what will happen out of an application."

She still looked skeptical.  I moved the class onto another exercise and didn't think too much more about it.

Several weeks later, I received the following email from Hermione:

"I took your advice about applying for jobs and I applied at Critical Mass for a 

Photo by Kaitlyn Baker on Unsplash

Junior Web Developer position knowing that I was NOT qualified and that they probably never call me back. Guess what? They called me back! They don't think I am ready for the Junior Web Developer position, but they want me to interview for their internship program. The interview is on . Which leads me to the crux of this email. Would you consider being a reference for me? And do you have any advice for this interview?"

I responded:

"Lol Excellent!  Good for you, Hermione.

Certainly I can be a reference (as a teacher) for you.

Probably the best advice I have for your interview is if you don't have the right answer, straight up tell them.  But then also tell them you'll have to answer (or know about whatever their asking you) tomorrow.  In other words, when you get home, you'll investigate it and get the answers.  

Bring a notepad to the interview and make notes about anything like that (so you look like you mean business).  Come with a couple of questions as well.  Research in advance anything in the job description you don't know about so you feel prepared.  Research the company a bit - know where their office is, ensure you can make it there on time, who are their current clients, some of the history, etc. 

Smile!  I don't know if you read my blog post about that, but smiling is HUGE.  If you can, try and get an interview somewhere else first to practice and get the jitters out (and maybe get a competing offer) 

Hope that helps!  Good luck!"

She replied:

"Thank you! I appreciate the reference and the advice.

I've been panicking a little, I really thought they would never call. I'm scrambling to get my portfolio site updated for the interview, as well as just get prepared in general. I do have a practicum lined up though, so no pressure...sort of."

In the end, Hermione got the internship.  She was nervous going into the internship because she didn't feel entirely qualified.  I told her not to worry and ask LOTS of questions.  She ended up successfully completed her internship and came out feeling better about it than she expected to.  It was a great lesson for her (and for me and all more students who I tell this story to) of how there are opportunities that you don't see in the job market.

Facebook Job Interview - Production Engineer

Facebook contacted me on LinkedIn recently looking to fill a 'Production Engineer' role.  I wasn't looking for job offers.  They reached out to me.   Apparently they had been doing this a bit though, targeting DevOps professionals as my buddy at the startup I was recently working with also got contacted.

Having Facebook reach out to you for a job opportunity?  Definitely intriguing.  One of the things on my 'IT Career Bucket List' would be to work at one of 'those' companies.  Google, Facebook, etc.  I thought if nothing else, it would be interesting to see where the hiring process went, since in the past they didn't actively recruit unknowns like me. 

I replied that I'd be interested to know more and so a phone interview was arranged with the Facebook technical recruiter.  At the appointed time (a couple days later) he called and I got the skinny on how things would potentially work.

The phone interview was the first step.  After that I provide them with my resume which some of their technical leads would look at.  If they felt I was a fit based on my resume, there would be two remote technical screens - 1 specifically focussed on the Linux OS, and the other on a coding language of my choice.  They wanted someone who was very comfortable at both.  Over the phone they gave me some sample questions - basic linux commands - to give me an idea of what the screens would be like.  If I managed to gain their approval in the technical interviews, Facebook would then fly me to the location of my choice (Menlo Park or Seattle) for face to face talks - both technical and otherwise.  Following that, if I was still up to snuff, I'd get an offer.  Once hired, there would be a 6 week on site 'boot camp' where I'd get trained in all things Facebook and brought up to speed the technical ins and outs of the team I'd be working with.

Getting hired would require me to move to Seattle or Menlo Park.  Real estate at both locations is exorbitant - like ridiculous.  In the event that I was hired, Facebook would offer me a full relocation package, with the potential for a temporary housing situation for 3-4 months while we looked for a permanent residence.  I was told that many employees commute from communities with more reasonably housing prices using Facebook commuter busses that have complimentary drinks and WIFI.  Health and Dental would be covered 100% for myself and my dependents.  Wednesdays are an optional work-from-home day, and Facebook offers 21 days of vacation per year - although I neglected to confirm whether that was 21 business days (over 4 weeks), or 3 weeks all in.  I'm a Canadian, so I asked about a work Visa.  He replied that Facebook has an immigration/legal department that would arrange a TN1 Visa for me, in the event that I was hired.

As far as the job itself - Facebook's version of the Production Engineer role is essentially described here.  At the time I was talking with them, Facebook had 42 teams each responsible for a particular feature in their system (FB messenger, Ads, Newsfeeds, etc.).  These teams consist of 4-5 developers with an embedded production engineer.  Everyone on the team goes on call, one week at a time, so it ends up being a 4-5 week on-call rotation.

In the end, after viewing my resume they decided not to pursue the hiring process with me further at this time.  They had been interviewing 'a lot of strong candidates recently that they felt were a stronger match for their immediate needs.'  I can't say I was heartbroken.  It would have been a big move for us that would have put pressure on me personally and financially - not to mention having one dependent in university and one in high school.

I have been pondering what it was about my resume that flagged it to the hiring managers.  Was it the fact that I've moved jobs every 2-3 years, and they had concerns that I wouldn't last long at FB?  Perhaps it was because of my lack of focus in technology - moving contracts 2-3 years means learning lots of new technology and never getting a chance to really focus.  I've asked the FB recruiter - we'll see what he comes back with.  (You might be wondering why I've moved jobs every 2-3 years...  Its prudent for independent business contractors like myself to 'keep moving' from a Canadian taxation perspective.)

The Perils and Pitfalls of OpenSource Software

Open-Source Software is the backbone of most of the internet.  Seriously.  For more than a decade, small business websites and main-stream web applications have used open-source software to develop their solutions.  Web projects (and frankly, most software projects in general) that don't have a dependency of some kind on an open-source component are the RARE exception to the rule.

Should we be concerned about this?  I think so.  Here's why:

1. Coders aren't implementing open-source code properly.  In my experience, any open-source code dependencies should be referenced locally.  Many coders fail to do this and reference external code libraries in their code.  What happens when that external 'point of origin' has a DNS issue, or is hit with a DDOS attack, or is just taken down?  Your site can break.
   
   Case in point - check out this article on msn.ca 'How One Programmer Broker the Internet'  In a nutshell, one open-source programmer got frustrated with a company over trademark name issue.  This developer's open-source project had the same name as a messaging app from a company in Canada.  He ended up retaliating by removing his project from the web.  It turned out his project had been leveraged by millions of coders the world over for their websites, and once his code was removed, their websites displayed this error:
            npm ERR! 404 'left-pad' is not in the npm registry
   I believe if developers had downloaded the NPM javascript libraries and referenced them locally on their servers, they wouldn't have run into this issue (as they'd have a local copy of the open-source code).
   
   Another case in point - I worked on a project a number of years ago that had dependencies on jakarta.org - an open source site at the time that was hosting a bunch of libraries and schemas for large open-source projects like Struts and Spring.  Some of the code in those projects had linked references to schemas (rules) that pointed to http://jakarta.org.  Unfortunately we didn't think of changing those links and one day jakarta.org went down....  and with it went our web site, because it couldn't reference those simple schemas off of jakarta.org.  After everything recovered we quickly downloaded those schemas (and any other external references we found) and referenced them locally.
   
2. Security  Do you know what is in that open-source system you're using?  Over 60 million web sites have been developed on an open-source content management system called WordPress.  Because it is open source, everyone can see the code - all the code - that it is built with.  This could potentially allow hackers to see weaknesses in the system.  However, WordPress has pretty strict code review and auditing in place to ensure that this kind of thing doesn't happen.  They also patch any issues that are found quickly, and release those patches to everyone.  The question then becomes:  Does your website administrator patch your CMS?
   
   Another issue I ran into related to security was with a different Content Management System.  I discovered an undocumented 'back-door' buried in the configuration that gave anyone who knew about it administrative access to the system (allowing them to log into the CMS as an Administrator and giving them the power to delete the entire site if they new what they were doing.  Some time later, I found out that some developers who had used this CMS weren't aware of that back door and left it open.  I informed them about it, and they quickly (and nervously) slammed it shut.   Get familiar with the code you are implementing!
   
3. Code Bloat (importing a bunch of libraries for one simple bit of functionality)  Sometimes developers will download a large open-source library to take advantage of a sub-set of functionality to help save time.  Unfortunately, this can lead to code bloat - your application is running slow because its loading up a monster library to take advantage of a small piece of functionality.
   
4. Support (or the lack of it)  Developers need to be discerning when they decide to use an open-source library.  There are vast numbers of open-source projects out there, but one needs to be wise about which one to use.  Some simple guidelines to choosing an open-source project are:
• How many downloads (implementations) does it have?  The more, the better, because that means its popular and likely reviewed more.
• Is there good support for it?  In other words, if you run into issues or errors trying to use it, will it be easy to find a solution for your issue in forums or from the creators?  
• Is it well documented?  If the documentation is thorough, or if there have been published books written about the project, you're likely in good hands.
• Is it easy to implement?  You don't want to waste your time trying to get your new open-source implementation up and working.  The facilities and resources are out there for project owners to provide either the documentation or VM snapshots (or what have you) to make setting up your own implementation quick and easy.
• How long has it been around?  Developers should wait to implement open-source projects with a short history.  Bleeding-edge isn't always the cutting edge.  Wait for projects to gain a critical mass in the industry before implementing them if you can help it.

The Value of IT Certifications and Skills

After I've told my students at school how I finally managed to get a full time job in IT, they've asked me what I thought was the most important thing I did to get that job.  I've always answered 'getting my SCJP certification' (Sun Certified Java Programmer). It's been over 10 years since I started working at the job.  While I still maintain certifications are important for developers who are trying to break into the industry, I've become less enthusiastic about them and what they mean now that I have more experience. 

Having said that, I ran into a couple of articles today that I thought were very interesting:
15 IT Certifications That Can Get You a $100,000 Salary
30 Tech Skills That Can Get You a $100,000 Salary

Granted, the way these articles are written is 'just a little bit of' hype.  These skills and/or certifications aren't going to net you a 100k salary all on their own.  They need to be married with a resume that has several years of IT experience and related, well developed soft skills.

IIS and setting NTAuthenticationProviders and authNTLM

More than once now I've run into situations with the NTLM configuration in IIS where it seems that the GUI setting doesn't do all you need it to to properly configure integrated windows authentication for a web site. Earlier this week I was asked to look into a situation for a particular project team where Firefox wasn't prompting them to enter a username/password in one environment (they'd just get an http 401.2 error), and yet it was giving them the prompt in another environment.  (IE was running fine in both environments, by the way).

I went through all the IIS GUI configs in IIS and the two boxes looked identical.  Since this appeared to be an issue with integrated authentication, I ran this script:

cscript c:\inetpub\Adminscripts\adsutil.vbs GET W3SVC/WEB_SITE_Number/ROOT/NTAuthenticationProviders

which showed me that the working server had nothing set (wasn't set), while the server that was giving them problems returned with 'Negotiate,NTLM.'  I subsequently set NTAuthenticationProviders to '' and it worked by using 'set' instead of 'get' in the above script and providing an empty parameter.

Note:  I understand the above setting to be different than the one below.  In more than one instance at work I've had this issue and proven that this is the case.  The script below, from my understanding, the same as the Windows Integrated Authentication checkbox on the Directory Security tab in IIS.

cscript c:\inetpub\adminscripts\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/authNTLM TRUE

IT Professional - Know Thyself

What are you good at?  It pays to know - literally.  It will also make you happier and more productive on the job.

The more I work in IT, the more I see where I'm most effective. I also see in comparison where others blow past me in their individual skills.  It's important for multifunctional teams to have a good idea of different team member's areas of expertise.  Leveraging (knowing when and who to ask for a specific kind of help) these team skills can greatly improve the effectiveness of a team.

Here's some real life examples:

I'm rather weak on writing complex sql statements.  I can profile sql, and can get most sql work done.  But if I've got a difficult query I'm working on, I'll go ask my team mate who is an expert on putting together complex sql.  Later on, he'll ask me for help with a visual studio or server/service configuration related question.  Its a good, productive working relationship that keeps everyone happy.  The BA's are happy because the data looks right.  My PM is happy because my task was finished on time. And my team mate and I our happy because we both learned something and did it faster than trying to google for it on our own.

I'm great at problem solving software related issues - particularly in a domain I'm familiar with.  Whether it's missing permissions, configuration, network connectivity, everyone sitting around me will tell you I'm very, VERY persistent at solving problems that are brought to me.  My neighbor one cube west is fantastic at IIS/MSTDC/COM related bug/configuration debugging.   She also knows our software domain better than almost anyone else in the building - she seems to have it memorized.  When I have an issue with our software that I don't know how to fix, or a business related problem in the software,  I talk to her without hesitation.  Conversely, when she is stuck on an issue that she doesn't know how to solve, she will buzz it past me.  Again, a productive working relationship that makes everybody happy.  The business is happy they got their patch as fast as they did.  Once again our PM is happy, etc....

In meetings I will keep quiet and not say very much unless I feel there is an important point being missed.  I'm not super great at arguing a point.  I'm not the person that's always called on to go to a meeting to convince another group of people that 'our way is the right way'.  Part of this is my personality, part of it is my belief that there is multiple ways to get 'a good enough' answer for the problem we are looking to resolve.  However, we have another fellow in our group that will absolutely blow you away with his gift of logical persuasion.  He's the guy that gets sent to those 'convince the group' meetings.  On the other hand, I'm called on when diplomacy and a consensus is required.

Know what you are good at.  Seek to know what your team mate's strengths are.  Become a happier, more productive development team.

Excessive logging kills AppPool (log4net)

We had an issue where one of our Application Pools (IIS 6) was 'dying'.  The interesting thing about it was while the application itself was dead (error page, sometimes showing an outOfMemory exception, other times showing a meaningless exception), the appPool itself appeared in the GUI to be running.  The more troubleshooting I do with Windows OS's, the more I do not trust what the GUI says.  Appearances in Windows can be deceiving.  For several weeks - until we understood the root cause of the issue - are resolution to the dead application was to restart the 'running' appPool. 

A side note - this appPool was running on a Network Load-balanced server, on the dominant node.  The appPool for this particular application only ever failed on the dominant node.  It never failed on the other node of the cluster, so we had reason to believe that the 'load' had something to do with the root issue.

In the end, we discovered the root cause of our dying application by accident.  We were cleaning up our files on the prod servers when it became apparent to us that log4net on another application (and another appPool) was set to log 'ALL'.  This resulted in 1.5Mb of log created about every 5 minutes.  I had a hunch that all this logging was having a memory impact on the server - we certainly didn't want to be logging at the 'ALL' level anyway, so we created a change request and fixed that issue ASAP.  Then we crossed our fingers, hoping that our dying AppPool issue would go away as well. 

It's been more than two weeks since we implemented this log4net change and our flaky appPool hasn't died once.  Lessons learned:

- Watch for excessive logging on prod servers
- Excessive logging can apparently cause other appPools to through outOfMemory Exceptions!

Windows VM False Performance Monitor Stats

We almost exclusively run VM's now in most of our environments. Recently, we added two new Windows VM's in production for a particular solution.  Thinking we were being prudent, we asked to have the boxes provisioned with 4 Gigs of Memory since that's what the old 'hardware' boxes had. We deployed our solutions to the new boxes and tested them without a hitch.  So a week later a switch was flipped and the new VM's went live while the old boxes no longer took requests.  After a few days, we discovered (by looking at the task manager) that we were getting some conflicting and concerning readings for memory utilization. Check this out:

- the task manager graph said we were using 3.7 gigs of memory
- looking at the actual numbers below the graph, we only had 300Mb of memory available on the machine
- looking in the processes tab, none of the processes were using a significant amount of memory (several were using between 200-300MB, but that was within normal tolerances when compared with servers running the same system in other environments.  However these same processes where paging memory like they shouldn't have been!  Page faults were very high.
- Totaling the memory used in the individual processes, we were NOT using 3.7 Gigs of memory.

After doing some more investigation, we asked the team that built the server how the memory had been provisioned for these new boxes.  It turned out that they both had originally been given 1Gig of memory and then later (after they were running) 3 more Gigs were added.  At this point our hunch was that these  boxes weren't seeing the additional 3Gigs that had been subsequently added.  After doing a hard reboot on both boxes, this turned out to be the case.

Lesson learned: make sure you follow VM procedure (hard re-boots) for adding/removing resources to VM servers.  Short-cuts can create some pretty confusing statistics in the OS.

Debugging/Troubleshooting SQL performance/blocking issues

I picked up a good tip from one of our DBA's yesterday.  He uses sp_who2 to see which users have active threads/processes running against a particular database.  This stored proc returns the PIDs as well (Process ID number).  I had known about this previously.  What was new to me was he also used:

dbcc inputbuffer(pidNumberHere)

to get the database to tell him which stored proc was being executed by that PID.  That seems to me like a handy trick.  It certainly solved the problem I was having.

WCF and Large Result sets

Dealing with oil resource data across several disparate systems and also trying to make our API's more generic and consistent, we use WCF a lot (web services with windows apps). We've had a couple of instances where we've had to tweak the config settings for either our publishers or our clients to return some large datasets over WCF.
For the most part, I have found the error messages relating to constrained WCF config settings refreshingly to the point.  They pretty much pointed me directly to the attribute in the file that I needed to change in order to fix my issue.
For memory related errors on the client, I found that juggling with the maxBufferPoolSize and maxReceivedMessageSize in my binding to help.  I moved the maxBufferPoolSize from 9999999 to 99999999, and changed the maxReceivedMessageSize to 350000000.  Sometimes after that I would get a timeout error on the client machine (depending on what kind of load the server was under).  So I'd have to make the timeout settings on the binding higher.
Obviously it helps when the publisher has the same values for the same attributes on its binding.

Books I want to read soon...

Here's two books I want to read soon:

The Nomadic Developer: Surviving and Thriving in the World of Technology Consulting







And

Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler))

Windows OS Performance Guidelines

These are notes from a workshop I was able to be a part of a couple of weeks ago.  The workshop was on Windows Performance Monitor and monitoring vital signs on Windows Servers.  The windows performance monitor has a multitude of counters that a person could potentially monitor on a server.  I'm just going to point out a few critical ones that were highlighted in the workshop and what their tolerances are.

Some points to note before I get into these values:  

1. Don't always look at the graph first. The counter graphs can be scaled right out to lunch so they aren't necessarily a good first glance indicator.  For a particular counter, your focus should first be on the Minimum and Maximum values in perfmon just below the graph.
2. In the past, I would tend to keep an eye out for sympathetic counter relationships on the graphs.  However, keeping an eye out of inverse counter relationships is also a good idea (where one counter could be decreasing in value while another is increasing)
3. Counters can get corrupted.  Apparently this happens more often than one would think.  They can be rebuilt - directions are in this KB post.
4. You can attach a PID (process ID) to a perfmon counter in an OS older than Windows Server 2008 by modifying reg keys. Details are in this kb article. Versions of perfmon on 64bit OS's come with this already set up.

On to the objects/counters of note:

• Process Object
• Handle Count - greater than 500 handles may point to a problem.  
• Private Bytes - greater than 250MB could be a problem.  I've seens procs over a Gig and they definitely were a problem (they can get that high).               
• Working set - greater than 250MB could be a problem
• Thread Count - Greater than 500 threads needs to be watched to ensure they aren't increasing over time
•  Processor Object
• Processor Time - all core instances.  _Total can get you an overall trend.  Greater than 91% utilization is potentially an issue.
• Network Interface Object - you need to know what the spec is for your network interface to determine it's capacity.  Anything over 80% of capacity could point to a problem
• Current Bandwidth - help you determine the NIC's capacity
• Output Queue Length - greater than 2 is an problem
• Bytes Total - greater then 65% of capacity utilized is past the warning threashold (blinking read with siren)
•  Memory Object
• Free System Page Table Entries - the higher the better here.  Lower than 5000 is considered critical.  I've seen boxes 'run' (aka hobble) around 2500.
• Available Megabytes - again higher here is desirable.  Less than 100 MB or 5% free is very problematic
• Pool Non Paged Bytes - greater than 80% consumed is out of spec (not at all good)
• Pool Paged Bytes - Same as non-pages bytes.  Anything between 60-80% should be watched.
• Logical and Physical disk Objects - they have the same critical counters so I've put them together here
• Idle percentage - 19 to 0 percent is critical. Anything over 55 is warning.
• Current or Avg Disk Queue Length - 3 to 31 in the queue - you need to keep an eye on it.  greater than 32 is an issue
• %Avg Disk Sec Read or Write - 25ms and above is critical

 Other interesting points of note:
- Mark Russinovich was the original developer of Perfmon.  His blog is apparently pretty good and our facilitator was very impressed with him.
- Windows Server 2008  does processor/core parking.  This means the server will 'retire' (effectively turn off to save power) CPU's when there isn't a heavy load on the box.  Our facilitator told us that one ISP moved all their boxes to Windows Server 2008 for this reason and their power bills were 15% less a month. 

Newbie Advice

Having gone through an IT technical school, I realize they push a lot of information and technology at you. There's a lot to assimilate in less than a year (depending on the school). On this page I'd like to point out some technologies that perhaps weren't covered much in classes that are widely used by good shops in the industry, and also offer some career advice for the future. Before I delve deeper, let me say that this information is based on my experience, which is mostly on the Java technology side of the fence.

Technical Heads Up

1. Design Patterns. One thing many of the technical schools don't have time to cover in their curriculum is code design patterns. It doesn't matter if you doing .NET development or Java development, design patterns can really make a difference in organizing, maintaining, and the reusability of your code. A great book to pick up for an introduction on design patterns is published by O'Reilly and called Head First Design Patterns. If you don't have access to the book, a good place to start is to google some popular patterns that are being used in code now. Dependancy Injection is a pattern that is really big right now. Two frameworks that use it a lot are Spring and Google's Guice. Other pattern I've seen used frequently are the decorator pattern, the abstract factory pattern, factory pattern, the singleton pattern, and the proxy pattern.

2. Using a Repository. Using a repository, or a version control system, should be required by every development shop. Frankly, if you're in a company where they aren't using one, either push to get one installed, or start looking for another place to code. A repository is basically an application that functions as the premier source/bank/datahub/repository for all code. This application should be on a server with lots of disk space and accessable to all developers in the company. Repositories allow developers to manage code changes at many different levels allowing many developers to work on the same code base at the same time. Some repositories I've used in the past are Subversion, CVS, StarTeam, and Perforce. Googling any of these will give you more information.

3. Unit Testing. Unit testing is the development practice of coding 'tests' that validate individual classes or 'units' of code, to ensure they are functioning as required. This practice has developed over the past 4-5 years or so and has been found to be very effective in giving developers the confidence to refactor their code and ensure that it's functionality stays consistent over time. Some unit testing technologies are JUnit, JWebUnit, HttpUnit, DBUnit, and NUnit. 

4. Continuous Integration. Continuous integration is a process that automatically rebuilds and tests (with unit tests) an application or code base. Continuous integration employs an application like CruiseControl or Anthill Pro which is configured to automatically check out the code base from the repository, compile it, run it's unit tests, and sometime even deploy it to a test environment. It allows developers to check for integration problems in the code, and warns developers of broken code, allowing and reminding them to fix it ASAP.

Soft Skills and Career Advice

1. Change is GOOD. Once you've gotten your feet wet in the industry, being a Java developer in one company can mean something quite different that being a Java developer in another company. There is such a vast variety of Java technology that two Java developers can have quite different experiences in developing applications. Whole development processes and methodologies can be very different in differenct companies. One company could be flying by the seat of their pants, developing with not backups or repository, and manually pushing out code, while another is totally immersed in Agile methodology and outsourcing everything but development.... etc. Keeping this in mind, my advice is to not get comfortable in any one company but move around a bit - that is, after 18 months or two years at one place, get a different job. You'll be put into a position where you'll either learn a lot of new stuff, or you'll be mentoring. Either way, it'll be an experience that will likely teach you more then if you had stayed put. It'll also give you some motivation to brush up on your resume writing, interview skills, and expand you network. 

2. Walk & Talk While software developers, for the most part, end up working in cube farms, it should not mean that they need to stay in their cubby hole all day and do work. This may be just my style, but I find when I'm working on a big integration project or on a project with a lot of departments or teams involved, getting up and talking to people really helps things move along. Whether it's getting socializing a new means of managing your server configuration in the code base, or talking to a DBA about a script that you'll need to release with you new code, talking face to face is far more effective than emails, I've found. I generally use emails if I can't find the person, or I want to make sure there is hard documented proof about a decision that is being made, or if I want to include a bigger audience in my communication. Successful software development is more about relationships that a lot a people realize. Relationships are more apt to be cultivated face to face... plus it's good exercise.

 

Career Paths in IT

I've been teaching at SAIT again this week.  Generally before every class gets going in the morning I like to do a little 'blurb' on concepts or thoughts that aren't covered in the course the students get.  One of the things I find is that the vast majority of students (and the population at large for that matter) don't realize how many different career paths there are in just the software side of IT.  Guaranteed there are some students in the course that I teach who, by the time I get the chance to teach them, are very concerned they made the wrong decision by getting into software programming.  They don't feel like they are cut out for programming.  I try to put their fears at rest by suggesting that there are other opportunities for them - careers that can utilize the coding experience that they are getting, but allow them to use talents that they are more comfortable with.  Here's a list (by no means exhaustive) of some other opportunities in the software field.

- Project Management.  Project managers who have coding experience or at least a technical background are a step ahead of PM's who aren't techies.  I myself have considered taking some project management courses or getting my PMP, but thought the better of it after seriously thinking about what I like to do at work.  PMs need to have a great soft-skill set.  While I don't think I'd have a problem with that, I would missing troubleshooting/solving problems.  They also have to be politically savvy and have patience for meetings.

- Business Analyst.  BA's also need to have a great soft-skill set, as they need to be talking with everyone (the business, the developers, and the PM) and writing down the requirements.  It also helps if they are technically competent, and they enjoy documenting/writing.  For some reason in Calgary right now, there have been a lot of postings for BAs over the past 6 months. I just found out today that a variety of institutions offer certificates/diplomas for a Business Analyst career path. (NAIT and Mount Royal University are two local options)

- Quality Assurance.  This field has been growing a lot over the past 8 - 10 years, and frankly I don't think projects that I've been on over the past 5 years would have been successful without our QA teams.  QA experts need to have a penchant for detail and process.  They need to be able to understand the business and relevant business rules well.  Most of the QA people I've worked with haven't had a certification, but you apparently can get more info on a software QA certification here.

- Release Management.  This is what I (like to think I) specialize in.  We ensure that teams are using a continuous integration server and QA has the builds they need for testing where and when then need them.  We have two main goals: 1. Keep downtime in production to a minimum and 2. Keep lost man days in QA to a minimum.  I like release engineering because I get to see the whole picture - I have to understand the code and it's dependencies, I need to understand the infrastructure, and I'm responsible for the product getting through all the environments to production.  For me (being a bit proactive), this has also lead to automating deployments of database objects and datascripts, server configurations, and code, as well as installing monitoring tools to ensure that every environment is ready to 'rock'.

- Security Analyst.  These dudes make sure the good people have access to everything they need access to, and the bad people are locked out.  They spend a lot of time combing though logs and configuring IPSec rules.  They also perform software audits on code, all kinds of different tests on web sites, and network audits.  I wouldn't mind moving my career this direction in the future as they also need to have a pretty good idea of the whole system/infrastructure/architecture.  Great web sites for more resources on these types of positions are here and here.

Even more specialized positions:
- GIS (Geographic Information System) developer or architect or analyst.  Working with digital 2d and 3d mapping systems.  Having some limited experience with this at the ERCB (deploying mapping systems that display reservoirs,etc in Alberta) I think this is pretty cool stuff.
- Search Engineer (Search Engine Optimization specialist)  I've dabbled in this off and on for my clients.  You definitely need to be on top of your game and know how the search algorithms are changing.
- Business Process Specialist - there's lots of software to specialize in here.  Tibco, Agile Point, SAP, JDEdwards, etc.
- Document Management Specialist - Sharepoint, Edocs, Documentum, etc.  This involves the automated management of documents - not the writing of them, but their storage.  Government bodies and law firms definitely have a need for these kinds of systems.

The importance of soft skills in IT

Soft skills are crucial to being a successful developer in the IT industry.  Some of the soft skills I've found beneficial in my career are:

• Being a good communicator.  It's not just about talking, but listening as well.  Dirareah of the mouth is never cool, especially when a job needs to get done.  Strive for mutual understanding in a conversation. Communicating in the IT world isn't just limited to talking and listening though.  
• Having the discernment to know when to write an email versus actually talking to a person is important as well. I do not recommend doing all your communication through email.  Communication in emails can easily be mis-understood or mis-construed and take on an entirely different meaning and tone than you originally intended.  Emails are great for documenting decisions.  
• Make sure that the content of your email deals with the subject you've put in  your subject line.  For example, having a subject line of 'Quote for project XYZ' but then talking about quotes for that project AND other projects in that email is bad news.  Send you quotes for the other projects in a different email with a more relevent subject line (I know this from experience).
• Developing relationships.  Extending communication a bit further and actually developing relationship with people in the office is huge as well.  Questions like "How was your vacation?", "Where are you going on vacation?", or "What did you do over the weekend?" are great for this. When you've got a relationship in place, there's trust.  Many times I've been granted exceptional privileges and become more productive in my work because I've had a relationship with a key person who trusts me.  These relationships can also help in the future as well when you are looking for that next gig.
• Having patience.  Whether it's letting somebody else go to the front of the 'line' for access to production, or having forbearance when dealing with a difficult client, patience can go a long way to strengthening that 'trust' in your relationships.  Sometimes work (or working with clients) in IT can get frustrating.  Things I do to help me step back and get some perspective are:

• go for a walk
• find something else to do for a while
• talk to someone - use them as a sounding board
• take a day off or go on vacation

• Positive Attitude. I like to keep my workplace fun, interesting, and upbeat.  To promote this around me I'll sometimes:
• Bring baking from home to share
• Audibly rejoice in small victories (getting a solution migrated and compiling from VS2005 to VS2010 for example)
• Write thank-you notes (either hard or soft copy) to others who help me
• Encourage people who are down. We had a large project I was helping with and the lead was concerned that between the time and the technical challenges he had, he wasn't going to be able to finish on time.  I believe he could and I told him so (and backed it up with my help).  In the end, he did finish.
• Being Flexible.  Generally, you are always working for a client.  Invariably, they will change their mind or want an enhancement.  Microsoft will upgrade their software, the US president will change daylight savings time one more time, the CTO will be convinced that Tibco will solve all the business problems again.  I've found that being able to take these changes in stride is crucial to maintaining my mental health (avoid being overcome with frustration).  It's all part of the territory - and frankly, job security.

Grep features

I was asked to search several teams' code-bases recently based on a list of over 270 different search strings. We are a Windows OS organization and doing a search like that would take some time. By the way, has anyone noticed that the Windows explorer search tool is flaky? Even if I used a tool like this it would still take a while.
I have cygwin installed, and I know grep works well - when I search for something I know is in the file, I find it (unlike Windows sometimes). What I didn't know was that there is a -f option on grep that allows you to provide a file with all you search terms separated by newlines. An astute worker made me aware of this and I tried it out this morning. It worked nicely and saved me some time. Here's how I implemented it:

grep -r -i -f file_of_search_strings.txt * | grep -v .svn | grep -v Assert | grep -v .sql >> resultFile.txt

-r is a recursive search based on where I'm current sitting on the command line
-i is a case insensitive search
-f allows me to pass in my file of search strings
* tells grep to search every folder/file at this level. Used with -r it tells grep to search every single folder and file that is currently at my current cursor position, or a child of the folder I've cd'ed to.
The | grep -v .svn tells grep to ignore/not display files with *.svn in the results
and then >> allows me to pipe my results to a file for inspection.

You may have to run the dos2unix command on the text file with your list of search items to get grep to see the new lines in the file correctly.

MS Sql Server Stored Proc Search Query

If you work in an org where there's a lot of business logic put into (MS Sql Server) stored procedures or you have an application that is stored proc intensive, you might find this code snipped helpful.  It searches all the stored procs in the database for a specific string

select distinct o.name from syscomments c
inner join sysobjects o
on c.id = o.id
where o.type = 'p'
and c.text like '%string to search for%'
order by o.name

Key Questions in Software Sustainment

Here are some of the questions I ask myself when a software system has stopped running properly...

The first big question is: What changed between when the application was running well, and the time the application stopped running well?  To get to the bottom of that question, there are a number of other questions that can help point the way.

Question isolation - Is this problem isolated in any way?  Is it only on a specific network, environment, or group of servers?  Does it only happen for a specific group of users or a specific client?  Is there a period of time it's isolated to?  Is it isolated to a particular 'item' in your data?  I've seen users try to make an application  use a different browser version than the documented supported versions.  Some times it takes a while to get to the bottom of simple issues like that.  Also, we've run into situations where an organization will not have given users the rights on their machines to install third party active X components for their browser that the application they are trying to use requires.

Question data integrity - Whether you are looking at legacy data or no logic to manage special characters in your data, you need to break the problem down. With legacy data, is the problem isolated to a particular user or group of users, or a particular item?  Or if it's an ETL function, does relational integrity from the first DB line up to the second DB correctly - are you missing or adding 'types' of data that the second DB either is or isn't expecting?  Sometimes you need to pinpoint the exact row or time when the issue occurred to determine what the problem was.

Question continuity - Did something stop running or listening? A service, appPool, web site, 3rd party server?  Are your cron jobs or scheduled tasks still there? Monitoring would quickly and easily answer this question for you.

Question communication -  Are the lines of communication open to all of the dependencies that your application has?  Has a network cable been severed by a backhoe down the street? (I've seen that before)  Is there too much communication going on?  Too many calls from one routine can take you system down.  We had an issue like that with some javascript that called a data access function to a GIS server.  As soon as one too many layers got added to the map, performance died as there were too many round trip calls to the server in one request. Internal systems that need to do identity verification or IP Geolocation are heavily dependent on external third parties to operate.  These external vendors can in turn can be dependent on other external services.  Know all your dependencies and have monitoring and SLA's in place for all of them or you could be sweating bullets.

Question dependencies - Here I'm thinking more in the context of internal dependencies - what internal 3rd party services are you dependent on?  Databases, reporting tools, monitoring systems, document management systems,  any internal system that your application depends on that you aren't responsible for fits into this category.  Are they up and running?  How do you know?  Are they running properly?

Question the un-questionable - Is your hvac system working at your Co-Lo?  As redundant as your service provider tries to be, there will always be something under the radar.  Always.  I've seen an external hvac system take down an entire enterprise.  Many of you have likely seen a McAfee or a Norton patch take down an enterprise.  How secure is your UPS (Uninterrupted Power Supply) management console?  I've logged into one I've found by accident using username:admin password:password!

Question known changes - software enhancements and patches, file or db permissions, config changes, etc. I've seen issues where a changed file path will take down a critical Ftp routine, a 3rd party software patch for a document management system crippled a production system by removing key indexes in the database, and an (unfortunately un-automated) database refresh will have missing roles or users or the db will still be restricted mode.  Doing anything manually can get you into trouble.  A fat finger can push a wrong dll/library or mis-type an entry in a config file.  Even fat fingering an automated deploy can get you into problems.  Automated deploys still depend on data that is manually entered.  We have pointer to app servers in our web.config files entered wrong and multiple entries in machine.config for a particular component.

Question security - Malicious changes are possible either internally or externally.  Unfortunately, the question of security is a larger one everyday.  And it has to be scrutinized at every level.  Does your system have a firewall and IP Sec rules in place?  Does you application provide you with an audit trail?  How secure is your production data? 

Resources at your disposal that can help in your investigation:
Log files - event logs, server logs, if you're logging to tables in the DB, don't forget to look there.
Users - it's like CSI - you need to get ALL the information you can about their problem.  You cannot be afraid to ask.
Thread dumps - killing a hurting server and ensuring that it does a thread dump when it terminates can be very effective in your problem search
Networking tools - WireShark, telnet, ping, netstat - these are great for checking your communication.
Monitoring tools - Nagios, SCOM, GroundworkOpenSource
Books -  Michael Nygard's Release It! and Luke Hohmann's Beyond Software Architecture.

Google Tools post

Good link here that lists the top Google Tools for website developers:
http://sixrevisions.com/tools/the-top-15-google-products-for-people-who-build-websites/
Personally, I start to get lost in the plethora of tools out there. Just getting Adwords properly integrated with Analytics is a headache I don't really have time to deal with (when I have so many other higher priority things to manage). I wish some of these tools where a little more straight forward to use.
My hesitation in making the investment to learning one well is that it will change too soon and too much for me to keep up with (I do have a 9-5 job along with at least half a dozen clients on the side). I need to focus my attention on where I get the most return on my time investment - and learning how every single Google Tool works doesn't rate that high.
It is a good post for reference, though. I like knowing what all the different tools are out there.

More lessons learned (not mine though)

Here's another good link to a lessons learned article: http://java.dzone.com/articles/lessons-learned-taking-project It talks about how to communicate, what to have in place, and how to manage client expectations on an 'emergency fix' project. I have a special interest in this type of thing as I'm testing a new business idea here: http://www.netfocusconsulting.com/Experts.jsp