Facebook Job Interview - Production Engineer

Facebook contacted me on LinkedIn recently looking to fill a 'Production Engineer' role.  I wasn't looking for job offers.  They reached out to me.   Apparently they had been doing this a bit though, targeting DevOps professionals as my buddy at the startup I was recently working with also got contacted.

Having Facebook reach out to you for a job opportunity?  Definitely intriguing.  One of the things on my 'IT Career Bucket List' would be to work at one of 'those' companies.  Google, Facebook, etc.  I thought if nothing else, it would be interesting to see where the hiring process went, since in the past they didn't actively recruit unknowns like me. 

I replied that I'd be interested to know more and so a phone interview was arranged with the Facebook technical recruiter.  At the appointed time (a couple days later) he called and I got the skinny on how things would potentially work.

The phone interview was the first step.  After that I provide them with my resume which some of their technical leads would look at.  If they felt I was a fit based on my resume, there would be two remote technical screens - 1 specifically focussed on the Linux OS, and the other on a coding language of my choice.  They wanted someone who was very comfortable at both.  Over the phone they gave me some sample questions - basic linux commands - to give me an idea of what the screens would be like.  If I managed to gain their approval in the technical interviews, Facebook would then fly me to the location of my choice (Menlo Park or Seattle) for face to face talks - both technical and otherwise.  Following that, if I was still up to snuff, I'd get an offer.  Once hired, there would be a 6 week on site 'boot camp' where I'd get trained in all things Facebook and brought up to speed the technical ins and outs of the team I'd be working with.

Getting hired would require me to move to Seattle or Menlo Park.  Real estate at both locations is exorbitant - like ridiculous.  In the event that I was hired, Facebook would offer me a full relocation package, with the potential for a temporary housing situation for 3-4 months while we looked for a permanent residence.  I was told that many employees commute from communities with more reasonably housing prices using Facebook commuter busses that have complimentary drinks and WIFI.  Health and Dental would be covered 100% for myself and my dependents.  Wednesdays are an optional work-from-home day, and Facebook offers 21 days of vacation per year - although I neglected to confirm whether that was 21 business days (over 4 weeks), or 3 weeks all in.  I'm a Canadian, so I asked about a work Visa.  He replied that Facebook has an immigration/legal department that would arrange a TN1 Visa for me, in the event that I was hired.

As far as the job itself - Facebook's version of the Production Engineer role is essentially described here.  At the time I was talking with them, Facebook had 42 teams each responsible for a particular feature in their system (FB messenger, Ads, Newsfeeds, etc.).  These teams consist of 4-5 developers with an embedded production engineer.  Everyone on the team goes on call, one week at a time, so it ends up being a 4-5 week on-call rotation.

In the end, after viewing my resume they decided not to pursue the hiring process with me further at this time.  They had been interviewing 'a lot of strong candidates recently that they felt were a stronger match for their immediate needs.'  I can't say I was heartbroken.  It would have been a big move for us that would have put pressure on me personally and financially - not to mention having one dependent in university and one in high school.

I have been pondering what it was about my resume that flagged it to the hiring managers.  Was it the fact that I've moved jobs every 2-3 years, and they had concerns that I wouldn't last long at FB?  Perhaps it was because of my lack of focus in technology - moving contracts 2-3 years means learning lots of new technology and never getting a chance to really focus.  I've asked the FB recruiter - we'll see what he comes back with.  (You might be wondering why I've moved jobs every 2-3 years...  Its prudent for independent business contractors like myself to 'keep moving' from a Canadian taxation perspective.)

My Path to the AWS Certified Solutions Architect - Associate Exam

Its been a while since I've tried to get any kind of industry certification. Life was busy. I was full-time consulting and had clients to support after work as well. Why would I even bother with all the study? Does a certification amount to much anymore? Lately, several things converged to motivate me to write this cert exam...

1. My resume was getting stale. While I was able to keep a steady stream of contracts going, many of them were using older technologies that weren't current. This concerned me.
2. I got a new gig at a start-up that gave me a hands-on opportunity to work with newer technology in the cloud. After setting up the infrastructure and CI/CD for several QA environments and geo-load balanced UAT and PRD environments in the Google Cloud Platform, a requirement for hosting our data in Canada became paramount. AWS had recently launched data centres in Canada, and so in May we decided to migrate our infrastructure there. Having successfully completed that migration I wondered how easy it would be for me to follow up with a Solutions Architect certification.
3. A couple of guys at work got me turned on to udemy.com. Catching some sales I was able to get a couple of courses on AWS certification for $15 each (regularly they are over $150 each). One course was by ACloudGuru for the AWS SysOps certification. The other was by the Linux Academy for AWS Solutions Architect certification. These two courses gave me a good foundation for the material that is covered on the exam.

I originally did the SysOps course first, as I thought the that course was more in line with my job description at work. Finishing that course (leaving its practice test for later), I took a look at the practice questions on AWS here and felt like I'd come up short if I wrote the exam. That's when I decided to switch gears and do the Solution Architect course and exam. Most people online recommend doing that one first.

Incidentally, both courses helped me get a better grasp AWS Best Practices, and I was able to implement several improvements to our infrastructure at work because of what I learned. After another week and a half of going through the Solutions Architect course and reviewing the material, I felt more confident. I took the practice tests from both courses and passed them with good marks, so I thought I was ready. I scheduled my exam for the next week and also, for good measure, purchased a 20 question 'official practice exam'.

I had the week off during which I was scheduled to write my exam. Thursday was the big day. Tuesday morning I wrote the 'official practice exam' which was full of scenario based questions - and got 60% - A FAIL! Apparently the passing mark floats a bit between 62 and 66% (go figure). The practice exams in the courses seemed a bit easier. One of the things I didn't realize was how the different 'domains' for the exam were weighed:

In my 'official practice exam' I had scored:

1.0 Designing highly available, cost-efficient, fault-tolerant, scalable systems: 50%

2.0 Implementation/Deployment: 100%

3.0 Data Security: 75%

4.0 Troubleshooting: 50%

Clearly my 100% in implementation/deployment wasn't going to help me much with the domains balanced like that. I had some work to do!

A Cloud Guru has a forum specifically for the exam. I poured of that, specifically looking at posts where people who've written the exam share their experiences and what they wished they would have studied. I made notes of what I didn't know from those posts, and I also went over a lot of official AWS documentation for specific services (mostly FAQs and Best Practices) and took notes from them, too. And then I studied hard. 6 pages of 8 point font text. I kept adding things to those notes as well.

Thursday rolled around and I wrote the exam. I tried my best. The exam is 55 multiple choice questions and you have 80 minutes to complete it. Some of the questions I knew the answer, hands down. Others (like 'choose two correct answers') got a bit dicey. I chose the answers that made the most sense to me. I didn't feel like I was blind-sided by any of the questions, however there were definitely some things I (still) could have studied more (AD/User Federation types, for example). I finished answering the questions with about 15 minutes to spare. I had flagged some questions, so I went back and reviewed them and changed a couple of answers. When all was said and done, I passed and scored 72% - not great, but good enough for a certification. 

Here's how things panned out:

1.0 Designing highly available, cost-efficient, fault-tolerant, scalable systems: 75%

2.0 Implementation/Deployment: 80%

3.0 Data Security: 55%

4.0 Troubleshooting: 80%

I was quite happy with the improvements I'd made in the first and last domains.

Apparently there is quite a bit of content cross-over between the AWS Solution Architect - Associate exam and the other two associate certifications: SysOps and Developer. Potentially I could study a bit and write those fairly soon. However its $150 USD to write the exams, and I'm wondering what difference is in having the 3 certs verses just the 1. I'd be interested to hear your thoughts. Would it be work the extra $300 USD? Personally, I'm content to take a break from studying for now

Google Cloud Platform & Stackdriver Monitoring - First Impressions

I've been working with the Google Cloud Platform at work for a little over a month now, and I'm getting comfortable with it.  I haven't done an exact price comparison with EC2/AWS, but apparently its a bit cheaper.  Based on my last 6 weeks of experience, I'm pretty happy with the pricing so far.  Static IP addresses are free, and their dashboard tells you where you can make hosting optimizations to save money which I think is great.  There's lots of good documentation, and the libraries for adding software are full featured and work well.  While GCP not be as full-featured as AWS, I find their menus and dashboards easier and more intuitive to navigate than AWS.

Hiccups I ran into:
- I can't dynamically add cpu or memory.  I've got to stop the instance and restart it.  Same with disk space.  I'd also like to be able to make some of my disks smaller, but I can't seem to do that either without some kind of reboot.
- If an instance has a secondary SSD drive,  it seems that there are issues with discovering it on a 'clone' or a restart after a memory or CPU change.  I have to log in through the back end (fortunately GCP provides an interface for this) and comment out the reference to the secondary drive in the /etc/fstab file to get it going again.  This seems buggy.

I recently spent a couple of days configuring Google Stackdriver Monitoring this week, and after a

couple of frustrations with not being sure how to get started, I installed an agent on a server and I was off to the races.  Installing agents is definitely the way to get going.  I found that it was easiest to create Uptime Monitors and associated Notifications at the instance level, on a server by server basis.  Doing it this way allowed me to group the Uptime Monitors to the instance, something I couldn't do when I created the Uptime Monitors outside of the instance.  It was simple to monitor external dependency servers that we don't own, but need running for our testing.  I integrated my Notifications with HipChat in a snap.  I also installed plugins for monitoring Postgres and MySql - these worked great so long as I had the user/role/permissions set correctly.  I'm super impressed with Stackdriver Monitoring, and will probably use it even if we have to switch over to host with AWS.

Our biggest roadblock with the Google Cloud Platform currently is they don't have a datacentre in Canada.  That could be a big selling feature for many of my clients, and because they don't have it, we may have to consider other options (like AWS/EC2, which is spinning up a new data centre in Montreal towards the end of 2016...)  Privacy matters!  Hope you're listening Google...

EC2 and the AWS (Amazon Web Services) Free Tier - My First Experience

The Amazon Web Services Management Console

I had my first go-round with EC2 in AWS this last week in a real-life context.  I was teaching a class on Content Management Systems at SAIT, and I wanted my students to experience what its like to install WordPress and MySQL on a Linux VM.  I also wanted to personally get some experience with AWS, so I thought 'why not kill two birds with one stone?'

About 6 weeks ago I had purchased Amazon Web Services IN ACTION, written by Andreas and Michael Wittig and published by Manning.  It gave me a great primer on setting up my AWS account, my Billing Alert, and my first couple of VM's.  I leveraged that experience, crossed my fingers, spun up 18 VM's for my students, and hoped I wouldn't get charged a mint for having them running 24/7 for a few days.  It was a Friday around 1pm when I created them and gave them to my students to use.  Imagine my surprise when I checked my billing page in AWS on Monday and discovered they had only charged me $2.81!

I had clearly reached some kind of threshold as after that day I got charged, on average, about $8/day - for all 18 VM's.  They were charging me 2 cents per hour per VM, and some small data transfers.  Granted, I used a 'T1 Micro' - with 1 CPU, .613 Gib memory and 8 GiB storage.  Still, I was quite happy.

On the last day of class, I split the students up into two teams and gave them large web projects to do, and spun up a 'T1 Micro' for each team.  I gave them some scripts they could run to create some swap memory if they needed it.  They ran those scripts right away, and within an hour (with 9 people uploading files and content into those systems continuously) those T1 Micros CPUs pinned.  So I quickly imaged them over lunch and spun up an 'M3 Large' VMs (2 CPU's, 7.5 GiB memory) for each team and threw their images on.  I ran into one issue spinning up the new team VM's - I had to spin down the original Team VM's first before I could start the new ones because there is a limit/quota of 20 VMs on the 'free-tier' in AWS.   Aside from that and the changed IP addresses, the transition was seamless.  I was a happy camper and now that the students had responsive VM's, so were they.

My total bill for the week - $38!  A colleague pointed out that I probably could have added auto-scaling to those team VM's and increase the CPU and memory in place, without losing the current IP's.  He's right, I probably could have, but I didn't have the experience and didn't want to waste class time (and potentially lose the student's work) by trying something I didn't know how to do.  All in all, I was very impressed with my first run of EC2 in AWS.  It was very reasonable, responsive, and easy to use.  I'd definitely do it again.

The Perils and Pitfalls of OpenSource Software

Open-Source Software is the backbone of most of the internet.  Seriously.  For more than a decade, small business websites and main-stream web applications have used open-source software to develop their solutions.  Web projects (and frankly, most software projects in general) that don't have a dependency of some kind on an open-source component are the RARE exception to the rule.

Should we be concerned about this?  I think so.  Here's why:

1. Coders aren't implementing open-source code properly.  In my experience, any open-source code dependencies should be referenced locally.  Many coders fail to do this and reference external code libraries in their code.  What happens when that external 'point of origin' has a DNS issue, or is hit with a DDOS attack, or is just taken down?  Your site can break.
   
   Case in point - check out this article on msn.ca 'How One Programmer Broker the Internet'  In a nutshell, one open-source programmer got frustrated with a company over trademark name issue.  This developer's open-source project had the same name as a messaging app from a company in Canada.  He ended up retaliating by removing his project from the web.  It turned out his project had been leveraged by millions of coders the world over for their websites, and once his code was removed, their websites displayed this error:
            npm ERR! 404 'left-pad' is not in the npm registry
   I believe if developers had downloaded the NPM javascript libraries and referenced them locally on their servers, they wouldn't have run into this issue (as they'd have a local copy of the open-source code).
   
   Another case in point - I worked on a project a number of years ago that had dependencies on jakarta.org - an open source site at the time that was hosting a bunch of libraries and schemas for large open-source projects like Struts and Spring.  Some of the code in those projects had linked references to schemas (rules) that pointed to http://jakarta.org.  Unfortunately we didn't think of changing those links and one day jakarta.org went down....  and with it went our web site, because it couldn't reference those simple schemas off of jakarta.org.  After everything recovered we quickly downloaded those schemas (and any other external references we found) and referenced them locally.
   
2. Security  Do you know what is in that open-source system you're using?  Over 60 million web sites have been developed on an open-source content management system called WordPress.  Because it is open source, everyone can see the code - all the code - that it is built with.  This could potentially allow hackers to see weaknesses in the system.  However, WordPress has pretty strict code review and auditing in place to ensure that this kind of thing doesn't happen.  They also patch any issues that are found quickly, and release those patches to everyone.  The question then becomes:  Does your website administrator patch your CMS?
   
   Another issue I ran into related to security was with a different Content Management System.  I discovered an undocumented 'back-door' buried in the configuration that gave anyone who knew about it administrative access to the system (allowing them to log into the CMS as an Administrator and giving them the power to delete the entire site if they new what they were doing.  Some time later, I found out that some developers who had used this CMS weren't aware of that back door and left it open.  I informed them about it, and they quickly (and nervously) slammed it shut.   Get familiar with the code you are implementing!
   
3. Code Bloat (importing a bunch of libraries for one simple bit of functionality)  Sometimes developers will download a large open-source library to take advantage of a sub-set of functionality to help save time.  Unfortunately, this can lead to code bloat - your application is running slow because its loading up a monster library to take advantage of a small piece of functionality.
   
4. Support (or the lack of it)  Developers need to be discerning when they decide to use an open-source library.  There are vast numbers of open-source projects out there, but one needs to be wise about which one to use.  Some simple guidelines to choosing an open-source project are:
• How many downloads (implementations) does it have?  The more, the better, because that means its popular and likely reviewed more.
• Is there good support for it?  In other words, if you run into issues or errors trying to use it, will it be easy to find a solution for your issue in forums or from the creators?  
• Is it well documented?  If the documentation is thorough, or if there have been published books written about the project, you're likely in good hands.
• Is it easy to implement?  You don't want to waste your time trying to get your new open-source implementation up and working.  The facilities and resources are out there for project owners to provide either the documentation or VM snapshots (or what have you) to make setting up your own implementation quick and easy.
• How long has it been around?  Developers should wait to implement open-source projects with a short history.  Bleeding-edge isn't always the cutting edge.  Wait for projects to gain a critical mass in the industry before implementing them if you can help it.

Random Thoughts About the DevOps Movement in IT - Part 2

The Phoenix Project (image of the book on the left) was recommended to me over a year ago as a good read about DevOps.  It emphasizes the benefits of Dev Ops to management, shareholders, and a company.   Impressions I had of the book were:

• Its a great, fun read, however it skims over the difficulties and trials of the actual automation of the technical systems and software development process - where the rubber really hits the road.  It seems to practically romanticize the idea of automation in IT a bit - like DevOps is the goose that will lay your golden egg.  Unfortunately, there's significantly more work to get that golden egg in my experience.
• It also glosses over how to get the Security team on board with what DevOps wants to do.  In many companies, the Security team holds the trump card and if they decide to change all your certs from 128 bit encryption to 2048 bit encryption (don't laugh, I've seen it happen and we had to regenerate certs for all applicable servers in all envs).  Their wish is your command unless you can convince someone influential that that kind of encryption is overkill in a non-prod environment.

Finding and Cracking That Golden Egg
Hurdles I've encountered enroute to the DevOps 'golden egg' are:

• Silo-ed Application Projects.  
• Lack of consistent naming conventions for deployment artifacts and build tags/versions is majorly detrimental to the automation process. 
• Lack of understanding of the dependencies between application projects (API contracts, or library dependencies).  If you don't understand your dependencies between applications, they may not compile corectly, or they may not communicate properly with each other.
• Lack of consistent development methodology and culture between teams.  If you have one team that doesn't get behind the new culture, and they insist on manually deploying and 'tweaking' their code/artifacts after deployment, they risk the entire release.  Getting everyone on board with culture is challenging.  
• Overusing a tool.  When you have a hammer (a good DevOps tool), everything is a nail.  Generally, many of the DevOps tools out there all have their niche.  You could potentially use Chef, Puppet, or Ansible to do all your provisioning and automation.  Is that the best solution though?  I'm inclined to say no.  How much hacking and tweaking are you having to do to get all your automation to work with that one tool?  Use the tools for what they are best at, what they were originally made for.  Many of these tools have Open Source licenses and new functionality is being added to them all the time.  While it might seem that the new functionality is turning your favourite tool into the 'one tool to rule them all',  you might have a big headache getting there, trying to push a square peg in a round hole.  
• Lack of version control.  Everything must be stored in a repository.  The stuff that isn't will bite you.  VM Templates, DB baselines, your automation config - it all needs to go in there.
• DB Management.  All automated DB scripts should be re-runnable and stored in a repo.  
• Security.  How to satisfy the Security Team?  How to automate certificate generation when they are requiring you to use signed certs?  How to manage all those passwords in your configuration?
• Edge Cases.  Any significantly sized enterprise is going to have/require environments that land outside your standard cookie cutter automation.  Causes I've seen for this are:
• Billing cycles - we needed a block of environments that were flexible with moving time so QA didn't have to wait 30 days to test the next set of bills and invoices.  
• Stubbing vs. Real Integration - Depending on the complexity of integration testing required, there may be many variations of how your integration is set up in various environments - what end-points are mocked vs. which ones point to some 'real' service. 
• New Changes/Requirements - Perhaps new functionality requires new servers or services.  This can make your support environments look different that your development environments.
• Licensing issues.  When everything is automated, it can be easy to loose track of how many environments you have 'active' versus how many you are licensed for.  License compliance can be a huge issue with automation - check this interesting post out 'Running Java on Docker?  You're Breaking the law!'
• Downstream Dependencies   This is where the Ops of DevOps comes into play.  Any downstream dependencies that your automated system might have need to be monitored and understood.  You can't meet your SLA with your client if your downstream dependencies can't meet that same SLA.  Important systems to consider here are: LDAP, DNS, Network, your ISP, and other integration points.
• YAML files.  Yes, perhaps they are more terse than storing your deployment config in XML.  However, I'm at a bit of a loss to see how they are better than a well named and formatted CSV file.  Sure, you can 'see' and manage a hierarchy with them.  But you can do the same in a CSV file with the proper taxonomy.  YAML files utilize more processing power to parse and have extraneous lines in them because their trying to manage (and give a visual representation of) the hierarchy of the properties.  I've seen YAML files where these extra lines with no values account for a significant percentage of the total lines in the file, making the file more difficult to maintain and prone to fat fingers.  Several major DevOps tools use these files and I really can't see a good reason why except that they were the 'new, cool thing to do.'

Random Thoughts About the DevOps Movement in IT - Part 1

DevOps related books I have in my library currently

Almost 5 years before DevOps was given the name it has today, I helped implement a modern DevOps solution at a company I was consulting with.  While our enterprise wasn't huge, and we didn't have a lot of integration (compared to some) it was a pragmatic solution that used Kickstart and Ant with a PHP/Mysql database.  We had Continuous Integration (CI) and Continuous Delivery (CD) - members of our QA team could select an environment, select a build, press a button, and the deployment happened.  Our environments at the time consisted of JBoss containers on Unix machines, and IIS web-apps and services on Windows machines.

Software development methodologies and practices have evolved a lot since then.  It's been a challenge to keep up.  I'm a pragmatic guy, and I thought one could get pretty much all the functionality needed for an automated build and deployment stack just using Ant, CruiseControl, Kickstart, VMWare, and some helpful API's.  Clearly I was wrong.

Cause for Pause
Without a doubt, the DevOps movement has captured the imagination of many a software developer.  Just look at all the tools out there now!  Chef, Puppet, Bamboo, RunDeck, Octopus, Docker, TeamCity, Jenkins, RubiconRed, Ansible, Vagrant, Gradle, Grails, AWS...  I could go on.  I'm beginning to question whether or not the polarization and proliferation of these tools has been helpful.  I find many companies looking to fill a Devops role are asking for resources who have experience with the specific tool stack they are using.  That must make human resource managers pull their hair out.  Even personally, I'm concerned about hitching my wagon to the wrong horse.  If I decide to accept a position with a company who is using a Chef/Docker stack (for example), but the industry decides that Ansible/Bamboo is the holy grail,  have I committed professional suicide?  Probably not, but it does make one consider job opportunities carefully.

Looking Forward
This P and P (proliferation and polarization) of DevOps tools makes me wonder what the future is going to look like for the DevOps movement.  Consider what Microsoft did with C#.  Instead of continuing to diverge and go their own way with the replacement for VB, they created a language with a syntax that essentially brought the software development industry back together (in a way).  Brilliant move.  Java developers quickly ported their favourite tools/frameworks over to C# and suddenly developing with Microsoft tools was cool again.  It would sure be nice if something like that happened with the DevOps tools.

Something else to consider for the future...  so far in my experience with overseas, outsourced teams, they have not yet embraced the DevOps movement.  As a result, they aren't as efficient or as competitive as they could be.  When they do jump on the DevOps bandwagon and truly tap into it's potential, it could be a game changer for people like me....

Tools to Watch

• Perhaps Amazon is the 'new Microsoft' with its expansive and ever expanding AWS tool stack?  There's definitely some momentum and smart thinking going on there.  They appear to have automated provisioning all wrapped up in a bow. 
• Atlassian is another company (I didn't realize they were based out of Australia) that is putting together DevOps tools that have a lot of momentum in the industry.  Their niche is collaboration tools.
• Puppet and Chef both have a strong foothold in the DevOps community.  Both of them are adding new features all the time, enabling them to automate the deployment and provisioning of more products and systems.  Many people use Puppet and Chef in conjunction with RunDeck or Docker to get all the automation they are looking for.

Teaching an IT class (JavaScript, XML, C#, VB, CMS/PHP)

I've taught part-time at SAIT for 11 years now.  I've had more than 30 different classes with 5 different courses.  Recently I've been challenged by a colleague (also a part-time teacher here at SAIT) to make my courses better and more interactive.  While I thought my courses were pretty interactive to begin with, his ideas have definitely pushed me further in a good way.

All the courses I've taught have been introduction to different code languages or working with code:
Visual Basic, C#, XML, Javascript, PHP, and Content Management Systems like WordPress and Drupal.  Up until recently, my courses had been - at a high level - structured as follows:  I demonstrate a new concept/code idea to the class and I have them follow along.  Once they are reasonably comfortable with implementing it (perhaps an exercise or two of following along like this) I give them an exercise that they have to try and do themselves.  Sometimes I'll add a twist to make them think a bit.  For the most part this has worked great.  Its looked like this (an example with XML):

• Intro to the Language and syntax - I'll go through several power point slides that discuss: 
• What does well formed XML mean?  
• How do you ensure your XML is well formed?  
• What are the keys/rules to well formed XML? 
• I'll ask the class questions as we go along and get them to answer.
• We'll code a simple XML file together.  I'll make intentional mistakes along the way and see if the class is paying attention, or just call them out and ask 'Do you think this is right?'  I will also put comments in my code so the students can refer to them later.  Once done, I'll give this complete exercise to the students so they have it to refer to.
• Then I'll give them an XML file that is not well-formed (completely broken) and get them to try and fix it.  
• After about 15 minutes, I'll try and fix it myself in front of them, getting them to help me.
• Once that's done with my comments, I'll again put it in a shared folder for the students to grab so they can refer to it later.
• I'll give them another broken XML file and get them to fix it. 
• Again, after about 15 minutes, I'll try and fix it myself in front of them, getting them to help me.
• Once that's done with my comments, I'll again put it in a shared folder for the students to grab so they can refer to it later.
• Then, next concept.... and repeat.

This has worked quite well.  I've followed this process with all my courses and it seems to keep students engaged and helps them grasp new concepts.

As I mentioned earlier, I've been challenged by a colleague to make my courses better, even more collaborative, and (hopefully) gets the students more return on their tuition investment.  Here's what I'm changing and improving:

• Using a Repo.  Instead of using a shared folder to give files to students and have them turn in assignments, I'm using BitBucket with SourceTree (and forcing them to use it too).  This is great practice for them for the industry, and so far has worked really well.  My SourceTree client does get bogged down at times (with over 30 repositories) but it hasn't been a major deal.
• Commenting. I continue putting lots of comments in my code for the class to refer to.  They really like this and find it helpful when they review after class.  I'll definitely keep doing this
• Error Challenge.  I tend to fat-finger a bit when I'm coding in front of the class (even when I'm working off of a printed copy of a complete exercise).  Sometimes these errors are hard to catch and I waste some class time while I try and figure out what I did.  So instead of waste our collective time and to help them pay attention, I instituted a little game.  If a student catches an error I make in my code, they get a point.  Points accrue for the duration of the course.  The student at the end of the course with the most points gets a free Starbuck's gift card (or something like it).  This worked really well, and I'm definitely going to do it again.
• Student Journal. Student feedback is key for me.  I teach in 'Fast-Track' programs where the students know that they are going to get inundated hard and fast with new concepts all the time.  Ensuring I'm not bull-dozing them with new stuff is key, so having a feedback loop helps.  Up to this point I've just been asking them in class 'How's the speed - am I going to fast?'  But my colleague suggested getting the students to journal and having them  commit that to their repo so I can see it as another feedback loop.  I did this in my last course and it worked quite well.  I don't have them journal every day - just once a week or so, but the feedback I got from them individually was great. 
• Group Projects.  Out in the industry, most of these students are going to be working on teams, so having them do exercises in class as teams seems logical.  It turns out that students, for the most part, really like having group projects to work on.  They can collaborate together and learn from each other.  It also pushes them to compete a little which makes for an even better result.  I have them create a team repo for the project(s), and this also gives them experience with some BitBucket administration (branching and merging, etc.) which also prepares them for the industry.

IOException - Disk Quota Exceeded error - yet there's lots of disk space

I use a Mac for work.  I've run into an interesting issue with making backups of client web sites and zip files with my Mac...  It seems that whenever I've zipped a file and backed it up to my Mac, I get these wonderful __MACOSX folders - that's 2 underscores at the beginning of the folder name injected into the folder mix.

Restoring a site from one of those backups on a linux machine can have adverse consequences because of those 'injected' __MACOSX folders.  It seems that certain flavours of linux really don't like that folder name and as a result will not write a single file to the machine any more.... at all.  Instead, it throws an a 'Disk Quota Exceeded' error.  You can try touching a file, you can try unzipping some other files - whatever you do, you'll get that error.   The error didn't go away for me until I deleted all __MACOSX folders that got copied up to my machine in the zip file.  Once I did that - it was all good.

The interesting thing about this error was I could have tons of disk space still on the machine, and yet it would still throw that error until I removed those offending __MACOSX folders.

There are various commands out there for removing these folders from zip files created on a Mac... something like this:
zip -d your-archive.zip "__MACOSX*"

The Value of IT Certifications and Skills

After I've told my students at school how I finally managed to get a full time job in IT, they've asked me what I thought was the most important thing I did to get that job.  I've always answered 'getting my SCJP certification' (Sun Certified Java Programmer). It's been over 10 years since I started working at the job.  While I still maintain certifications are important for developers who are trying to break into the industry, I've become less enthusiastic about them and what they mean now that I have more experience. 

Having said that, I ran into a couple of articles today that I thought were very interesting:
15 IT Certifications That Can Get You a $100,000 Salary
30 Tech Skills That Can Get You a $100,000 Salary

Granted, the way these articles are written is 'just a little bit of' hype.  These skills and/or certifications aren't going to net you a 100k salary all on their own.  They need to be married with a resume that has several years of IT experience and related, well developed soft skills.

IT New Hire/Contract? Thoughts on IT On-Boarding

Being a contractor, I get to see and experience the on-boarding practice of different companies in different industries on a regular basis. This gives me an opportunity to provide a 'value-add' for the companies that I'm contracted with. Because of my fresh perspective, I can more easily pinpoint areas of opportunity for improvement in the first week on the job.

One thing that has always surprised me, and remained fairly consistent in all of the companies that I've worked with is that they don't seem to have the infrastructure ready for new hires.  I always seem to spend the first week doing one of the following:

• waiting for a computer or even a desk to sit at.
• waiting for access to my computer or to files, folders, applications, servers, etc.
• setting up/configuring my computer - installing required applications, setting up internal website bookmarks in browsers, etc.
• waiting for licenses for the applications I need to run to do my job.

I find this lack of organization dumbfounding.  HR knows that a new hire is getting on-boarded weeks in advance.  Similarly, the team/department should know as well.  I don't think there are any real good excuses for having new hires sit on their hands for the first week while everyone gets organized.  Bringing new hires on board should be a Standard Operating Procedure.

While I'm waiting for access or a license, I busy myself with looking for things that people with history at the company would view as the 'status-quo', but are really issues that could/should be considered problematic or improvable.

Opportunities like:

• Single Points of Failure.  (email server(s), monitoring server(s), shared drive(s), VPN access, etc)
• Lack of network segregation between environments (can I ping/telnet from a dev or staging server to production?)
• Misnamed or legacy names for servers or monitoring notifications that are understood by employees but a source of confusions for newbies
• Lack of templates for common documentation
• Lack of exportable outlook email rules that can be shared with the team
• Lack of exportable browser favorites that can be shared with the team
• Consolidating and organizing documentation (critical documentation in 12 different folders scattered through different directories is a recipe for confusion)
• Lack of Automation - continuous integration through to automating deployment.  Automating monitoring
• Preventing Fat-Finger errors (adding confirmation messages in start scripts)

Granted, not all of these will be applicable for every IT position.  Also, share your findings with your new supervisor and get his blessing before going and changing everything.  Suggest with encouragement, don't criticize! You don't to come across like your trying to relieve someone of their position.  You are there to make things better for everyone.

So, next time you are on-boarded for a new IT position, keep these things in mind.  Hopefully they can help both you and your new company.  Also, if anyone has some other suggestions of things to look for, please add a comment!

Dell Latitude Diagnostic Error Code 2000-0142 Hard Drive Failure

'Apparently' the hard drive on my laptop failed yesterday.  Out of the blue I got a couple of popup windows complaining about some dll not being able to load, and then the windows blue screen of death.  I thought maybe my machine got too hot, but I tried rebooting after letting it cool off and it didn't do anything after the Dell splash page during the boot.  So I ran a BIOS diagnostic and got the following error:
Msg: Error Code 2000-0142
Msg: Unit 1: Drive Self Test
Failed Status Byte=77
I called Dell and they basically confirmed what the diagnostic test said - my hard drive was fried.

Not wanting to give up on my hard drive that easily, I googled to see what I could do.  I discovered I could try to repair windows using the Windows Recovery Console.  So I found my Windows CD and booted from it and went to the Repair option.  Once I had entered the Admin password, I got a command prompt and after a bit of searching around I discovered that most of my hard drive seemed to be intact.  This, however, was still in the dos prompt of the Windows Recovery Console.  I couldn't back up my files to a USB from there.

I tried chkdsk, chkdsk /p, and chkdsk /r - those all worked successfully.  Yet my machine still wouldn't book.  So I preused the list of commands available to the Windows Recovery Console (here) and tried a few more - bootcfg, fixboot, and fixmbr.  In the end, it was fixmbr (which repairs the master boot record on the specified disk) that ended up working for me.

My laptop is 4 years old now.  I removed the battery a couple of months ago, and with problems like this starting to happen, I going to start shopping for a new one.  Hopefully this info helps someone out there though.

Access Denied - Windows Service - StreamWriter- VB.Net

I'm going to cut right to the chase on this one.  I didn't realize that Windows obfuscates error messages for security.  Without that gem of knowledge, I spent WAY more time than I should have last week trying to get a Windows Service to write a file to the local file system on a server.  (I have another post related to obfuscated error messages with Windows here, but it focuses on IIS).

I was continually getting an access denied message trying to programatically create and write a file using a StreamWriter in VB.Net.  My Windows Service was running as a particular service account, and I had  given that particular account Modify access to the parent folder where the file was supposed to be created - but it would not write.  I tried pushing to a different server to see if it would work there - still no dice.  I tried adding security policies (run as a service, run as a batch job, run as part of the OS) to the service account running the service - again, the annoying message 'Access Denied'.  For a while it was thought that maybe the file size was an issue - it was a little over a Mb.  I deployed another version that wrote just one line.  No luck.

In the end, I enlisted a colleague's help (thanks Dorina) and we discovered that the problem wasn't the user at all.  It was in the path that I was trying to write to.  I was using the Path.combine() method to put the path together which, in the debugger appeared to put the file name and path together correctly.  However when we used Procmon to look at the output it complained on a createFile command that the 'file' was 'isDirectory'.  Upon closer examination, it turned out that my Nunit test was calling one constructor that correctly used that Path.combine() method, and my Service called another constructor that did not use the Path.combine() method.

So in the end, my AccessDenied error, was really an incorrect path issue.  Windows (apparently) doesn't display the correct error message so as not to advertise potentially sensitive paths.  And I need to ensure that my Nunits exercise the same methods that my implementation classes use.

IT Professional - Know Thyself

What are you good at?  It pays to know - literally.  It will also make you happier and more productive on the job.

The more I work in IT, the more I see where I'm most effective. I also see in comparison where others blow past me in their individual skills.  It's important for multifunctional teams to have a good idea of different team member's areas of expertise.  Leveraging (knowing when and who to ask for a specific kind of help) these team skills can greatly improve the effectiveness of a team.

Here's some real life examples:

I'm rather weak on writing complex sql statements.  I can profile sql, and can get most sql work done.  But if I've got a difficult query I'm working on, I'll go ask my team mate who is an expert on putting together complex sql.  Later on, he'll ask me for help with a visual studio or server/service configuration related question.  Its a good, productive working relationship that keeps everyone happy.  The BA's are happy because the data looks right.  My PM is happy because my task was finished on time. And my team mate and I our happy because we both learned something and did it faster than trying to google for it on our own.

I'm great at problem solving software related issues - particularly in a domain I'm familiar with.  Whether it's missing permissions, configuration, network connectivity, everyone sitting around me will tell you I'm very, VERY persistent at solving problems that are brought to me.  My neighbor one cube west is fantastic at IIS/MSTDC/COM related bug/configuration debugging.   She also knows our software domain better than almost anyone else in the building - she seems to have it memorized.  When I have an issue with our software that I don't know how to fix, or a business related problem in the software,  I talk to her without hesitation.  Conversely, when she is stuck on an issue that she doesn't know how to solve, she will buzz it past me.  Again, a productive working relationship that makes everybody happy.  The business is happy they got their patch as fast as they did.  Once again our PM is happy, etc....

In meetings I will keep quiet and not say very much unless I feel there is an important point being missed.  I'm not super great at arguing a point.  I'm not the person that's always called on to go to a meeting to convince another group of people that 'our way is the right way'.  Part of this is my personality, part of it is my belief that there is multiple ways to get 'a good enough' answer for the problem we are looking to resolve.  However, we have another fellow in our group that will absolutely blow you away with his gift of logical persuasion.  He's the guy that gets sent to those 'convince the group' meetings.  On the other hand, I'm called on when diplomacy and a consensus is required.

Know what you are good at.  Seek to know what your team mate's strengths are.  Become a happier, more productive development team.

Newbie Advice

Having gone through an IT technical school, I realize they push a lot of information and technology at you. There's a lot to assimilate in less than a year (depending on the school). On this page I'd like to point out some technologies that perhaps weren't covered much in classes that are widely used by good shops in the industry, and also offer some career advice for the future. Before I delve deeper, let me say that this information is based on my experience, which is mostly on the Java technology side of the fence.

Technical Heads Up

1. Design Patterns. One thing many of the technical schools don't have time to cover in their curriculum is code design patterns. It doesn't matter if you doing .NET development or Java development, design patterns can really make a difference in organizing, maintaining, and the reusability of your code. A great book to pick up for an introduction on design patterns is published by O'Reilly and called Head First Design Patterns. If you don't have access to the book, a good place to start is to google some popular patterns that are being used in code now. Dependancy Injection is a pattern that is really big right now. Two frameworks that use it a lot are Spring and Google's Guice. Other pattern I've seen used frequently are the decorator pattern, the abstract factory pattern, factory pattern, the singleton pattern, and the proxy pattern.

2. Using a Repository. Using a repository, or a version control system, should be required by every development shop. Frankly, if you're in a company where they aren't using one, either push to get one installed, or start looking for another place to code. A repository is basically an application that functions as the premier source/bank/datahub/repository for all code. This application should be on a server with lots of disk space and accessable to all developers in the company. Repositories allow developers to manage code changes at many different levels allowing many developers to work on the same code base at the same time. Some repositories I've used in the past are Subversion, CVS, StarTeam, and Perforce. Googling any of these will give you more information.

3. Unit Testing. Unit testing is the development practice of coding 'tests' that validate individual classes or 'units' of code, to ensure they are functioning as required. This practice has developed over the past 4-5 years or so and has been found to be very effective in giving developers the confidence to refactor their code and ensure that it's functionality stays consistent over time. Some unit testing technologies are JUnit, JWebUnit, HttpUnit, DBUnit, and NUnit. 

4. Continuous Integration. Continuous integration is a process that automatically rebuilds and tests (with unit tests) an application or code base. Continuous integration employs an application like CruiseControl or Anthill Pro which is configured to automatically check out the code base from the repository, compile it, run it's unit tests, and sometime even deploy it to a test environment. It allows developers to check for integration problems in the code, and warns developers of broken code, allowing and reminding them to fix it ASAP.

Soft Skills and Career Advice

1. Change is GOOD. Once you've gotten your feet wet in the industry, being a Java developer in one company can mean something quite different that being a Java developer in another company. There is such a vast variety of Java technology that two Java developers can have quite different experiences in developing applications. Whole development processes and methodologies can be very different in differenct companies. One company could be flying by the seat of their pants, developing with not backups or repository, and manually pushing out code, while another is totally immersed in Agile methodology and outsourcing everything but development.... etc. Keeping this in mind, my advice is to not get comfortable in any one company but move around a bit - that is, after 18 months or two years at one place, get a different job. You'll be put into a position where you'll either learn a lot of new stuff, or you'll be mentoring. Either way, it'll be an experience that will likely teach you more then if you had stayed put. It'll also give you some motivation to brush up on your resume writing, interview skills, and expand you network. 

2. Walk & Talk While software developers, for the most part, end up working in cube farms, it should not mean that they need to stay in their cubby hole all day and do work. This may be just my style, but I find when I'm working on a big integration project or on a project with a lot of departments or teams involved, getting up and talking to people really helps things move along. Whether it's getting socializing a new means of managing your server configuration in the code base, or talking to a DBA about a script that you'll need to release with you new code, talking face to face is far more effective than emails, I've found. I generally use emails if I can't find the person, or I want to make sure there is hard documented proof about a decision that is being made, or if I want to include a bigger audience in my communication. Successful software development is more about relationships that a lot a people realize. Relationships are more apt to be cultivated face to face... plus it's good exercise.

 

Career Paths in IT

I've been teaching at SAIT again this week.  Generally before every class gets going in the morning I like to do a little 'blurb' on concepts or thoughts that aren't covered in the course the students get.  One of the things I find is that the vast majority of students (and the population at large for that matter) don't realize how many different career paths there are in just the software side of IT.  Guaranteed there are some students in the course that I teach who, by the time I get the chance to teach them, are very concerned they made the wrong decision by getting into software programming.  They don't feel like they are cut out for programming.  I try to put their fears at rest by suggesting that there are other opportunities for them - careers that can utilize the coding experience that they are getting, but allow them to use talents that they are more comfortable with.  Here's a list (by no means exhaustive) of some other opportunities in the software field.

- Project Management.  Project managers who have coding experience or at least a technical background are a step ahead of PM's who aren't techies.  I myself have considered taking some project management courses or getting my PMP, but thought the better of it after seriously thinking about what I like to do at work.  PMs need to have a great soft-skill set.  While I don't think I'd have a problem with that, I would missing troubleshooting/solving problems.  They also have to be politically savvy and have patience for meetings.

- Business Analyst.  BA's also need to have a great soft-skill set, as they need to be talking with everyone (the business, the developers, and the PM) and writing down the requirements.  It also helps if they are technically competent, and they enjoy documenting/writing.  For some reason in Calgary right now, there have been a lot of postings for BAs over the past 6 months. I just found out today that a variety of institutions offer certificates/diplomas for a Business Analyst career path. (NAIT and Mount Royal University are two local options)

- Quality Assurance.  This field has been growing a lot over the past 8 - 10 years, and frankly I don't think projects that I've been on over the past 5 years would have been successful without our QA teams.  QA experts need to have a penchant for detail and process.  They need to be able to understand the business and relevant business rules well.  Most of the QA people I've worked with haven't had a certification, but you apparently can get more info on a software QA certification here.

- Release Management.  This is what I (like to think I) specialize in.  We ensure that teams are using a continuous integration server and QA has the builds they need for testing where and when then need them.  We have two main goals: 1. Keep downtime in production to a minimum and 2. Keep lost man days in QA to a minimum.  I like release engineering because I get to see the whole picture - I have to understand the code and it's dependencies, I need to understand the infrastructure, and I'm responsible for the product getting through all the environments to production.  For me (being a bit proactive), this has also lead to automating deployments of database objects and datascripts, server configurations, and code, as well as installing monitoring tools to ensure that every environment is ready to 'rock'.

- Security Analyst.  These dudes make sure the good people have access to everything they need access to, and the bad people are locked out.  They spend a lot of time combing though logs and configuring IPSec rules.  They also perform software audits on code, all kinds of different tests on web sites, and network audits.  I wouldn't mind moving my career this direction in the future as they also need to have a pretty good idea of the whole system/infrastructure/architecture.  Great web sites for more resources on these types of positions are here and here.

Even more specialized positions:
- GIS (Geographic Information System) developer or architect or analyst.  Working with digital 2d and 3d mapping systems.  Having some limited experience with this at the ERCB (deploying mapping systems that display reservoirs,etc in Alberta) I think this is pretty cool stuff.
- Search Engineer (Search Engine Optimization specialist)  I've dabbled in this off and on for my clients.  You definitely need to be on top of your game and know how the search algorithms are changing.
- Business Process Specialist - there's lots of software to specialize in here.  Tibco, Agile Point, SAP, JDEdwards, etc.
- Document Management Specialist - Sharepoint, Edocs, Documentum, etc.  This involves the automated management of documents - not the writing of them, but their storage.  Government bodies and law firms definitely have a need for these kinds of systems.

The importance of soft skills in IT

Soft skills are crucial to being a successful developer in the IT industry.  Some of the soft skills I've found beneficial in my career are:

• Being a good communicator.  It's not just about talking, but listening as well.  Dirareah of the mouth is never cool, especially when a job needs to get done.  Strive for mutual understanding in a conversation. Communicating in the IT world isn't just limited to talking and listening though.  
• Having the discernment to know when to write an email versus actually talking to a person is important as well. I do not recommend doing all your communication through email.  Communication in emails can easily be mis-understood or mis-construed and take on an entirely different meaning and tone than you originally intended.  Emails are great for documenting decisions.  
• Make sure that the content of your email deals with the subject you've put in  your subject line.  For example, having a subject line of 'Quote for project XYZ' but then talking about quotes for that project AND other projects in that email is bad news.  Send you quotes for the other projects in a different email with a more relevent subject line (I know this from experience).
• Developing relationships.  Extending communication a bit further and actually developing relationship with people in the office is huge as well.  Questions like "How was your vacation?", "Where are you going on vacation?", or "What did you do over the weekend?" are great for this. When you've got a relationship in place, there's trust.  Many times I've been granted exceptional privileges and become more productive in my work because I've had a relationship with a key person who trusts me.  These relationships can also help in the future as well when you are looking for that next gig.
• Having patience.  Whether it's letting somebody else go to the front of the 'line' for access to production, or having forbearance when dealing with a difficult client, patience can go a long way to strengthening that 'trust' in your relationships.  Sometimes work (or working with clients) in IT can get frustrating.  Things I do to help me step back and get some perspective are:

• go for a walk
• find something else to do for a while
• talk to someone - use them as a sounding board
• take a day off or go on vacation

• Positive Attitude. I like to keep my workplace fun, interesting, and upbeat.  To promote this around me I'll sometimes:
• Bring baking from home to share
• Audibly rejoice in small victories (getting a solution migrated and compiling from VS2005 to VS2010 for example)
• Write thank-you notes (either hard or soft copy) to others who help me
• Encourage people who are down. We had a large project I was helping with and the lead was concerned that between the time and the technical challenges he had, he wasn't going to be able to finish on time.  I believe he could and I told him so (and backed it up with my help).  In the end, he did finish.
• Being Flexible.  Generally, you are always working for a client.  Invariably, they will change their mind or want an enhancement.  Microsoft will upgrade their software, the US president will change daylight savings time one more time, the CTO will be convinced that Tibco will solve all the business problems again.  I've found that being able to take these changes in stride is crucial to maintaining my mental health (avoid being overcome with frustration).  It's all part of the territory - and frankly, job security.

Single Points of Failure

Within the last 8 months, all of the production servers for a client of mine were moved out of the building they've been in and into a 'bunker', or co-lo. There was a variety of reasons for doing this. As an architect choosing a Co-location provider, and planning the move, you would want to make sure as many systems there are as redundant as possible - power supply, UPS system, networking, etc.

Well, this week everyone painfully discovered a system that wasn't redundant (or if it was, it wasn't as redundant as it should have been). Apparently the HVac systems went down. This resulted in servers getting too hot and consequently having to be shut down. Production servers.

Everything was resolved in a couple of hours, but it just goes to show, it seems that there is always one system that is forgotten.

I've actually run into another situation like this before. It wasn't nearly as big an issue for our shop as it only affected development environments. However, the Calgary airport(!!) along with most of NE Calgary was without a network connection just as long as we were. Apparently a construction crew was using a backhoe doing some digging at an intersection and accidentally cut a main networking cable that supplied most of NE Calgary with it's network connection. Kind of makes you wonder if municipalities should be considering redundant underground networking, doesn't it?

Networking Architects  work hard to eliminate as many single points of failure in their systems as they can.  Some are hard to control though.  Recently McAfee released a virus library update that wasn't tested properly and as a result shut down a plethora of systems across North America.  It thought a Windows dll was a virus (false positive) and sent thousands of systems into a perpetual reboot.  It would be an interesting bit of process engineering to figure out the best way protect your production systems from the most current viruses AND protect them from bugs like this sent out by the 'big' virus scanning companies at the same time.

CPU's and Sql Server

Again in our staging environment getting ready for a release in production, we were seeing unexplained slow response times for our AgilePoint application when compared against production or development envs. After some digging we discovered that our production DB server that serviced the AgilePoint DB had twice as many CPU's as the dev DB server, and 4 times as many CPU's as the staging server.

What I didn't know is that in Sql Server you can assigned/limit how many CPU's the database can use. This setting is in Server Properties on the Advanced tab, and is called 'Max Degree of Parallelism'. Like many of the other settings with Server Properties in Sql Server, 0 equates to 'unlimited' or in this case 'all processors'.

Log parsing/network security tools

I discovered some new tools today that are useful for network security. QRadar from Q1 labs (http://www.q1labs.com/) is a really slick log parsing tool for organizations that are looking to implement a distributed log management offering to collect, archive, and analyze network and security event logs. It then parses this information into graphs and data that you can tune to alert you when things go awry. You can configure it to look at firewall logs, web server access logs, event logs, etc.
Splunk (http://www.splunk.com/) seems like it might be a competitor. At a glance, I'd say that QRadar has a lot more features and might be a lot more expensive.