Facebook Job Interview - Production Engineer

Facebook contacted me on LinkedIn recently looking to fill a 'Production Engineer' role.  I wasn't looking for job offers.  They reached out to me.   Apparently they had been doing this a bit though, targeting DevOps professionals as my buddy at the startup I was recently working with also got contacted.

Having Facebook reach out to you for a job opportunity?  Definitely intriguing.  One of the things on my 'IT Career Bucket List' would be to work at one of 'those' companies.  Google, Facebook, etc.  I thought if nothing else, it would be interesting to see where the hiring process went, since in the past they didn't actively recruit unknowns like me. 

I replied that I'd be interested to know more and so a phone interview was arranged with the Facebook technical recruiter.  At the appointed time (a couple days later) he called and I got the skinny on how things would potentially work.

The phone interview was the first step.  After that I provide them with my resume which some of their technical leads would look at.  If they felt I was a fit based on my resume, there would be two remote technical screens - 1 specifically focussed on the Linux OS, and the other on a coding language of my choice.  They wanted someone who was very comfortable at both.  Over the phone they gave me some sample questions - basic linux commands - to give me an idea of what the screens would be like.  If I managed to gain their approval in the technical interviews, Facebook would then fly me to the location of my choice (Menlo Park or Seattle) for face to face talks - both technical and otherwise.  Following that, if I was still up to snuff, I'd get an offer.  Once hired, there would be a 6 week on site 'boot camp' where I'd get trained in all things Facebook and brought up to speed the technical ins and outs of the team I'd be working with.

Getting hired would require me to move to Seattle or Menlo Park.  Real estate at both locations is exorbitant - like ridiculous.  In the event that I was hired, Facebook would offer me a full relocation package, with the potential for a temporary housing situation for 3-4 months while we looked for a permanent residence.  I was told that many employees commute from communities with more reasonably housing prices using Facebook commuter busses that have complimentary drinks and WIFI.  Health and Dental would be covered 100% for myself and my dependents.  Wednesdays are an optional work-from-home day, and Facebook offers 21 days of vacation per year - although I neglected to confirm whether that was 21 business days (over 4 weeks), or 3 weeks all in.  I'm a Canadian, so I asked about a work Visa.  He replied that Facebook has an immigration/legal department that would arrange a TN1 Visa for me, in the event that I was hired.

As far as the job itself - Facebook's version of the Production Engineer role is essentially described here.  At the time I was talking with them, Facebook had 42 teams each responsible for a particular feature in their system (FB messenger, Ads, Newsfeeds, etc.).  These teams consist of 4-5 developers with an embedded production engineer.  Everyone on the team goes on call, one week at a time, so it ends up being a 4-5 week on-call rotation.

In the end, after viewing my resume they decided not to pursue the hiring process with me further at this time.  They had been interviewing 'a lot of strong candidates recently that they felt were a stronger match for their immediate needs.'  I can't say I was heartbroken.  It would have been a big move for us that would have put pressure on me personally and financially - not to mention having one dependent in university and one in high school.

I have been pondering what it was about my resume that flagged it to the hiring managers.  Was it the fact that I've moved jobs every 2-3 years, and they had concerns that I wouldn't last long at FB?  Perhaps it was because of my lack of focus in technology - moving contracts 2-3 years means learning lots of new technology and never getting a chance to really focus.  I've asked the FB recruiter - we'll see what he comes back with.  (You might be wondering why I've moved jobs every 2-3 years...  Its prudent for independent business contractors like myself to 'keep moving' from a Canadian taxation perspective.)

The Perils and Pitfalls of OpenSource Software

Open-Source Software is the backbone of most of the internet.  Seriously.  For more than a decade, small business websites and main-stream web applications have used open-source software to develop their solutions.  Web projects (and frankly, most software projects in general) that don't have a dependency of some kind on an open-source component are the RARE exception to the rule.

Should we be concerned about this?  I think so.  Here's why:

1. Coders aren't implementing open-source code properly.  In my experience, any open-source code dependencies should be referenced locally.  Many coders fail to do this and reference external code libraries in their code.  What happens when that external 'point of origin' has a DNS issue, or is hit with a DDOS attack, or is just taken down?  Your site can break.
   
   Case in point - check out this article on msn.ca 'How One Programmer Broker the Internet'  In a nutshell, one open-source programmer got frustrated with a company over trademark name issue.  This developer's open-source project had the same name as a messaging app from a company in Canada.  He ended up retaliating by removing his project from the web.  It turned out his project had been leveraged by millions of coders the world over for their websites, and once his code was removed, their websites displayed this error:
            npm ERR! 404 'left-pad' is not in the npm registry
   I believe if developers had downloaded the NPM javascript libraries and referenced them locally on their servers, they wouldn't have run into this issue (as they'd have a local copy of the open-source code).
   
   Another case in point - I worked on a project a number of years ago that had dependencies on jakarta.org - an open source site at the time that was hosting a bunch of libraries and schemas for large open-source projects like Struts and Spring.  Some of the code in those projects had linked references to schemas (rules) that pointed to http://jakarta.org.  Unfortunately we didn't think of changing those links and one day jakarta.org went down....  and with it went our web site, because it couldn't reference those simple schemas off of jakarta.org.  After everything recovered we quickly downloaded those schemas (and any other external references we found) and referenced them locally.
   
2. Security  Do you know what is in that open-source system you're using?  Over 60 million web sites have been developed on an open-source content management system called WordPress.  Because it is open source, everyone can see the code - all the code - that it is built with.  This could potentially allow hackers to see weaknesses in the system.  However, WordPress has pretty strict code review and auditing in place to ensure that this kind of thing doesn't happen.  They also patch any issues that are found quickly, and release those patches to everyone.  The question then becomes:  Does your website administrator patch your CMS?
   
   Another issue I ran into related to security was with a different Content Management System.  I discovered an undocumented 'back-door' buried in the configuration that gave anyone who knew about it administrative access to the system (allowing them to log into the CMS as an Administrator and giving them the power to delete the entire site if they new what they were doing.  Some time later, I found out that some developers who had used this CMS weren't aware of that back door and left it open.  I informed them about it, and they quickly (and nervously) slammed it shut.   Get familiar with the code you are implementing!
   
3. Code Bloat (importing a bunch of libraries for one simple bit of functionality)  Sometimes developers will download a large open-source library to take advantage of a sub-set of functionality to help save time.  Unfortunately, this can lead to code bloat - your application is running slow because its loading up a monster library to take advantage of a small piece of functionality.
   
4. Support (or the lack of it)  Developers need to be discerning when they decide to use an open-source library.  There are vast numbers of open-source projects out there, but one needs to be wise about which one to use.  Some simple guidelines to choosing an open-source project are:
• How many downloads (implementations) does it have?  The more, the better, because that means its popular and likely reviewed more.
• Is there good support for it?  In other words, if you run into issues or errors trying to use it, will it be easy to find a solution for your issue in forums or from the creators?  
• Is it well documented?  If the documentation is thorough, or if there have been published books written about the project, you're likely in good hands.
• Is it easy to implement?  You don't want to waste your time trying to get your new open-source implementation up and working.  The facilities and resources are out there for project owners to provide either the documentation or VM snapshots (or what have you) to make setting up your own implementation quick and easy.
• How long has it been around?  Developers should wait to implement open-source projects with a short history.  Bleeding-edge isn't always the cutting edge.  Wait for projects to gain a critical mass in the industry before implementing them if you can help it.

Random Thoughts About the DevOps Movement in IT - Part 1

DevOps related books I have in my library currently

Almost 5 years before DevOps was given the name it has today, I helped implement a modern DevOps solution at a company I was consulting with.  While our enterprise wasn't huge, and we didn't have a lot of integration (compared to some) it was a pragmatic solution that used Kickstart and Ant with a PHP/Mysql database.  We had Continuous Integration (CI) and Continuous Delivery (CD) - members of our QA team could select an environment, select a build, press a button, and the deployment happened.  Our environments at the time consisted of JBoss containers on Unix machines, and IIS web-apps and services on Windows machines.

Software development methodologies and practices have evolved a lot since then.  It's been a challenge to keep up.  I'm a pragmatic guy, and I thought one could get pretty much all the functionality needed for an automated build and deployment stack just using Ant, CruiseControl, Kickstart, VMWare, and some helpful API's.  Clearly I was wrong.

Cause for Pause
Without a doubt, the DevOps movement has captured the imagination of many a software developer.  Just look at all the tools out there now!  Chef, Puppet, Bamboo, RunDeck, Octopus, Docker, TeamCity, Jenkins, RubiconRed, Ansible, Vagrant, Gradle, Grails, AWS...  I could go on.  I'm beginning to question whether or not the polarization and proliferation of these tools has been helpful.  I find many companies looking to fill a Devops role are asking for resources who have experience with the specific tool stack they are using.  That must make human resource managers pull their hair out.  Even personally, I'm concerned about hitching my wagon to the wrong horse.  If I decide to accept a position with a company who is using a Chef/Docker stack (for example), but the industry decides that Ansible/Bamboo is the holy grail,  have I committed professional suicide?  Probably not, but it does make one consider job opportunities carefully.

Looking Forward
This P and P (proliferation and polarization) of DevOps tools makes me wonder what the future is going to look like for the DevOps movement.  Consider what Microsoft did with C#.  Instead of continuing to diverge and go their own way with the replacement for VB, they created a language with a syntax that essentially brought the software development industry back together (in a way).  Brilliant move.  Java developers quickly ported their favourite tools/frameworks over to C# and suddenly developing with Microsoft tools was cool again.  It would sure be nice if something like that happened with the DevOps tools.

Something else to consider for the future...  so far in my experience with overseas, outsourced teams, they have not yet embraced the DevOps movement.  As a result, they aren't as efficient or as competitive as they could be.  When they do jump on the DevOps bandwagon and truly tap into it's potential, it could be a game changer for people like me....

Tools to Watch

• Perhaps Amazon is the 'new Microsoft' with its expansive and ever expanding AWS tool stack?  There's definitely some momentum and smart thinking going on there.  They appear to have automated provisioning all wrapped up in a bow. 
• Atlassian is another company (I didn't realize they were based out of Australia) that is putting together DevOps tools that have a lot of momentum in the industry.  Their niche is collaboration tools.
• Puppet and Chef both have a strong foothold in the DevOps community.  Both of them are adding new features all the time, enabling them to automate the deployment and provisioning of more products and systems.  Many people use Puppet and Chef in conjunction with RunDeck or Docker to get all the automation they are looking for.

Teaching an IT class (JavaScript, XML, C#, VB, CMS/PHP)

I've taught part-time at SAIT for 11 years now.  I've had more than 30 different classes with 5 different courses.  Recently I've been challenged by a colleague (also a part-time teacher here at SAIT) to make my courses better and more interactive.  While I thought my courses were pretty interactive to begin with, his ideas have definitely pushed me further in a good way.

All the courses I've taught have been introduction to different code languages or working with code:
Visual Basic, C#, XML, Javascript, PHP, and Content Management Systems like WordPress and Drupal.  Up until recently, my courses had been - at a high level - structured as follows:  I demonstrate a new concept/code idea to the class and I have them follow along.  Once they are reasonably comfortable with implementing it (perhaps an exercise or two of following along like this) I give them an exercise that they have to try and do themselves.  Sometimes I'll add a twist to make them think a bit.  For the most part this has worked great.  Its looked like this (an example with XML):

• Intro to the Language and syntax - I'll go through several power point slides that discuss: 
• What does well formed XML mean?  
• How do you ensure your XML is well formed?  
• What are the keys/rules to well formed XML? 
• I'll ask the class questions as we go along and get them to answer.
• We'll code a simple XML file together.  I'll make intentional mistakes along the way and see if the class is paying attention, or just call them out and ask 'Do you think this is right?'  I will also put comments in my code so the students can refer to them later.  Once done, I'll give this complete exercise to the students so they have it to refer to.
• Then I'll give them an XML file that is not well-formed (completely broken) and get them to try and fix it.  
• After about 15 minutes, I'll try and fix it myself in front of them, getting them to help me.
• Once that's done with my comments, I'll again put it in a shared folder for the students to grab so they can refer to it later.
• I'll give them another broken XML file and get them to fix it. 
• Again, after about 15 minutes, I'll try and fix it myself in front of them, getting them to help me.
• Once that's done with my comments, I'll again put it in a shared folder for the students to grab so they can refer to it later.
• Then, next concept.... and repeat.

This has worked quite well.  I've followed this process with all my courses and it seems to keep students engaged and helps them grasp new concepts.

As I mentioned earlier, I've been challenged by a colleague to make my courses better, even more collaborative, and (hopefully) gets the students more return on their tuition investment.  Here's what I'm changing and improving:

• Using a Repo.  Instead of using a shared folder to give files to students and have them turn in assignments, I'm using BitBucket with SourceTree (and forcing them to use it too).  This is great practice for them for the industry, and so far has worked really well.  My SourceTree client does get bogged down at times (with over 30 repositories) but it hasn't been a major deal.
• Commenting. I continue putting lots of comments in my code for the class to refer to.  They really like this and find it helpful when they review after class.  I'll definitely keep doing this
• Error Challenge.  I tend to fat-finger a bit when I'm coding in front of the class (even when I'm working off of a printed copy of a complete exercise).  Sometimes these errors are hard to catch and I waste some class time while I try and figure out what I did.  So instead of waste our collective time and to help them pay attention, I instituted a little game.  If a student catches an error I make in my code, they get a point.  Points accrue for the duration of the course.  The student at the end of the course with the most points gets a free Starbuck's gift card (or something like it).  This worked really well, and I'm definitely going to do it again.
• Student Journal. Student feedback is key for me.  I teach in 'Fast-Track' programs where the students know that they are going to get inundated hard and fast with new concepts all the time.  Ensuring I'm not bull-dozing them with new stuff is key, so having a feedback loop helps.  Up to this point I've just been asking them in class 'How's the speed - am I going to fast?'  But my colleague suggested getting the students to journal and having them  commit that to their repo so I can see it as another feedback loop.  I did this in my last course and it worked quite well.  I don't have them journal every day - just once a week or so, but the feedback I got from them individually was great. 
• Group Projects.  Out in the industry, most of these students are going to be working on teams, so having them do exercises in class as teams seems logical.  It turns out that students, for the most part, really like having group projects to work on.  They can collaborate together and learn from each other.  It also pushes them to compete a little which makes for an even better result.  I have them create a team repo for the project(s), and this also gives them experience with some BitBucket administration (branching and merging, etc.) which also prepares them for the industry.

Get Hired as an IT Newbie Part 1 - Have a Good Portfolio!

I participated in a focus group this past week at SAIT where we discussed their 'Web Developer Fast Track Program' and its relevance to the industry.  The goal behind the meeting was how can we make the course more relevant to the changes and continual evolution of technology that is happening in the industry so students are more prepared for positions when they leave.  The discussion was engaging and took some interesting turns.  Here's what I took away related to portfolios:

Student Portfolios
15 years ago I had gone through a similar IT fast-track program and we had to do portfolios of our work for potential employers.  I wasn't aware of a single employer who looked at my portfolio and as
a result I haven't placed much emphasis on it in my classes.  BIG MISTAKE.

It turns out that prospective employers for SAIT students DO look at their online portfolios.... and generally weren't super impressed.  Here's why:

• Industry attendees said that a student's portfolio should reflect the position or career track that the students are looking to get into.  There should be evidence in the portfolio that the student tried on their own to investigate, explore, and work with technologies and code that interests them.  Posting student projects (with every student having very similar projects) doesn't help anyone in making a hiring decision.
• Prospective employers are not just interested in the technologies that students used, but also: 
• Thought processes that the students went through in completing their projects - why they chose to make certain decisions (for example, use a canned WordPress theme instead of design/develop their own)
• Challenges students encountered in design and development and how they overcame those problems in their journey to the completed project - Employers are interested in soft skills like perseverance,  the ability to google a problem and uncover a solution on your own, the ability to asking for help when you are stumped (not before you've tried googling the problem)... etc.
• Is the student willing to go the extra mile, put in some extra effort and explore interesting tools and technologies outside the classroom?  The industry was quite clear that this was one of the main things they were looking for  - a motivated individual with the right attitude.  This kind of motivation should be clearly seen in a student's portfolio.

There was also some suggestions during the focus group that students should be given the opportunity to provide constructive feedback to their peers on their portfolios.  A good critique is a gift.  Can students accept constructive feedback and use it to improve?  Could providing peer critiques give them a better perspective and more experience with what is good (design, code, functionality, etc.) versus what is not good.  We thought so.

Finally, here's a couple of links to student portfolios that 'make the grade' so to speak, in my humble opinion.  One was a student of mine this past semester, the other is a current student at the University of Calgary:

- Gary S. Jennings (former SAIT student)

- Carrie Mah (U of C student)

IT New Hire/Contract? Thoughts on IT On-Boarding

Being a contractor, I get to see and experience the on-boarding practice of different companies in different industries on a regular basis. This gives me an opportunity to provide a 'value-add' for the companies that I'm contracted with. Because of my fresh perspective, I can more easily pinpoint areas of opportunity for improvement in the first week on the job.

One thing that has always surprised me, and remained fairly consistent in all of the companies that I've worked with is that they don't seem to have the infrastructure ready for new hires.  I always seem to spend the first week doing one of the following:

• waiting for a computer or even a desk to sit at.
• waiting for access to my computer or to files, folders, applications, servers, etc.
• setting up/configuring my computer - installing required applications, setting up internal website bookmarks in browsers, etc.
• waiting for licenses for the applications I need to run to do my job.

I find this lack of organization dumbfounding.  HR knows that a new hire is getting on-boarded weeks in advance.  Similarly, the team/department should know as well.  I don't think there are any real good excuses for having new hires sit on their hands for the first week while everyone gets organized.  Bringing new hires on board should be a Standard Operating Procedure.

While I'm waiting for access or a license, I busy myself with looking for things that people with history at the company would view as the 'status-quo', but are really issues that could/should be considered problematic or improvable.

Opportunities like:

• Single Points of Failure.  (email server(s), monitoring server(s), shared drive(s), VPN access, etc)
• Lack of network segregation between environments (can I ping/telnet from a dev or staging server to production?)
• Misnamed or legacy names for servers or monitoring notifications that are understood by employees but a source of confusions for newbies
• Lack of templates for common documentation
• Lack of exportable outlook email rules that can be shared with the team
• Lack of exportable browser favorites that can be shared with the team
• Consolidating and organizing documentation (critical documentation in 12 different folders scattered through different directories is a recipe for confusion)
• Lack of Automation - continuous integration through to automating deployment.  Automating monitoring
• Preventing Fat-Finger errors (adding confirmation messages in start scripts)

Granted, not all of these will be applicable for every IT position.  Also, share your findings with your new supervisor and get his blessing before going and changing everything.  Suggest with encouragement, don't criticize! You don't to come across like your trying to relieve someone of their position.  You are there to make things better for everyone.

So, next time you are on-boarded for a new IT position, keep these things in mind.  Hopefully they can help both you and your new company.  Also, if anyone has some other suggestions of things to look for, please add a comment!

Access Denied - Windows Service - StreamWriter- VB.Net

I'm going to cut right to the chase on this one.  I didn't realize that Windows obfuscates error messages for security.  Without that gem of knowledge, I spent WAY more time than I should have last week trying to get a Windows Service to write a file to the local file system on a server.  (I have another post related to obfuscated error messages with Windows here, but it focuses on IIS).

I was continually getting an access denied message trying to programatically create and write a file using a StreamWriter in VB.Net.  My Windows Service was running as a particular service account, and I had  given that particular account Modify access to the parent folder where the file was supposed to be created - but it would not write.  I tried pushing to a different server to see if it would work there - still no dice.  I tried adding security policies (run as a service, run as a batch job, run as part of the OS) to the service account running the service - again, the annoying message 'Access Denied'.  For a while it was thought that maybe the file size was an issue - it was a little over a Mb.  I deployed another version that wrote just one line.  No luck.

In the end, I enlisted a colleague's help (thanks Dorina) and we discovered that the problem wasn't the user at all.  It was in the path that I was trying to write to.  I was using the Path.combine() method to put the path together which, in the debugger appeared to put the file name and path together correctly.  However when we used Procmon to look at the output it complained on a createFile command that the 'file' was 'isDirectory'.  Upon closer examination, it turned out that my Nunit test was calling one constructor that correctly used that Path.combine() method, and my Service called another constructor that did not use the Path.combine() method.

So in the end, my AccessDenied error, was really an incorrect path issue.  Windows (apparently) doesn't display the correct error message so as not to advertise potentially sensitive paths.  And I need to ensure that my Nunits exercise the same methods that my implementation classes use.

WCF and Large Result sets

Dealing with oil resource data across several disparate systems and also trying to make our API's more generic and consistent, we use WCF a lot (web services with windows apps). We've had a couple of instances where we've had to tweak the config settings for either our publishers or our clients to return some large datasets over WCF.
For the most part, I have found the error messages relating to constrained WCF config settings refreshingly to the point.  They pretty much pointed me directly to the attribute in the file that I needed to change in order to fix my issue.
For memory related errors on the client, I found that juggling with the maxBufferPoolSize and maxReceivedMessageSize in my binding to help.  I moved the maxBufferPoolSize from 9999999 to 99999999, and changed the maxReceivedMessageSize to 350000000.  Sometimes after that I would get a timeout error on the client machine (depending on what kind of load the server was under).  So I'd have to make the timeout settings on the binding higher.
Obviously it helps when the publisher has the same values for the same attributes on its binding.

Thoughts on Design

I recently re-read The Last Lecture by Randy Pausch and was impressed with what he did with his classes that focused on User Interface Design.  The first day of class he would bring in an old VCR, put it on a table in front of class, and without saying a word he'd pull a sledgehammer from behind his desk and destroy it.  Needless to say it got the class's attention.  The message Randy was trying to convey in this demonstration is that poor end user design is detrimental to the product.  End User Design is critical.

There is another person who is a strong proponent of excellence in End User Design.  He is still alive today and he's be a driving force in perfection in Design in the IT industry.  He designs have been flying off of store shelves for the last 25 years or so.  If you haven't guessed yet, I'm thinking of Steve Jobs from Apple. I ran across a very interesting article about him in Bloomsberg Businessweek.  It's basically an interview with John Sculley, the CEO of Apple from the mid-80's to the mid-90's, and what it was like 'Being Steve's Boss'.  He talks a lot of Steve's philosophy and background in design, and Steve's influence on Apple in this area.  It's a great article.

It got me thinking...  In software I have not been super stringent about software design on the applications I've written.  Let me clarify - object oriented design is a must.  The train of thought I'm seeking to follow here is 'is it a requirement to implement design patterns in your code everywhere possible?' A major reason for this is I've worked with a number of people who think their job is refactoring and renaming classes and packages in a code base - to the determent of the project.  I often tell the students in my classes that there are numerous ways to 'skin a cat' as it were, when coming to a working solution in code.  I wonder though, is there always one perfect way?  Is this something to strive for?  Does it matter as long as the application works the way the client wants it to?  What do you think?

Personally, I think that the answers to these questions depends on a lot of things, not the least of which are your client, the project, and your team.  Elegant solutions look and feel so good when your done - both for you and for you client.  Perhaps the motivation and context for the project need to be analyzed before a decision is made whether or not you should 'shoot for the best designed, most elegant solution' possible?  Obviously if your project has constraints like budget, resources, and time, there's a limit to how much effort you can put into evolving your project into the best design in can be.  That is one big reason why design patterns and frameworks are so prevalent (and pretty much a requirement) in software engineering.

I like to read/sign out Fast Company from the Library - I'm actually thinking about getting a subscription.  Anyway, they have an annual 'Masters of Design' issue that is a lot of fun to read.  In it they highlight different world class designs and their designers.  Here's a link to their MOD tag cloud.

Open Source Bug/Issue Tracking System

In the last couple of weeks I've been helping facilitate the development process (through build & deployment automation, etc) on a new project, and I was asked to spin up a new instance of Trac for the project team. It didn't take me long to get it going.  I was amazed at the rate of adoption of this tool by the team as a whole.  The business was very positive about the tool and had no issues with learning it and beginning to log tickets.  I will definitely keep this tool in mind in the future as it is relatively easy to install, straight forward to use, integrates with Subversion and Active Directory, and fulfills all the basic requirements for a bug/issue tracking system that my project team was looking for.

Some good links (to blog articles) on Configuration Management

Here's some good links to blog articles on Configuration Management.  The first is about the importance of a CM person in a company.  I would tend to agree with one of the comments that talks about the size of the company making a difference.  Obviously a smaller startup doesn't need a dedicated CM person.  However, larger enterprises like the one I'm at now with numerous development environments, multiple servers, a GIS infrastructure, a document management infrastructure, and a beomoth of a mainframe legacy system needs a dedicated CM team.  Here's the link:
http://evgeny-goldin.com/blog/2010/08/21/configuration-management-position/

 The other link considers adding database automated deployment to you CM system.  I would highly recommend when I contrast my experience over the last two years with my previous release management experience.  Previously we automatically deployed database schema and data changes. In my more recent experience the DBA group has flatly declined any encouragement and offer to do this.  The difference in the amount of time to do database refreshes and debug issues with them is stark - like hours vs. days.  Here's the link:

http://www.simple-talk.com/sql/t-sql-programming/when-database-source-control-goes-bad/

.Net Self Serve Automated deploy

There are a couple of prerequisites that need to be in place in order for a self serve automated deployment system to work.
1. You should have a Nant, MsBuild, Visual Build, or a similar type of scripting file that is doing deploys for you already - see my previous posts on doing that here
2. Your folder structure for every project you want to deploy should be similar. To have a good Self-Serve Deploy tool, it needs to work on the assumption that directory structures are consistent for certain types of projects. This could be set up using your repository tags directory, or automatically copying deployment files to the required folder structure after the build is successful. A folder structure I've used in the past is b:\deployment\projectname\version\ My build file(s).  Any other required folders and files reside in the version folder with a consistent folder structure.
3. You need either a properties file or a DB table to manage which tag/version has been deployed to which environment. We've used a symlink in the past to denote the current 'tip' build. This example will use a file. The file is structured like:

<versions>

   <serverType>

       <server name="blahboxType" version="1.2" />

       <server name="fooboxType" version="1.0" />

   </serverType>

   <blahboxType>

       <project name="MyProject" version="3.1.0.23" />

       <project name="MyProject2" version="3.1.4.11" />

   </blahboxType>

   <fooboxType>

      <project name="fooProj1" version="6.2.123"/>

      <project name="fooProj2" version="6.3.92"/>

   </fooboxType>

</versions>

Once the deploy is finished to a particular environment, logic in the deployment file automatically updates the respective project in the appropriate version file.

Here's a bit of sample code for a client side asp page for a Self Serve app.  It passes some values to a server side asp page (deploy.asp) so that page will know which version of which project to deploy into which environment.  The onchange event of the env select control called an AJAX function that retrieves the currently deployed version of the project from the versions.xml file. 

<form action="deploy.asp" method="get" name="MyProject">

  <input name="project" type="hidden" value="MyProject" />

  <input name="serverType" type="hidden" value="blahboxType" />

  <select name="version">

     <option value="">--Select--</option>

     <option value="" & version & "">" & version & "</option>

  </select>

<select id="MyProject" name="env" onchange="getVersions('MyProject',this.value)">

    <option value="">--Select--</option>

    <option value="Test1">Test 1</option>

    <option value="Test2">Test 2</option>

    <option value="SB">Sandbox</option>

    <option value="STG">Staging</option>

    <option value="PRD">Production</option>

</select>

<div id="MyProjecttarget" style="color: green;">

-</div>

<input type="submit" value="Deploy" />

</form>

Then on my deploy.asp page, the request parameters are set to variables which are then used to dynamically build the path to the deploy file (in this case a visual build file called Deploy-<serverName>-<component>.bld)  This could be a nant, msbuild, or ant file as well, you'd just have to change which executable you're using in the executeString.  The command is then executed in a Windows Shell and if it has a return code of '0' it was successful.  Successful or not, we do some very simple logging to show who tried to deploy what, where, and when.
Here's the code:

 ' need a longer server timeout setting so our deploy can finish

     Server.ScriptTimeout=1200

'instantiate some vars

    env = ""

    version = ""

    project = ""

    severType = ""

   

'set the vars

    set version = Request.queryString("version")

    set env = Request.queryString("env")

    set project = Request.queryString("project")

    set serverType = Request.queryString("serverType")

    'gets the user

    Set WSHNetwork = CreateObject("WScript.Network")

    user = WSHNetwork.UserName

    if version="" then

        Response.Write("Please ensure you chose a version <br/>")

    elseif env="" then

        Response.Write("Please ensure you chose an environment")

    elseif project="" then

        Response.Write("error occured - project required")

    elseif LCase(env)="PRD" then

        Response.Write("We don't want to deploy to Production from this app")

    else

        Response.Write(version & " " & env & " " & project)

'set up a dynamic path here to the deployment file (nant or ant or visual build file)  This string assumes the

'build file is taking a parameter called Env - the environment to deploy to

        execString = "<pathToVisualBuildExecutable>\VisBuildCmd.exe /b " & Chr(34) & version & "\visual build\Deploy-" & serverType & "-" & component & ".bld" & Chr(34) & " Env=" & env

        Response.Write("<br/>" & executeString)

        set oFs = server.createobject("Scripting.FileSystemObject")

            set oTextFile = oFs.OpenTextFile("<pathOfFileToLogTo>\deploylog.txt", 8, True)

   

        Dim WshShell

        Set WshShell = CreateObject("WScript.Shell")

        proc = WshShell.Run(executeString,6,true)

        if proc="0" then

            response.write("<br/><br/><div style='color:green;font-size:16px;'>Deploy of build " & component & " " & version & " to " & env & " as " & user & " was successful")

            dt = now()

            logtext = dt & ":  Deploy of build " & component & " " & version & " to " & env & " was successful" & vbCrLf

            oTextFile.Write logtext

            oTextFile.Close

        else

            response.write("<br/><br/><div style='color:red;font-size:16px;'>Deploy of build " + component + " " + version + " to " + env + " failed using user " + user)

            dt = now()

            logtext = dt & ":  Deploy of build " & component & " " & version & " to " & env & " failed as user " & user & vbCrLf

            oTextFile.Write logtext

            oTextFile.Close

        end if

        response.write("<br/>Deploy times and results are logged at <pathOfFileToLogTo>\deploylog.txt")

        set oTextFile = nothing

        set oFS = nothing

        set WshShell=nothing

    end if

        response.write("<br/><a href='javascript:history.back()'>Back to deploy page<a/>")

Custom header in Cruise Control

We have multiple CruiseControl (CCNET) instances running on our build boxes and we've been wondering how to differentiate which one is which so people can know at a glance which one they are pointing at.
This turns out to be relatively easy. In a typical CCNET installation, there's a \cruiseinstalldir\webdashboard\templates directory. Inside it you'll find a file called TopMenu.vm. Open the file up and after the last '#end' put some literal spaces in and type the title of your CCNET site with some plain html formatting to make it obvious (make the font bigger and bold, for example)

...save it and refresh your dashboard page.

Looking for a Pragmatic Development Process

Consider for a moment, if you will, the delicate balance between robust, thoroughly quality assured code and the amount of time it takes to put that code into production. I ran hard into the brick wall of 'development process' this week - an obstacle numerous people in our business had been complaining about, but I'm not sure how many people on the IT side of our organization understood.

My story went like this. We have some feedback from one of our applications that gets displayed in another in office application that was getting truncated. This truncated information was deemed valuable by the SME's and so a fix was requested. I looked into the fix, and it turned out to be a one character change in a cold fusion file. Getting this change implemented in production would not even require a server restart. After getting the changed entered in our bug ticket system and socializing the change with the appropriate stakeholders, it seems that our development process requires that I wait a full month before my change is actually implemented in production... and we're (supposedly) an Agile shop. What (on earth) is wrong with this picture?

This isn't the first time the organization has run into this issue. There have been other similar issues in the past - an email address needed to be changed. Could you please wait 2 weeks? If fact, this has turned into a major issue with upper level management (read execs), to the point that an entire system rewrite is being considered using a totally different technology with the main purpose being to cut in half (or more) the time it takes to get from requirements to production.

Personally, I do not think that the business and management is going to find their silver bullet in implementing a different technology for development. I think the problem lies in our development process being too rigid. 'We have one release a month - your changes go in that release and if you miss getting your change in by the code complete date, too bad'. Different technology will expose other problems. For example, a lack of appropriate hooks (or a complete absence thereof) for proper unit testing. Or, an inability to version and automate the configuration and deployment of the new technology stack.

Save a few million bucks on 'new technology' (it would really only be new to the company). Seems to me that it wouldn't be that difficult to set up a small, very part time team of a senior developer, a senior qa analyst, a senior ba, and an rm dude to assess these 'changes' that don't fit into agile's bug definition, but should be made in production quickly. If the change is deemed relatively innocuous by the team, put on your cowboy hat's and let's have five minutes of fun. Maybe that's going a bit too far - you want to capture the change in your build system. Communicate the risk to your client (very likely the same person or close to them) whose asking for the change, and then make them happy.

Please, don't make them wait a month for their change.