Facebook Job Interview - Production Engineer

Facebook contacted me on LinkedIn recently looking to fill a 'Production Engineer' role.  I wasn't looking for job offers.  They reached out to me.   Apparently they had been doing this a bit though, targeting DevOps professionals as my buddy at the startup I was recently working with also got contacted.

Having Facebook reach out to you for a job opportunity?  Definitely intriguing.  One of the things on my 'IT Career Bucket List' would be to work at one of 'those' companies.  Google, Facebook, etc.  I thought if nothing else, it would be interesting to see where the hiring process went, since in the past they didn't actively recruit unknowns like me. 

I replied that I'd be interested to know more and so a phone interview was arranged with the Facebook technical recruiter.  At the appointed time (a couple days later) he called and I got the skinny on how things would potentially work.

The phone interview was the first step.  After that I provide them with my resume which some of their technical leads would look at.  If they felt I was a fit based on my resume, there would be two remote technical screens - 1 specifically focussed on the Linux OS, and the other on a coding language of my choice.  They wanted someone who was very comfortable at both.  Over the phone they gave me some sample questions - basic linux commands - to give me an idea of what the screens would be like.  If I managed to gain their approval in the technical interviews, Facebook would then fly me to the location of my choice (Menlo Park or Seattle) for face to face talks - both technical and otherwise.  Following that, if I was still up to snuff, I'd get an offer.  Once hired, there would be a 6 week on site 'boot camp' where I'd get trained in all things Facebook and brought up to speed the technical ins and outs of the team I'd be working with.

Getting hired would require me to move to Seattle or Menlo Park.  Real estate at both locations is exorbitant - like ridiculous.  In the event that I was hired, Facebook would offer me a full relocation package, with the potential for a temporary housing situation for 3-4 months while we looked for a permanent residence.  I was told that many employees commute from communities with more reasonably housing prices using Facebook commuter busses that have complimentary drinks and WIFI.  Health and Dental would be covered 100% for myself and my dependents.  Wednesdays are an optional work-from-home day, and Facebook offers 21 days of vacation per year - although I neglected to confirm whether that was 21 business days (over 4 weeks), or 3 weeks all in.  I'm a Canadian, so I asked about a work Visa.  He replied that Facebook has an immigration/legal department that would arrange a TN1 Visa for me, in the event that I was hired.

As far as the job itself - Facebook's version of the Production Engineer role is essentially described here.  At the time I was talking with them, Facebook had 42 teams each responsible for a particular feature in their system (FB messenger, Ads, Newsfeeds, etc.).  These teams consist of 4-5 developers with an embedded production engineer.  Everyone on the team goes on call, one week at a time, so it ends up being a 4-5 week on-call rotation.

In the end, after viewing my resume they decided not to pursue the hiring process with me further at this time.  They had been interviewing 'a lot of strong candidates recently that they felt were a stronger match for their immediate needs.'  I can't say I was heartbroken.  It would have been a big move for us that would have put pressure on me personally and financially - not to mention having one dependent in university and one in high school.

I have been pondering what it was about my resume that flagged it to the hiring managers.  Was it the fact that I've moved jobs every 2-3 years, and they had concerns that I wouldn't last long at FB?  Perhaps it was because of my lack of focus in technology - moving contracts 2-3 years means learning lots of new technology and never getting a chance to really focus.  I've asked the FB recruiter - we'll see what he comes back with.  (You might be wondering why I've moved jobs every 2-3 years...  Its prudent for independent business contractors like myself to 'keep moving' from a Canadian taxation perspective.)

IOException - Disk Quota Exceeded error - yet there's lots of disk space

I use a Mac for work.  I've run into an interesting issue with making backups of client web sites and zip files with my Mac...  It seems that whenever I've zipped a file and backed it up to my Mac, I get these wonderful __MACOSX folders - that's 2 underscores at the beginning of the folder name injected into the folder mix.

Restoring a site from one of those backups on a linux machine can have adverse consequences because of those 'injected' __MACOSX folders.  It seems that certain flavours of linux really don't like that folder name and as a result will not write a single file to the machine any more.... at all.  Instead, it throws an a 'Disk Quota Exceeded' error.  You can try touching a file, you can try unzipping some other files - whatever you do, you'll get that error.   The error didn't go away for me until I deleted all __MACOSX folders that got copied up to my machine in the zip file.  Once I did that - it was all good.

The interesting thing about this error was I could have tons of disk space still on the machine, and yet it would still throw that error until I removed those offending __MACOSX folders.

There are various commands out there for removing these folders from zip files created on a Mac... something like this:
zip -d your-archive.zip "__MACOSX*"

SMTP email issue with hosts file

I had an interesting issue last week.  I had a website with a contact page that uses SMTP sendmail to email a list of people whatever was entered on a form.  The curious thing about this issue was it would send emails to everyone in the list except email addresses that matched the domain name of the web site.  For example, it would send an email to perry.mckenzie@blah.com, but it wouldn't send an email to perry.mckenzie@continuouslylearning.blogspot.com, even though the server that was doing the sending was listed in the DNS as 'continuouslylearning.blogspot.com'.

I tried changing the email provider by pointing the Email Host to a server that I knew accepted anonymous email pushes, but it did the opposite - sending the emails successfully for users from blah@continuouslylearning.blogspot.com, but not sending them for blah@anyOtherAddress.com!

After looking through various logs (including /var/log/maillog), it seemed like sendmail was getting confused as the log was trying to associate the email address (someone@continuouslylearning.blogspot.com) with a user on the host and not finding the user.

sendmail failed error: localhost: someone@continuouslylearning.blogspot.com user not found

I tried removing an entry in the /etc/hosts file that pointed the IP address to continuouslylearning.blogspot.com, but that didn't seem to work.  What I didn't realize until later (with the help of my hosting provider) is that this was the correct change to make, I just needed to restart two services after I made the change:
service network restart
service sendmail restart
Then my email worked perfectly.

Testing with IE 6 (on Linux!)

I have a requirement from a client to ensure that their website looks good in IE6.  None of the machines I normally work on have IE6 installed. So I googled IE6 emulators as I had used one in the past that was pretty good, and came up with:
http://tredosoft.com/Multiple_IE 
and
http://www.browsercam.com/
However you have to sign up to get a free time trial and I thought maybe there was something better for me out there.  Some of the forums talked about having an old laptop that they keep around just for testing with IE6.  I do have an old desktop running XP, so I checked it.  It turned out I have been keeping up with all the updates for the most part and one of the service packs (I don't remember if it was 2 or 3) installed IE7.  Bummer. 
Googling a bit more, I discovered that you can install IE6 on Linux.  Pushing against this thread a bit more, I found this site:
http://www.tatanka.com.br/ies4linux/page/Main_Page
and went to this page:
http://www.tatanka.com.br/ies4linux/page/Installation:Fedora
followed the directions and IE6 was installed and running, properly displaying my client's website within 10 minutes.  The Tatanka gives you the option to install IE5.5 and IE5 as well, but when I tried them, the installations were corrupt.  Since IE6 was all I really needed, I was happy with leaving things the way they stood.

Grep features

I was asked to search several teams' code-bases recently based on a list of over 270 different search strings. We are a Windows OS organization and doing a search like that would take some time. By the way, has anyone noticed that the Windows explorer search tool is flaky? Even if I used a tool like this it would still take a while.
I have cygwin installed, and I know grep works well - when I search for something I know is in the file, I find it (unlike Windows sometimes). What I didn't know was that there is a -f option on grep that allows you to provide a file with all you search terms separated by newlines. An astute worker made me aware of this and I tried it out this morning. It worked nicely and saved me some time. Here's how I implemented it:

grep -r -i -f file_of_search_strings.txt * | grep -v .svn | grep -v Assert | grep -v .sql >> resultFile.txt

-r is a recursive search based on where I'm current sitting on the command line
-i is a case insensitive search
-f allows me to pass in my file of search strings
* tells grep to search every folder/file at this level. Used with -r it tells grep to search every single folder and file that is currently at my current cursor position, or a child of the folder I've cd'ed to.
The | grep -v .svn tells grep to ignore/not display files with *.svn in the results
and then >> allows me to pipe my results to a file for inspection.

You may have to run the dos2unix command on the text file with your list of search items to get grep to see the new lines in the file correctly.

More Software Emergency Fixing Experiences

I've recently helped two more clients with 'emergency fix' issues.

The first issue was a problem a national grocery chain had with an export from a SAS program that did a pull from multiple database tables into an excel spreadsheet, and subsequently imported this data into a MS Access database using a SSIS (Sql Server Integration Services) package.  The SAS export seemed to be fine - that is, it executed without errors, but the SSIS package would fail. From my understanding, the client would run this process about every month or so, so there were a fair number of records (1000's).  However, they started having this issue in Dec. and I didnt' get called about the issue until late April, so there was a backup of data.
The client didn't have the SQL BI studio installed, so we had to go through the idx file manually to see if there was anything obviously wrong.  There didn't appear to be.  The database and excel connections tested fine. There didn't appear to be anything riotously wrong with the data formatting or invalid characters in the excel spreadsheet.  So, what do you do?
Well,  for one thing, don't test with the full dataset if you think it might be a data related issue.  Keep making your dataset smaller until you are sure that none of the data works.  In this case, we cut the dataset in half, started from Feb to the end of Apr and the import worked!  This confirmed for us that there were no permissions issues, or software upgrade issues (we were considering that software upgrades might be part of the problem as more than one piece of the software had been upgraded in that time period).  The successful test confirmed for us that we were dealing with a data related issue.  From there it was just a matter of determining exactly which rows of data couldn't be imported, and what it was about the data in those rows that botched things.
Less than 5 hours of work to this point on this fix.

With the second emergency, a telecom company was having issues getting an automated FTP (file transfer protocol) process to work.  They had changed the paths of where the ftp files were residing and they were voip experts, not so much into the bash shell scripting thing.  Their first problem was pathing in the bash file.  That was an easy fix.  Then there was a problem with sending a notification email that apparently they never got to work.  This was a little bit more tricky.  They originally had the email coded similar to this example.

       #!/bin/bash
       telnet smtp.example.org 25 <<_EOF
       HELO relay.example.org       MAIL FROM:<joe@example.org>
       RCPT TO:<jane@example.org>
       DATA       From: Joe <joe@example.org>
       To: Jane <jane@example.org>
       Subject: Hello
       Hello, world!
       .
       QUIT       _EOF

However, after a bit of work I discovered that the server this script was running on was processing all the command too quickly for the smtp server to get the email off.  In the end, I had to do something more like this:

       #!/bin/sh
       ( echo HELO ccielogs.com       
       sleep 2
       echo MAIL FROM:<test@somedomain.com>
       sleep 2
       echo RCPT TO:<test@someotherdomain.com>
       sleep 2
       echo DATA
       sleep 2
       echo Subject:Test-Mail!
       sleep 2
       echo If you can read this, it Works!!!
       sleep 2
       echo .
       sleep 2
       echo QUIT
       ) | telnet XXX.XXX.XXX.XXX  25

Then to set the bash script to run in a windows scheduler (using cygwin) I had to do something like this:

       C:\cygwin\bin\bash.exe --login -c "/myfolder/myscript.sh" 

Less than 5 hours to fix.

linux reboot and chkconfig

Setting up a new linux server pair for clients, I've did some things I haven't done in a while, so I thought I'd document them here so I'd remember in the future.

To set up a service in linux, use chkconfig. Running the chkconfig --list command shows all the potential daemons that are sitting in /etc/init.d/ and whether they are set to on|off|etc for each linux run level. The main run levels to set are 2,3,4,5. You can set the to on for http like this:
chkconfig --level 2345 httpd on

It's also a good idea to go to the script for that service in /etc/init.d (httpd in this case) and modify the chkconfig line at the top of the file to add those changed params.

To reboot the server you can run reboot. It actually calls shutdown -r or shutdown -h depending on the linux implementation.