Debugging/Troubleshooting SQL performance/blocking issues

I picked up a good tip from one of our DBA's yesterday.  He uses sp_who2 to see which users have active threads/processes running against a particular database.  This stored proc returns the PIDs as well (Process ID number).  I had known about this previously.  What was new to me was he also used:

dbcc inputbuffer(pidNumberHere)

to get the database to tell him which stored proc was being executed by that PID.  That seems to me like a handy trick.  It certainly solved the problem I was having.

WCF and Large Result sets

Dealing with oil resource data across several disparate systems and also trying to make our API's more generic and consistent, we use WCF a lot (web services with windows apps). We've had a couple of instances where we've had to tweak the config settings for either our publishers or our clients to return some large datasets over WCF.
For the most part, I have found the error messages relating to constrained WCF config settings refreshingly to the point.  They pretty much pointed me directly to the attribute in the file that I needed to change in order to fix my issue.
For memory related errors on the client, I found that juggling with the maxBufferPoolSize and maxReceivedMessageSize in my binding to help.  I moved the maxBufferPoolSize from 9999999 to 99999999, and changed the maxReceivedMessageSize to 350000000.  Sometimes after that I would get a timeout error on the client machine (depending on what kind of load the server was under).  So I'd have to make the timeout settings on the binding higher.
Obviously it helps when the publisher has the same values for the same attributes on its binding.

SMTP email issue with hosts file

I had an interesting issue last week.  I had a website with a contact page that uses SMTP sendmail to email a list of people whatever was entered on a form.  The curious thing about this issue was it would send emails to everyone in the list except email addresses that matched the domain name of the web site.  For example, it would send an email to perry.mckenzie@blah.com, but it wouldn't send an email to perry.mckenzie@continuouslylearning.blogspot.com, even though the server that was doing the sending was listed in the DNS as 'continuouslylearning.blogspot.com'.

I tried changing the email provider by pointing the Email Host to a server that I knew accepted anonymous email pushes, but it did the opposite - sending the emails successfully for users from blah@continuouslylearning.blogspot.com, but not sending them for blah@anyOtherAddress.com!

After looking through various logs (including /var/log/maillog), it seemed like sendmail was getting confused as the log was trying to associate the email address (someone@continuouslylearning.blogspot.com) with a user on the host and not finding the user.

sendmail failed error: localhost: someone@continuouslylearning.blogspot.com user not found

I tried removing an entry in the /etc/hosts file that pointed the IP address to continuouslylearning.blogspot.com, but that didn't seem to work.  What I didn't realize until later (with the help of my hosting provider) is that this was the correct change to make, I just needed to restart two services after I made the change:
service network restart
service sendmail restart
Then my email worked perfectly.

Books I want to read soon...

Here's two books I want to read soon:

The Nomadic Developer: Surviving and Thriving in the World of Technology Consulting







And

Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler))

Thoughts on Design

I recently re-read The Last Lecture by Randy Pausch and was impressed with what he did with his classes that focused on User Interface Design.  The first day of class he would bring in an old VCR, put it on a table in front of class, and without saying a word he'd pull a sledgehammer from behind his desk and destroy it.  Needless to say it got the class's attention.  The message Randy was trying to convey in this demonstration is that poor end user design is detrimental to the product.  End User Design is critical.

There is another person who is a strong proponent of excellence in End User Design.  He is still alive today and he's be a driving force in perfection in Design in the IT industry.  He designs have been flying off of store shelves for the last 25 years or so.  If you haven't guessed yet, I'm thinking of Steve Jobs from Apple. I ran across a very interesting article about him in Bloomsberg Businessweek.  It's basically an interview with John Sculley, the CEO of Apple from the mid-80's to the mid-90's, and what it was like 'Being Steve's Boss'.  He talks a lot of Steve's philosophy and background in design, and Steve's influence on Apple in this area.  It's a great article.

It got me thinking...  In software I have not been super stringent about software design on the applications I've written.  Let me clarify - object oriented design is a must.  The train of thought I'm seeking to follow here is 'is it a requirement to implement design patterns in your code everywhere possible?' A major reason for this is I've worked with a number of people who think their job is refactoring and renaming classes and packages in a code base - to the determent of the project.  I often tell the students in my classes that there are numerous ways to 'skin a cat' as it were, when coming to a working solution in code.  I wonder though, is there always one perfect way?  Is this something to strive for?  Does it matter as long as the application works the way the client wants it to?  What do you think?

Personally, I think that the answers to these questions depends on a lot of things, not the least of which are your client, the project, and your team.  Elegant solutions look and feel so good when your done - both for you and for you client.  Perhaps the motivation and context for the project need to be analyzed before a decision is made whether or not you should 'shoot for the best designed, most elegant solution' possible?  Obviously if your project has constraints like budget, resources, and time, there's a limit to how much effort you can put into evolving your project into the best design in can be.  That is one big reason why design patterns and frameworks are so prevalent (and pretty much a requirement) in software engineering.

I like to read/sign out Fast Company from the Library - I'm actually thinking about getting a subscription.  Anyway, they have an annual 'Masters of Design' issue that is a lot of fun to read.  In it they highlight different world class designs and their designers.  Here's a link to their MOD tag cloud.

Windows OS Performance Guidelines

These are notes from a workshop I was able to be a part of a couple of weeks ago.  The workshop was on Windows Performance Monitor and monitoring vital signs on Windows Servers.  The windows performance monitor has a multitude of counters that a person could potentially monitor on a server.  I'm just going to point out a few critical ones that were highlighted in the workshop and what their tolerances are.

Some points to note before I get into these values:  

1. Don't always look at the graph first. The counter graphs can be scaled right out to lunch so they aren't necessarily a good first glance indicator.  For a particular counter, your focus should first be on the Minimum and Maximum values in perfmon just below the graph.
2. In the past, I would tend to keep an eye out for sympathetic counter relationships on the graphs.  However, keeping an eye out of inverse counter relationships is also a good idea (where one counter could be decreasing in value while another is increasing)
3. Counters can get corrupted.  Apparently this happens more often than one would think.  They can be rebuilt - directions are in this KB post.
4. You can attach a PID (process ID) to a perfmon counter in an OS older than Windows Server 2008 by modifying reg keys. Details are in this kb article. Versions of perfmon on 64bit OS's come with this already set up.

On to the objects/counters of note:

• Process Object
• Handle Count - greater than 500 handles may point to a problem.  
• Private Bytes - greater than 250MB could be a problem.  I've seens procs over a Gig and they definitely were a problem (they can get that high).               
• Working set - greater than 250MB could be a problem
• Thread Count - Greater than 500 threads needs to be watched to ensure they aren't increasing over time
•  Processor Object
• Processor Time - all core instances.  _Total can get you an overall trend.  Greater than 91% utilization is potentially an issue.
• Network Interface Object - you need to know what the spec is for your network interface to determine it's capacity.  Anything over 80% of capacity could point to a problem
• Current Bandwidth - help you determine the NIC's capacity
• Output Queue Length - greater than 2 is an problem
• Bytes Total - greater then 65% of capacity utilized is past the warning threashold (blinking read with siren)
•  Memory Object
• Free System Page Table Entries - the higher the better here.  Lower than 5000 is considered critical.  I've seen boxes 'run' (aka hobble) around 2500.
• Available Megabytes - again higher here is desirable.  Less than 100 MB or 5% free is very problematic
• Pool Non Paged Bytes - greater than 80% consumed is out of spec (not at all good)
• Pool Paged Bytes - Same as non-pages bytes.  Anything between 60-80% should be watched.
• Logical and Physical disk Objects - they have the same critical counters so I've put them together here
• Idle percentage - 19 to 0 percent is critical. Anything over 55 is warning.
• Current or Avg Disk Queue Length - 3 to 31 in the queue - you need to keep an eye on it.  greater than 32 is an issue
• %Avg Disk Sec Read or Write - 25ms and above is critical

 Other interesting points of note:
- Mark Russinovich was the original developer of Perfmon.  His blog is apparently pretty good and our facilitator was very impressed with him.
- Windows Server 2008  does processor/core parking.  This means the server will 'retire' (effectively turn off to save power) CPU's when there isn't a heavy load on the box.  Our facilitator told us that one ISP moved all their boxes to Windows Server 2008 for this reason and their power bills were 15% less a month. 

Newbie Advice

Having gone through an IT technical school, I realize they push a lot of information and technology at you. There's a lot to assimilate in less than a year (depending on the school). On this page I'd like to point out some technologies that perhaps weren't covered much in classes that are widely used by good shops in the industry, and also offer some career advice for the future. Before I delve deeper, let me say that this information is based on my experience, which is mostly on the Java technology side of the fence.

Technical Heads Up

1. Design Patterns. One thing many of the technical schools don't have time to cover in their curriculum is code design patterns. It doesn't matter if you doing .NET development or Java development, design patterns can really make a difference in organizing, maintaining, and the reusability of your code. A great book to pick up for an introduction on design patterns is published by O'Reilly and called Head First Design Patterns. If you don't have access to the book, a good place to start is to google some popular patterns that are being used in code now. Dependancy Injection is a pattern that is really big right now. Two frameworks that use it a lot are Spring and Google's Guice. Other pattern I've seen used frequently are the decorator pattern, the abstract factory pattern, factory pattern, the singleton pattern, and the proxy pattern.

2. Using a Repository. Using a repository, or a version control system, should be required by every development shop. Frankly, if you're in a company where they aren't using one, either push to get one installed, or start looking for another place to code. A repository is basically an application that functions as the premier source/bank/datahub/repository for all code. This application should be on a server with lots of disk space and accessable to all developers in the company. Repositories allow developers to manage code changes at many different levels allowing many developers to work on the same code base at the same time. Some repositories I've used in the past are Subversion, CVS, StarTeam, and Perforce. Googling any of these will give you more information.

3. Unit Testing. Unit testing is the development practice of coding 'tests' that validate individual classes or 'units' of code, to ensure they are functioning as required. This practice has developed over the past 4-5 years or so and has been found to be very effective in giving developers the confidence to refactor their code and ensure that it's functionality stays consistent over time. Some unit testing technologies are JUnit, JWebUnit, HttpUnit, DBUnit, and NUnit. 

4. Continuous Integration. Continuous integration is a process that automatically rebuilds and tests (with unit tests) an application or code base. Continuous integration employs an application like CruiseControl or Anthill Pro which is configured to automatically check out the code base from the repository, compile it, run it's unit tests, and sometime even deploy it to a test environment. It allows developers to check for integration problems in the code, and warns developers of broken code, allowing and reminding them to fix it ASAP.

Soft Skills and Career Advice

1. Change is GOOD. Once you've gotten your feet wet in the industry, being a Java developer in one company can mean something quite different that being a Java developer in another company. There is such a vast variety of Java technology that two Java developers can have quite different experiences in developing applications. Whole development processes and methodologies can be very different in differenct companies. One company could be flying by the seat of their pants, developing with not backups or repository, and manually pushing out code, while another is totally immersed in Agile methodology and outsourcing everything but development.... etc. Keeping this in mind, my advice is to not get comfortable in any one company but move around a bit - that is, after 18 months or two years at one place, get a different job. You'll be put into a position where you'll either learn a lot of new stuff, or you'll be mentoring. Either way, it'll be an experience that will likely teach you more then if you had stayed put. It'll also give you some motivation to brush up on your resume writing, interview skills, and expand you network. 

2. Walk & Talk While software developers, for the most part, end up working in cube farms, it should not mean that they need to stay in their cubby hole all day and do work. This may be just my style, but I find when I'm working on a big integration project or on a project with a lot of departments or teams involved, getting up and talking to people really helps things move along. Whether it's getting socializing a new means of managing your server configuration in the code base, or talking to a DBA about a script that you'll need to release with you new code, talking face to face is far more effective than emails, I've found. I generally use emails if I can't find the person, or I want to make sure there is hard documented proof about a decision that is being made, or if I want to include a bigger audience in my communication. Successful software development is more about relationships that a lot a people realize. Relationships are more apt to be cultivated face to face... plus it's good exercise.