IIS and setting NTAuthenticationProviders and authNTLM

More than once now I've run into situations with the NTLM configuration in IIS where it seems that the GUI setting doesn't do all you need it to to properly configure integrated windows authentication for a web site. Earlier this week I was asked to look into a situation for a particular project team where Firefox wasn't prompting them to enter a username/password in one environment (they'd just get an http 401.2 error), and yet it was giving them the prompt in another environment.  (IE was running fine in both environments, by the way).

I went through all the IIS GUI configs in IIS and the two boxes looked identical.  Since this appeared to be an issue with integrated authentication, I ran this script:

cscript c:\inetpub\Adminscripts\adsutil.vbs GET W3SVC/WEB_SITE_Number/ROOT/NTAuthenticationProviders

which showed me that the working server had nothing set (wasn't set), while the server that was giving them problems returned with 'Negotiate,NTLM.'  I subsequently set NTAuthenticationProviders to '' and it worked by using 'set' instead of 'get' in the above script and providing an empty parameter.

Note:  I understand the above setting to be different than the one below.  In more than one instance at work I've had this issue and proven that this is the case.  The script below, from my understanding, the same as the Windows Integrated Authentication checkbox on the Directory Security tab in IIS.

cscript c:\inetpub\adminscripts\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/authNTLM TRUE

Common Obfuscated Error Messages with IIS

I've noticed that IIS tends to obfuscate error messages - displaying messages that are different than the actual issue that is happening to the application.  This may very likely be because we are running a plethora of web sites with different architectures and history on one instance of IIS.  I thought I document some of those 'tricky' error messages here just in case someone else runs into them (or I forget what they really mean).

1. 'An existing connection was forcibly closed by the remote host'  I ran into this error numerous times trying to configure a WCF web service to bring back a large result set (60k + rows).  Playing with the values for maxBuffer PoolSize and MaxReceivedMessageSize in the web.config, I came to discover that this error was really seemed to be masking was an out of memory error that forced the shutdown of the AppPool that was serving the web service.

2. 'BinaryFormatter Version Incompatibility errror'  Generally I've seen this error when you have IIS site on a web server trying to communicate with another IIS site on an application server, and there is no response from the IIS site on the application server - either that web site isn't running, or your url pointing to the web site on the application server is wrong.

IIS default app pool proc terminated

Here's a great link to error codes for w3wp (event id 1009) and what they likely mean:
http://blogs.iis.net/brian-murphy-booth/archive/2007/03/22/how-to-troubleshoot-an-iis-event-id-1009-error.aspx
My current issue is error 0x0 which is documented in this link but fairly sparse. I checked the debugger flags with gflags and everything was unchecked. I'm still looking for a resolution to my issue.

***

I did find a resolution to this issue. I went through all the configs and our DRP (Disaster Recovery Procedure) document to see if a could figure it out. No dice there. I downloaded and hooked up procmon to see if it would tell me which registry key or file was causing the issue. No dice there either. It turned out that since our app had a dependency on SoftArtisan's FileUpEe, the fileUpEe dll's were registered in the wrong order! I had registered them 'com' folder first, 'dotnet' folder second (per the DRP). They needed to be registered 'dotnet' first and 'com' second. That made my IIS appPool stop cratering (it was definitely the issue).

More IIS automated deploys

Been working more with IIS automated deploys and found a couple of good posts in David Wang's Blog along with some other good blog posts.

This one is related to how to manipulate IIS list data. He provides a handy vbs script to do this that worked nicely for me in adding a new application mapping to IIS.

This one talks about IIS App pool crashes - fatal communication errors between the Application Pool and IIS.

This one talks about app pool recycling and IIS availability. There's some great conversations past David's article further down the page.

This one (not David Wang) shows how to change COM+ MSDTC settings programmatically. This script worked great for me as well.

This one (Ian Morrish) shows how to manipulate COM+ Security launch and activation permissions using DComPerm from the windows SDK.

With all of these scripts, I found I had to carefully read the directions and make sure I had the parameters that were being passed in correct. I was often passing in erroneous parameters on my first couple of tries.

Automated Windows Deploy scripts and tricks

Permissions (domain is optional):
cacls drive:\folder /T /E /G "domain\username:C" - :C modify permissions
cacls drive:\folder /T /E /G "domain\username:R" - :R read permissions
cacls drive:\folder /T /E /G "username:F" - :F full permissions

net localgroup "IIS_WPG" "domain\username" /add - adds user to IIS_WPG localgroup

IIS configs:
Set the friendly name for an IIS website:
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/AppFriendlyName IIS_WEB_SITE_Name
Set the port of a website:
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ServerBindings ":8080:"
Install an IIS Application pool:
cscript drive:\path-to\adsutil.vbs CREATE w3svc/AppPools/AppPoolName MyApplicationPool
Set identity type on AppPool
cscript drive:\path-to\adsutil.vbs SET W3SVC/AppPools/WebSiteName/AppPoolIdentityType 3 (2 is the predefined network service account)
Set appPool username
cscript drive:\path-to\adsutil.vbs SET W3SVC/AppPools/WebSiteName/WAMUserName username
Set appPool password
cscript drive:\path-to\adsutil.vbs SET W3SVC/AppPools/WebSiteName/WAMUserPass password
Delete a web site:
cscript c:\windows\system32\iisweb.vbs /delete "WebSite name"\
Create a web site:
cscript c:\windows\system32\iisweb.vbs /create "IIS File Path" "WebSite name" /dontstart
Set server bindings
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ServerBindings :80:myWebSite :80:myWebSite.com
Set virtual directory permissions
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/AccessFlags 513
Set enable anonymous user for vDir
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/AuthAnonymous TRUE
Set vDir anonymous username
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/AnonymousUserName username
Set vDir anonymous password
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/AnonymousUserPass password
Set vDir to use integrated windows authentication
cscript drive:\path-to\adsutil.vbs SET W3SVC/WEB_SITE_Number/ROOT/vDir_Name/authNTLM TRUE
Set vDir ASP.NET version
C:\WINDOWS\Microsoft.NET\Framework\version_number\aspnet_regiis -s W3SVC/WEB_SITE_Number/root/vDir_Name

IIS binary formatting error

Oh man, did I get a slap in the face yesterday. I was helping/coordinating a patch into production for our team and was asked if we needed to 'drainstop' (read turn off load balancing) to deploy to our app servers. I said no, I had never done that before here. Later I realized that I probably haven't done a prod patch to the app servers here period.
Anyway, we pushed our change onto running servers and everything appeared to deploy fine until we actually tried to use them. It seemed that even though we pushed new components and gac'ed them, that was ignored because the server still had a running request on the old one. Once that request was gone, our app servers were down. When our app servers go down and our web servers try to access them in that state, we get this wonderfully intuitive error called a 'binary formatter error' with a bunch of junk that doesn't make any sense.
So, in the end, two lessons learned:
1. Always drainstop servers when you're deploying onto a live system
2. Binary Formatter errors can mean that you communication with a remote servers is down.

Windows and Automated builds

We've run into a couple of interesting situations with Windows with our automated build scripts that I wanted to document. Both of them have to do with inconsistencies with what the Windows GUI is saying about a configuration versus what the operating system is actually using for a configuration.
Both of these situations were discovered while we were trying to do an expansive upgrade of our code to use the new .NET framework. In the first instance we were manually trying to install/uninstall dll's from the gac on a particular box. We manually dragged and dropped a particular dll into the c:\windows\assembly folder and the GUI would say everything was successful with registering the dll in the gac. However, check it from the command line, or try to run something in the application against the newly gac'ed dll and the OS didn't seem to see it. The only way we could get the OS to successfully see the change was to gac the dll from the command line.
In the other instance (which is sort of similar in nature), our automated deploy scripts are creating a Virtual Directory underneath an IIS web site on a remote server. After the Vdir is created, we had a 'step' that automatically updated the ASP.NET version on the Virtual Directory fo 2.0.50727 (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis -s W3SVC/%WEB_SITE_NUMBER%/Root/%WebSiteName%). This script ran successfully, and the ASP.NET tab in the VDir on the web site displayed the change. However, IIS didn't actually see the change and continued to run the web site on the old 1 .1.4322 version - even though the GUI in IIS said it was. We knew this because the C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files folder didn't have the new corresponding virtual directory folder (this is where IIS keeps the compiled pages - it's like the 'working' folder in Tomcat). It seems the IIS metabase was updated, but the registry was not. I have a hunch that this might be because during that particular deploy, we left IIS running while we did the deploy. Perhaps shutting it down would make the update work properly through to the registry.

IIS debugging

Been working lately on getting a new IIS web application into production. We've been continuing to run into some more issues with anonymous/integrated windows authentication as the build is progressing through different environments. However, I found this cool tool yesterday that has some neat utilities for working/debugging IIS. It's called the Internet Information Services (IIS) Resource Kit Tools and you can download it here. Some of the cool utilities in that package are:

• WFetch - allows you to fully customize an HTTP request and send it to a Web server so that you can see the raw HTTP request and response data. Sort of like Ethereal for IIS.
• SelfSSL - allow you to create and sign you own ssl certificates for testing with ssl in dev environments
• A Metabase Explorer - allows you to view all the configuration properties IIS and it's web sites, app pools, virtual directories, and more
• And an Apache to IIS migration tool which would apparently allow me to migrate an apache web server (and it's configs) to an IIS6.0 server.
  

IIS and Nagios

Working with IIS on a particular app, we had a situation where the behavior of the web server seemed to be inconsistent with how we had configured it. We had configured it to allow Authenticated access based on Integrated Windows authentication only, no anonymous access. When we browsed to the page we wanted to view on the web server itself, the page came up fine. But when we tried to browse to the page from another box, we were required to authenticate. This was not expected since it should have been using the Integrated Windows authentication to log in transparently.
After comparing it with another server that worked and seeing that most of the GUI configuration looked identical, I started googling and found this article that answered the problem for the most part. The only difference we found was even the order of the different NTAuthentication providers matters - the working server had "NTLM,Negotiate"; the server that didn't work was set up as "Negotiate,NTLM". Changing the broken server to look like the other server fixed our problem.

Our infrastructure teams has been doing some changes and moved the configurations for the email servers a bit. Unfortunately, this wasn't transparent to my Nagios/GroundworkOpensource installation (since I was lazy and hadn't reconfigured the email 'from' address for host). What ended up happening was all the Nagios emails ended up getting sent outside the internal network (because the domain name was still 'localhost.localdomain') and getting caught by the public antispamm appliance we have. I ended up switching notifications to use 'service-notify-by-sendemail' instead of 'service-notify-by-email' and then overriding the host in the command to point to our local email server. And that worked. Similar to this post on the groundwork forum.

Automatically deploying IIS web apps

We've been working on automated builds and deploys at work. All of our web apps run on IIS and configuring 'good' automated deploys for these applications has been challenging. But I think we're seeing the light at the end of the tunnel now.

Microsoft provides a good number of support scripts (in vbs) to configure IIS, App Pools, and Virtual Directories from the command line. Some of them come with the IIS installation, some of them we had to download - if I remember correctly.

There are two 'sets' of these CLI vbs scripts that I'm aware of that work with IIS. One is found in the ../inetpub/adminscript folder. There's a bunch of vbs scripts in there. The other set is found in c:/windows/system32/iis*

Here's some examples of what you can do with these scripts:

• create a virtual directory: cscript c:\windows\system32\iisvdir.vbs /create webSiteName virDirName virDirPath /s serverName
• set perms on a virtual directory: c:\inetpub\adminscripts\adsutil.vbs SET w3svc/webSiteNumber/Root/virDirName/AccessFlags permNum (like 513)
• set ASP.NET version for the virtual Directory c:\windows\Microsoft.net\framework\v2.0.50727\aspnet_regiis -s w3svc/webSiteNumber/root/virDirName
• and tons more...
  

You can view all the website numbers in the IIS manager. They show up in a column.