evilrouters.net

This is a little “off-topic” from my usual Cisco-related posts (remember, I still do some server-side stuff too!), but we got hit by this so I thought I’d share (thanks to mardraum for letting me know about the solution).

Earlier this week, on Patch Tuesday, Microsoft released KB973917, “Description of the update that implements Extended Protection for Authentication in Internet Information Services (IIS)“. The related update was responsible for hosing a couple of servers that we have that run IIS 6.0 on Windows Server 2003 SP2. A quick fix was to uninstall that update, but Microsoft the next day posted KB2009746, “Internet Information Services 6.0 may not function correctly after installing KB973917“, which describes perfectly the issue we were seeing:

Consider the following scenario. You have an Internet Information Services (IIS) 6.0 web server running on Windows Server 2003 Service Pack 2. The Microsoft update KB973917 gets installed on the server. After installing KB973917, the IIS 6.0 application pools cannot start up successfully. An inspection of the event logs show that the IIS worker processes are terminating unexpectedly, showing event messages similar to the following:

     Event Type: Warning
     Event Source: W3SVC
     Event Category: None
     Event ID: 1009
     Date:  12/9/2009
     Time:  10:55:01 AM
     User:  N/A
     Computer: WEBSERVER01
     Description:
     A process serving application pool 'DefaultAppPool' terminated unexpectedly. The process id was '1234'. 
     The process exit code was '0xffffffff'.

In some cases, the IIS application pool(s) are eventually disabled by the Rapid Fail Protection feature. Users who try browsing to the web sites hosted on the server may complain that the web sites are unavailable and cannot be accessed.

The article goes on to describe the root cause…

Previous to the installation of the KB973917 update, one or more of the core IIS .dll files were not at the correct file version. Specifically, the earlier installation of Windows Server 2003 Service Pack 2 on the server did not complete successfully, leaving some of the IIS .dlls at the Service Pack 1 level while bringing others up to the Service Pack 2 level. The IIS services had been able to run successfully even with that file mismatch in place. However, the installation of the KB973917 update exposes this pre-existing file mismatch environment to the degree that IIS is now unable to function properly.

…as well as the resolution…

To resolve this problem, reinstall Service Pack 2 for Windows Server 2003 on the web server. This will bring all IIS 6.0 components up to the correct file versions, and will maintain the installation of the KB973917 update. Reinstalling the KB973917 update should not be necessary.

I can confirm that reinstalling SP2 for Windows Server 2003 does, indeed, fix the issue (for us, at least!). We reinstalled KB973917 afterwards and both affected servers are fine.