ASP .NET: when a crash is not a crash

In the mysterious viscera of Microsoft Internet Information Server, nothing is as it seems. When applications are designed to work over the web, they must bristle with all sorts of bits designed to make the stateless stateful and many users become one inside the web server’s inner sanctum.

What I am talking about are the fundamental differences in behavior between a bit of .NET code and ASP .NET code, and the processes that keep ASP .NET applications capable and highly available.

In traditional desktop applications, an unhandled exception would cause an exit with an error condition and probably log it to the application event log. If this were to happen in a web application, it would become unavailable without the end-user having the ability to restart it. For this reason, ASP .NET web applications have a number of features that allow it to recover almost seamlessly from an unhandled exception or a lack of resources on the server. Unfortunately for troubleshooting purposes, this means that your web application can exit gracefully before you even hit the problem you’re trying to debug.

The Application Domain, or AppDomain, can isolate many “applications” which are actually running in the same process. If one of these applications misbehaves, it can be “unloaded” from the process, and in many cases the objects inside can be recovered and copied into a new AppDomain. The ASP .NET Worker Process, which is the process that can host one or more of these AppDomains, has many functions that can determine when to load and unload an application, and to detect whether or not the application is “healthy”. In many cases, the parameters for determining application health are set by the administrator, whether he knows it or not!

An AppDomain recycle can occur for many reasons:

  • The application is using too much memory
  • The application has serviced a set number of requests
  • The application is taking a long time to respond
  • The application has remained idle for a long time

In the case of newer versions of IIS, particularly versions 6 and 7, the Worker Process Isolation model will log the reasons for an application recycle, but not for all reasons. In IIS 5, your worker process simply exits if there are no more AppDomains loaded into it. The fact that the worker process can just exit without leaving an event log entry or throwing an unhandled exception makes debugging incredibly frustrating.

I have not found a foolproof method for stopping an ASP .NET worker process from recycling, so I rely on ASP .NET 2.0 health monitoring to log any “conventional” application restarts due to a lack of memory or other environmental factors and then adjust my machine.config processModel and web.config to try to prevent the recycle while I am trying to debug the application. All that is needed is one line added to the master web.config, which you will find in the .NET Framework configuration folder (%systemroot%microsoft.netframeworkv2.0.50727config). Adding a rule to log every application recycle will actually allow you to see the reason why an application has shut down for a recycle.

First, look for the healthMonitoring node, then rules, and add the following:

<

healthMonitoring>

<rules>
<add name=”Application Lifetime Events Default” eventName=”Application Lifetime Events” provider=”EventLogProvider” profile=”Default” minInstances=”1″ maxLimit=”Infinite” minInterval=”00:01:00″ custom=”” />

Once this has been saved, ASP .NET will leave behind a friendly reminder that it has recycled the application pool:

Event Type:  Information

Event Source:         ASP.NET 2.0.50727.0

Event Category:      Web Event

Event ID:      1305

Date:            2008-09-23

Time:           13:33:19

User:            N/A

Computer:    XXXXXXX

Description:

Event code: 1002

Event message: Application is shutting down. Reason: Configuration changed.

 

In this case, I would recommend disabling any anti-virus software, as it may be modifying some of the ASP .NET application source code. There are many other reasons for an application shutdown that can usually be solved by modifying the Framework’s machine.config file.