I work for DACOS Software GmbH and my experience of ANTS Profiler has been a very good one. ANTS Profiler did a good job in finding bottlenecks in the code, but it also helped us to pinpoint some dangerous bugs.

Server responding slowly
We were recently working on developing a C# client-server application based on .NET Framework 1.1. The client application was a Windows Forms program and the server was a console application, which was receiving client requests, interacting with a database, performing some computations, and then sending responses back to the client.

The problem we had was that the server was leaking memory at a moderate rate.

Several possibilities were looked into (listed on expected probability to be true):

1. Our code collects references to objects, thus preventing the garbage collector from claiming the memory back to the system. This could be, for instance, due to an unintelligent implementation of a cache.
2. Something goes wrong in unmanaged code, where the garbage collector cannot help. Our code is completely managed, but calls to other components (such as Spring.NET or log4net) may result in execution of an unmanaged code.
3. Something really bad happens in managed code. This is a very wide category starting from bugs in the Finalizer (infinite loops, creation of two instances of class Class1 in the Finalizer of Class1) and to some unlucky pinning strategy.
4. Bug in the garbage collector. Actually, that was not really considered.

The common problem when you are profiling or debugging your application is that you rarely know where the problem actually is. You may have some suspicious pieces of code (some complex algorithm in your code, serialization, communication with database), but you should never rely on these feelings and directly start making code changes.

Searching for the root of the problem
In order to solve the problem, we needed to find a Profiler, which

1. is user friendly, so that the problem could be investigated from different perspectives
2. is accurate in its measurement and modest in resources consumption  in terms of time and memory overhead.

Of course, I could find the problem with another profiler or debugger, but with ANTS Profiler the problem was solved, in my opinion, in the quickest way.

In order to get to the root of the problem, we profiled the memory usage of the server using ANTS Profiler. Once profiling started, we kicked off a mock client that sent typical real user requests to the server.

The following screenshot shows the live objects in server application.

There's plenty to talk about on this screenshot, but I will focus on what is most important in our case (you can find more information on detailed usage of different views of ANTS Profiler in the ANTS Profiler documentation). As you can see, I've opened the view All classes panel and sorted it on the number of instances. This allows me to see which classes make up the most of current heap objects. I rarely sort by size of object, because general purpose objects such as strings, different collections, decimals, and other primitives would come up in the first few rows. This makes it difficult to determine which classes are actually responsible for memory allocations.

The result that came out wasn't exactly what I had expected. A whole bunch of OracleParameter objects was hanging in the memory. Quick check in the code – right, Dispose() will not be called for OracleParameter. Actually this shouldn't result in memory leak, because the Finalizer() should have recycled them, but it seems like this didn't happen.

This should have been fixed anyway, so I added disposing of OracleParameter objects after operations with OracleCommand.

However, the real problem still wasn't solved, because for some reason, we had a bunch of objects that were not deterministically disposed of. This could be due to a programming error from a software developer or perhaps that it wasn't done because of design issues.

You'll find below a screenshot of the hanging OracleCommands, Spring objects etc. after 100 client connections and disconnections. I initialized the OracleCommands on client connection and didn't explicitly dispose of this on disconnection letting the garbage collector determine that these objects can be recycled. The number of OracleCommands created on start-up is not really large, and because of architectural reasons we wouldn't like to implement deterministic disposals for these objects.

The resolving of the problem
Based on these findings, I could probably conclude that the problem was not in unmanaged code, because even completely managed objects were also not being collected.

I was also able to exclude the first assumption (concerning caching or holding references to objects in some other way).

On the screenshots you can see a Hierarchy panel at the bottom. This panel shows which objects are referenced by the current object (the one selected on the All objects panel) and which objects reference the current object. By navigating in this hierarchy, it is possible to find "the root of all evil" – i.e. the root object, holding the references to other objects preventing them to be garbage collected. By the way, this feature could be implemented in a more user-friendly way (for instance, in the form of a graph; at the moment it is necessary to roam in object dependencies until the root can be found). Hopefully, Red Gate will address this issue in their next release.

At this point of searching, I wanted to determine how our objects were finalized and what the Finalizer thread was actually doing. Although ANTS profiler provides some support for profiling of multithreaded applications, the last part of searching was done in a pure Visual Studio environment. Probably, a debugger (for instance, CLR debugger) would have done a better job at this point, but as ANTS profiler already significantly narrowed down the area of possible causes of memory leak, the searching was not very long.

At first, I tried to trigger garbage collection myself writing

GC.Collect();
GC.WaitForPendingFinalizers();

This call didn't return! It looked like a deadlock.

After some time googling on "finalizer thread blocked", I found the following blog post from Chris Brumme http://blogs.msdn.com/cbrumme/archive/2004/02/02/66219.aspx (see section "Failure to pump"). The problem came down to the threading model that we used for the server application. By default, Visual Studio creates a console application with the [STAThread] set – this generates a single-threaded apartment model. Unfortunately a call to Console.WriteLine() blocked the thread, and so the Finalizer thread could not run in our server application. Since we were not using any STA specific functionality on the server (e.g. GUI clipboard interaction) we could safely change from STA to MTA (multithreaded apartment model). A quick recompilation and the memory leak problems had disappeared!

Here is the final screenshot taken after 100 user connections and disconnection to server with no dead objects hanging in the memory.

I would like to thank Red Gate developers for a great tool which makes searching of bottlenecks and bugs really easy and even entertaining. ANTS profiler is definitely worth its price, and after some minor improvements it will be a must-have tool for every developer.

Andrew Kuklin
Software Developer
DACOS Software GmbH