Geoff Bones

18 July 2014

Code kata 7: Producer-consumer problem

First, apologies for nodding off and not posting for the last few weeks. I have the usual set of excuses, which you can guess easily enough without the need to read them here.

So, on to the kata: something a bit different, this time. It’s a repeat of a kata we had a go at last summer, which is a take on the classic producer-consumer concurrency problem. There are so many great library features in .NET4 to help with concurrent problems like this one and make it possible to write a simple and concise solution. This kata is really an exercise in familiarising yourself with how to use them.

The kata

The concentration of NOx gasses from an industrial chimney is logged by three probes, all of which are installed inside the chimney.

The probes are supplied with a software interface, distributed in a proprietary library. The interface can be used to sample data from a single probe and provides the caller with discrete values between 0 and 1, representing the NOx concentration. It is a feature of the library that calls to read the NOx value are likely to suffer some delay before returning.

Data from the probes must be sent to a data logger, which is a mechanical device designed to meet industry standards rather than to be simple to use. The same library provides the software interface to the logger. The calls to the logger only accepts values in batches of five. Calls to plot each batch can take a variable amount of time.

The aim of the kata is to write a program that uses the probe library to read values from all three probes as quickly as they can produce them and to send the values, in time order[1], to the data logger. To achieve this, your approach should read from the probes and write to the data logger in parallel. For the sake of the exercise, the program should terminate after reading 200 values.

There are a few points to note:

• The Probe and DataLogger objects provided by the libraries are not thread safe; however, different objects acquired from the library may be accessed concurrently on different threads.
• The library isn’t 100% bug-free; the data logger code is known to throw exceptions incorrectly. It would be better if you were to handle these.
• There is an external memory constraint which means your program must store no more than 50 values that are waiting to be sent to the logger. If it reaches this threshold, it should continue to read values from the probes, discarding those it has no room for. The most useful values are the ones that have the longest timespan between them and those immediately before them. If you have time, amend your program to discard values that are less useful in preference to more useful ones.

You can find the library with the Probe and DataLogger interfaces at https://github.com/GeoffAtRedGate/Producer-Consumer-Kata.

[1] you will have to order the values by the time you receive them, as the probe doesn’t time-stamp its readings.

How did it go?

I mentioned that we’d run this kata last summer. On that occasion, we hadn’t made any mention of producer-consumer, and some of the pairs who had tackled it back then hadn’t even seen that it required a multi-threaded solution. This time I thought I’d fix that by declaring the problem up front, and this time everyone came with a solution to the producer-consumer problem. Even, I still think the kata is too complicated. It doesn’t need the story about chimneys and probes; these only serve as clutter.

Perhaps this is why one pair had missed the point about using .NET4 threading support and had attempted a solution that buffered the Values in a Queue protected by a lock and used an AutoResetEvent to synchronise the producer and consumer threads. It wasn’t beautiful (although it probably was about as well-structured as was possible in .NET2). The real problem with the approach, though, was the difficulty in reading it and determining whether it was correct or not.

Given this, it was nice to see some solutions using async / await to manage threads and BlockingCollection and ConcurrentQueue to make buffer of Values that automatically gave the required thread safety and synchronisation. The outcome was code that easy to read and verify, and I think the people who had taken this route had benefitted from a better awareness of the threading support in .NET4 and how to take advantage of it.