We were meeting with a client recently who was experiencing some major issues with performance with one of their key applications. Their staff had identified the issue as being a disk performance issue – there was simply a ton of data to read and write to disk and it was causing a huge bottleneck in performance. Understand that there was no database involved – the analysis process consisted a series of small command-line applications that would take a CSV file as an input, massage the data, and output a CSV file that would then be used as the input for another command-line application. Although each individual app was relatively small, the entire system was quite large when you put them altogether.

Most of our discussions revolved around bringing in higher speed single drives or implementing a RAID array to help with IO performance because they were looking for a quicker fix than rebuilding the application – but I started to think about how to increase application performance without having to complete a major rewrite because that’s way more fun to think about than a hardware solution (to me at least). Since their data was coming in as a CSV file, I figured they were reading the numbers as string, converting those numbers to integers, and storing them in a data structure of sorts. So I began wondering what kind of performance gain they could get if the data was read in once and serialized back and forth using .NET binary serialization.

My test consisted of generating a byte array of various item lengths and writing that file as a CSV file as well as using the .NET binary serializer to write the binary representation to a file as well. Then I read that file back into memory 25,000 times and calculated the operations per second for each method. I also recorded the file size (in KB). My first run only included reading the text into a string array – it did not include any casting from a string into an int. The findings are as follows:

As you can see, the text import quickly degrades – I graphed it out and it follows a curve that degrade exponentially. File sizes tend to follow a linear trend with the binary size about 30% of the size of the text file size. For smaller number of items (Under 300) it looks like the text import works faster than the binary import.

Next, I tried to see what the performance would like with parsing each text element into a byte:

When performing type casting, the text-based approach is only faster when there are about 100 items and it tails off even more dramatically. I would venture a guess that if I was deserializing a more complex object instead of a primitive type, that the cost of casting would be even greater. The binary approach, on the hand, seems to hold fairly steady regardless of the amount of items with which it is dealing (through both tests even).

Just thought it was kind of interesting if you were ever wondering about binary serialization performances statistics (albeit only for integer arrays).