Measuring the Right Things

In my previous career many years ago, I worked at a hospital along with about ten other pharmacists. The department manager came up with one primary metric for evaluating pharmacists: the average time taken to fill each drug order.

While this system did produce a metric, it was the wrong one. The worst pharmacist in the department got the best score while the best pharmacist got the worst rating. The worst pharmacist would work as fast as possible without thinking too much about any issues. It was rare that he would call a physician for clarification if something about the order didn’t seem right. The best pharmacist in the department took a lot of time to make sure that each drug order made sense, and often had questions for the physicians. I’m sure she had the highest (worst) average. If I were ever a patient in that hospital, she is the one I would want to fill my orders.

Measuring the performance of DBAs can also be tricky. When I was a DBA, there were no formal metrics in place, and my evaluations often seemed arbitrary. Once I received a bad mark because someone complained when I wouldn’t give out the sa password for a production SQL Server. (It’s helpful when your manager understands your job, which was not the case for me at the time!) If there are SLAs in place for uptime, that is something you can measure. Typically, DBAs are expected to keep SQL Server available, secure, and performing well.

I ran across this article reporting the results of a survey asking how to measure the performance of developers. The top answer, speed of developer, doesn’t seem like a good one. You can write a lot of bad code very fast. The number of lines of code written could also be easy to measure, but again, this doesn’t mean the code is good. Either way, developers are expected to create new applications and features quickly.

You may have noticed that DBAs and developers have conflicting goals. DBAs must keep the system stable while developers must introduce change that could affect stability. Fortunately, more companies are embracing DevOps. In a DevOps organisation, everyone is working for a common goal. Features are released quickly while stability is maintained.

Speed and quality are both critical in a DevOps organisation for bringing value to the customer, and some great metrics measure the performance of DevOps. Here are a few of those metrics:

  • The frequency of releases
  • Recovery time after a failure
  • Change failure rate
  • Amount of time spent on unplanned work and rework

Being able to measure the performance of team members is essential. Just make sure you measure the right thing, not just the easy thing.