Bug metrics are a notoriously erratic way to judge the performance of a development team and project, but despite this almost all software projects use them. There is a lot of data you can get from an electronic bug-tracking system, from bugs per lines of code, bugs per component, to defect trend graphs and bug fix rates. It is tempting to try to find meaning in the data, but how useful is this data, ultimately, in driving up software quality, in the long term?

If you judge software testers on the number of bugs that they find, then more bugs will be found. If you judge developers on the number of bugs for which their code is responsible, then you’ll get much less buggy code, but you’ll probably struggle to ship a product on any reasonable timescale. Over the course of a project, it’s easy for the team and even individual developers to feel oppressed by the bugs, and under intense pressure to produce ‘better’ code. Bugs continue to be logged and reported assiduously, but many of them simply disappear into the backlog, to be fixed “at some later time”. As the pressure of the ship date mounts, developers are simply forced to cut corners, to change their perception of what “done” really means, in order to increase their velocity, and meet the deadline. Software quality and team morale suffers as result, and despite being rigorously tracked and reported, bugs fester from release to release, since there is never time to fix them. Before long the team finds itself mired in the oubliette.

So how can we use bug metrics to drive up software quality software, over the long term, while enabling the team to ship on a predictable and reasonable timescale? In all likelihood, the surprising answer is “we can’t“. In fact, the ultimate goal of an agile development team might be to dispense with the use of an electronic bug tracking system altogether!

Certainly at Red Gate, some teams are using JIRA for incoming customer defects, but also a more holistic “technical debt wall”, consisting of post-it notes describing the most important issues causing “drag” on a team. They then collectively seek to resolve these issues, whilst striving to remain close to zero new internal defects.

The team works to cultivate an atmosphere of zero tolerance to bugs. If you cause a bug, fix it immediately; if you find a bug in the area in which you’re working, tidy it up, or find someone on the team who can. If you can’t fix a bug as part of the current sprint, decide, very quickly, how and even if it will be fixed. This is not easy to achieve; it requires, among other things, an environment where it is “safe” for the team to stop and fix bugs, where developers and testers work very closely together, and both are strongly aligned with the customer, so they understand what they need from the software and which bugs are significant and which not.

However, when you get there, what becomes important is not the number of bugs, and how long they stay open in your bug-tracking system, but a deeper understanding of the types of bugs that are causing the most pain, and their root cause. The team are then judged on criteria such as how quickly they learn from their mistakes, by for example, tightening up automated test suites so that the same type of bug doesn’t crop up time and again, or by improving the acceptance criteria associated with user stories, so that the team can focus on fixing what’s important, as soon as the problem arises.

These are criteria that really will drive up software quality over the long term, and allow teams to produce software on predictable timescales, and with the freedom to feel they can “do the right thing”.

What do you think? Is this a truly achievable goal for most teams, or just pie-in-the-sky thinking?

Cheers,
Tony.