What Counts for a DBA – Patience

One day, you are sitting at your desk, minding your own business. Your DBA colleague walks up and exasperatedly tells you that there is a query running 5 hours every day, and that the code is simply awful. After just a few hours of refactoring and tuning, the code could be repaired, blazingly fast, and ready to go. What do you do? What…do you do?

The decision sounds easy, and if you have any passion for technology, your immediate impulse is almost certainly going to be: “Fix it, and immediately!” Is this the right answer? Perhaps so, but you’re thinking like a technologist rather than a (good) manager. A manager will first have to consider other questions such as

  1. Are any other processes being affected?
  2. Are users waiting on this query to finish, or is this an automated process that delivers results asynchronously?
  3. How long will it take to test?
  4. How many of the manager’s pointy haired colleagues are likely to get involved?

If it was a developer who randomly noticed the horrible code, it is likely that the answers to this question will be:

  1. No
  2. Automated
  3. Far longer than to code
  4. Nearly all of them, plus a few of their friends and relations;

In addition to these obvious questions, a manager will also wonder why the pre-deployment code review didn’t pick this problem up at the right time, when any necessary changes were practically free, and question why if nothing else, the testers didn’t notice that the process was running for a long time, investigate, and wave a black flag at the appropriate time.

Knowing that your team clearly wouldn’t write malodorous code unknowingly, the hope is that this code exists as it does because it was technical debt that couldn’t be repaid in the time allotted to development. It just had to wait. Technical debt is basically stuff you know you could have done better but just didn’t have time to, but pledge to fix later. Technical debt should not affect the user experience, just the difficulty of maintaining and extending the application in the future. The way projects go, you don’t get to pay off your technical debt at the time you wish, so it has to wait, and everyone has to be patient.

Almost from the time of finishing with a task, good developers will be aware that they could have done almost every task in a better way. Tasks such as gathering the client’s requirements, designing the solution, and even coding the solution more elegantly. All of these need to be finished before the end of the time-box allows (especially if you were the one who agreed to finish by a certain time!) As a data architect, I agonize over new structures, naming, how to apply constraints, etc. trying to get the best solution. Sometimes after I have released my design to the developers to start accessing the table structures, I realize a much better way to solve the problem that might only take a few hours of my time to complete. My impatient self really wants to halt the project and make everyone change to my better solution. Of course, unless it can save everyone appreciable time, or manifests itself as a major road blocking issue that will affect the end user, it is usually not worth the cost to the project. So I try to document all those things that I wish I had done and patiently hope to get another crack at it sometime.

To not be disruptive to the development process, technical debt can only be repaid when the time is right. You may just have to grit your teeth whenever you notice that long-running query and be patient. A time will come to reward your patience because you’ll be able to sequence the work you need to do it in the most efficient way. Take our original long running query that worked well enough at the time, but needed tuning. When it comes up that we need to add a single column to the output of the 5 hour long running procedure, it needs to be tested again anyhow. Now the rewrite will essentially only extend the project time taken by the those few hours your colleague estimated that it would take to code the change, saving many hours of testing and planning time, and then many hours of processing and deployment time. Everyone wins from the accounting team to the end user to the programmer in between.