What use is a Development DBA?

"I can't help thinking that unless you have a good DBA on a development team and use him or her as a consultant on all database matters, we're all losing out. I end up having work thrown at me that I could teach a trainee to do, which is a waste of my talents, and the development team's database skills might be 'good enough' but could be so much better." Doug Burns assesses the role of the Development DBA and its increasing importance to the success of software projects.

I’m sure that there are many Developers who would respond to that question immediately, with something like:

‘None, other than to obstruct me at every opportunity!’

‘Well, he’d be more use if he stopped breaking things!’

As a DBA, and having worked alongside many DBAs with different attributes and personalities, I have some sympathy with that perspective. As the number and size of databases used by organizations grows steadily, so does the size of the teams of DBAs in these organizations. However, it doesn’t necessarily follow that the developers will have a helpful, skilled resource working with them, to whom they can turn for advice on database matters.

Many organizations don’t even have separate Production and Development DBAs and I think this is a mistake – a strong, development-focused DBA can prove extremely valuable to any database-centric development project. In this article I’ll explain why, starting with a description of the basic role of a Development DBA and then taking a look at how I think a Project Team can extend that role to get the best results from a DBA, and vice versa.

What do Development DBAs do?

A Development DBA looks after and is responsible for the development and test databases. It truly is as simple as that, although I think you should consider this a starting point.

Many of the more experienced DBAs tend to be Production DBAs. The reason is simple – your critical business systems are in the hands of your Production DBAs and you have to be able to trust them to be competent and reliable. If something goes wrong in a development environment, time might be lost, but customers shouldn’t be affected directly. However, implicit in that view is that Development and Test databases aren’t very important, and Production-orientated DBAs often exhibit very casual attitudes towards caring for non-Production environments. I’ve been guilty of that at times, and it can cause problems. While even most developers would accept that Production should always be the first priority, if you’re a Developer, you might have to stop work completely if your database is down. Likewise, an entire stream of Testing can grind to a halt for days because of a single error by a DBA, or if there is no DBA around to help fix a problem or, better still, prevent problems happening in the first place. Developers and testers are often working to very tight delivery schedules and this sort of downtime can be critical.

However, even when a dedicated Development DBA is in the team, he often tends to be underused. Development DBA roles have often been filled by someone who is training and in their first few years on the job (and therefore more prone to mistakes). This can sometimes cause trust issues with the developers.

Ironically, though, it’s mainly because of the time pressures, and fear of a DBA “stonewalling” and further impeding their progress, that many developers are reluctant to engage a DBA too deeply in their projects. If they consult a DBA at all, it will generally be for a fairly routine task such as moving data around or creating user accounts. While such tasks are important, it reduces the role of development DBA to little more than a highly paid operator, with sufficient privileges to do the bidding of developers. This is a real pity, in my view, as a good development DBA can offer so much more to the team than this.

Evolving the role of the Development DBA

Although their role is often the more senior, a Production DBA might only be using a fraction of a DBA’s skills. If a system is configured well, then all that should be left to fix are small intermittent problems, with the occasional recovery. Once a system is implemented, things should die down until the next implementation. Apart from when you hit a severe problem, the pace of work is sometimes a little slower, with more planning and care taken over each action. Massive hardware resources and better tools have also made the job easier. In some cases performance problems are solved by just throwing more money at them.

When the Production environment is reasonably stable, I’d suggest that your best DBAs should take the opportunity to start working more closely with the Dev and Test teams instead of leaving this to the more junior DBAs. Why?

RDBMS software has become richer in features and more complicated in recent years. Most DBAs would be the first to admit that they struggle to keep up with these improvements and, because the core RDBMS engine is often the most stable component, some of the biggest changes are in the addition of tools and techniques that improve the application/database interface. In other words, the operational DBA aspects don’t change much, but the way you develop database-centric applications does. For example, the area that is probably improved most frequently is the SQL Optimizer, on which any application you develop is going to depend.

However, as a Developer you have enough on your plate already, right? Improved languages, APIs, XML and IDEs coming out of your ears! If DBAs struggle to keep up with the current state of RDBMS software, how on earth can you be expected to do so, along with all the developments in your own field of expertise?

That’s where a great Development DBA can be a true asset – as a consultant on all matters relating to the best use of the database. Because, after all, databases are what all DBAs care about! With this thought in mind, here are a few suggestions for making the most effective use of your development DBA, and generally improving the DBA-developer harmony within the project team.

Top tips for DBA-Developer co-operation

The crux of what I’m suggesting here is simply that developers and testers work much more closely with a dedicated development DBA, right from the very start of the project. I’ve seen this work very successfully at many sites, but probably seen it fail at many more. So, let me summarize what I see as seven of the key factors for the success of the DBA-Developer relationship. Three are advice for DBAs, three for developers and the last, and perhaps the most important one, is for both.

What the Development DBAs need to do for the Developers

Be helpful!

Suggest new approaches, tools and features. Developers love learning new techniques but, as RDBMS software isn’t their primary area of interest, they can’t keep up with improvements as well as a DBA should.

Set up short master-classes covering new features or even just the important basics. I’ve seen this work well a few times. Developers start to feel that they’re improving and can be more self-sufficient, and it feels good for the DBA when you see those new approaches start to appear in their code. You’re making a positive contribution. Most importantly, the application quality improves, for the benefit of all.

If you do these things then you can start to provide the tools to help developers do their work more efficiently. Rather than just moaning about the poor quality of the SQL that turns up in one of your databases, you can start to show them how to use utilities to get at the execution plan and teach them how to interpret it.

Just like DBAs, Developers are not stupid and don’t wilfully produce poor database code, it’s just that no-one has shown them a better way, yet.

Understand the pressures that Developers are under.

DBAs are very aware of the need to work quickly sometimes. If a Production database is down it’s obvious that recovery time is crucial. However, whilst we have a strong appreciation for those time-scales, we can overlook the fact that Project Teams are often working to very tight deadlines too, dictated by a business need for an application to go live. The time-scales might be hours and days, rather than the minutes and seconds of a Production problem, but they are still a constant pressure. What could appear to be a small matter of a few hours lost because of a broken database could be a disaster to a test team who only have today left to complete their testing.

Remember that it’s not your database!

A central subject of DBA philosophy is the balance between ownership and responsibility. If I’m to be held responsible for the non-operation of a database (and I will be) then I should have the implicit right to stop anyone from doing anything that will compromise its operation. However, having a say in related matters is not the same as having complete ownership. A development database is there for developers to use, not to keep me in a job and let me practice my DBA skills. The database belongs to the developers and I’m just taking care of it for them. If they do things that screw up its availability, then I reserve the right to say ‘I warned you about that’, but they must be allowed sufficient flexibility to do their jobs and not encounter a scowling brick wall every time they approach a DBAs desk. We are not database guardians, who must be obeyed, but should be there to help and advise.

What the Developers need to do for Development DBAs

Don’t treat them like Junior Operators.

If there is one thing that I hate about being a Development DBA it’s being treated as though my only skills are moving data around with export and import utilities, cloning databases at the file level, running SQL scripts and creating user accounts. They might all be essential requirements but, really, if that’s all your DBAs are doing, you’re probably paying them too much and wasting their talents.

Try to replace requests with questions.

This is an extension of the previous point. If you come to my desk and say, “Here, we need you to run this script to create a bunch of new objects for a new messaging system we’re developing”, what would be the likely outcome? Probably I would run your script and move on. However, if you come to my desk and say, “We need to create a new messaging system, what’s your advice on the possible approaches?” then I would ask first whether you’d considered the RDBMS built-in messaging functionality.

Think of the man-months that could be saved! (By the way, this is a true example – but unfortunately I turned up at that site after the bug-ridden in-house messaging system was already in place). Let me put it this way. If you tell me where you’re trying to get to, there’s a good chance I might know a better route that you might not be aware of. You’d probably be amazed by what a modern RDBMS is capable of.

Get them involved early

I am always disappointed when I see code that replicates the built-in functionality of the RDBMS. Even if it works, I could have saved someone time if they’d asked me at the outset whether there was an easier way of doing this. I have seen too many SQL statements that are attempting to build XML output files or perform simple data integrity checks, when I know that the server could take care of that with a couple of lines of code. The earlier you ask questions, the better. We might not have an answer, but you might be surprised how often we do. It also gives us the opportunity to stop you from following a path that will come back to haunt you later, when it’s too late to rectify.

What Developers and DBAs need to do for each other

Show each other a little more respect.

Personally, I’m a keen fan and eager participant in any DBA vs Developer banter (Storage Administrators are another favorite target of mine.). However, I can assure you that it is only banter, because I know that the truth is that we’re all in IT because we like its peculiar challenges and the fulfillment we get from the job. Oh, and the money! I’m a DBA through choice. I used to be a Developer and I know several people who’ve made the switch the other way. In the end, we all want systems that run well and make the business happy and the fastest, smoothest way to get there is for us to work well together.

You know more about VB/C/Java/Websphere (insert your own particular skills there), I know more about databases. It’s simple. If I want to know about C, I’ll come and ask you. If you want to know more about databases (and that includes SQL), you should come and ask me. Of course, you could Google around for information (which appears to be the default approach to all problem-solving these days), but who knows you’ll find anything if you’re not sure what you’re looking for in the first place?

Summary

If more DBAs and Developers could at least try to follow these suggestions, I think everyone would benefit.

The Development DBA? Well, instead of being reduced to a highly-paid operator with the necessary privileges to follow the developer’s strict instructions, we can inhabit an interesting consulting role as a core part of the development team with some design responsibility. We’ll also understand the application more than we do at the moment, so will appear less stupid when discussing it and will understand the logic behind the work that we’re asked to do.

The Developers? It might come as a shock to some, but DBAs tend to know lots of cool and interesting features which can make your life easier for you. We just don’t bother telling developers, because they seldom ask us and they treat us like Junior Operators. Oh, damn, I slipped into that DBA/Developer banter again, didn’t I?

I can’t help thinking that unless you have a good DBA on a development team and use him or her as a consultant on all database matters, we’re all losing out. I end up having work thrown at me that I could teach a trainee to do, which is a waste of my talents, and the development team’s database skills might be ‘good enough’ but could be so much better.