Grant Fritchey

19 November 2020

Challenges to Implementing Database DevOps: What If I Get It Wrong?

Another great question that I think I can try to answer:

I always gets bogged down in the details and worry about doing it wrong and making a mess that is difficult to maintain. How do you deal with this?

I’m doing a series of blog posts on the questions from a recent panel discussion. We had such great audience interaction, we weren’t able to answer all the questions, so I’m doing that now.

Getting It Wrong

How likely is it that your initial forays into automating database deployments are going to be an unmitigated disaster?

Probably near to 100%.

However, that’s no excuse to not do the work. I’ve been doing database deployment automation for well over fifteen years now. I still get it wrong sometimes. I make mistakes both simple and colossal. That’s no reason to avoid the process. In fact, it’s probably a necessary part of learning how to do database automation.

Let’s talk about what goes wrong and why things go wrong.

In general, taken in isolation, most of the tools that make up a database deployment automation pipeline are relatively straight forward and understandable. Working with source control, whether Git or something else, can be picked up quite quickly. Getting your database code into source control through a tool like Redgate Deploy is usually just a couple of mouse clicks away. Creating a pipeline through a flow control tool like Azure DevOps pipeline is logical and clear. The issue doesn’t lie with any one tool. The real challenge is that all these tools have to be combined. Even if we take an integrated solution like AWS Developer Tools, each of the individual tools, CodeCommit, CodeBuild, CodePipeline, is a challenge in itself, but then must be combined with the other tools in the solution, as well as external tools. This all gets quite messy.

Another reason things go wrong is because we, as technologists, tend to focus first on tooling. The tools needed to set up automated deployments are actually not the biggest challenge. The biggest challenge is getting a well defined process. There are two reasons why this is a challenge. First, we have a tendency to try to simply automate what we do now. Quickly we’ll find that “the way we’ve always done it” is just not something that can be automated. In fact, the pain we’re hitting today is the entire reason we need to change. Second, we don’t agree on what the process should be. Getting a buy-in for how, where, when, we do the deployments can be a lot of work.

OK. So, what do we do about it?

Getting It Right

First and foremost, assume you’re going to get it wrong initially. It’s OK. It’s just part of the process of learning, understanding and growing. Next, we need to attack this using very small steps the first time we try it. As you gain understanding of each of the tools and how to use them together, you can grow and expand your processes. Far too many people attempt to create a full-blown, complete, from the developers laptop right up to production, process immediately. It’s understandable because the organization wants you to deliver stuff right now.

Instead, tackle each step as it’s own thing. Learn how to get your database into source control. Then learn how to get it out of source control. Work with your team to have multiple people accessing the same code base. All of this will present you with learning opportunities, meaning, you’ll get stuff wrong.

Using the incremental approach, now see if you can set up a Continuous Integration (CI) process using whatever toolset you have. Keep it simple initially. No additional automated tests beyond the CI process itself. Getting a change in the database to successfully deploy is a test, all by itself.

If we simply assume we’re going to get some of the stuff wrong, and then, take an incremental approach to understand, and then build, each step individually, we’re going to be much more successful overall.

Conclusion

It’s best to never lose sight of the fact that DevOps and the automation of database deployments is all about process in support of people. Focus there first. The tooling will much more naturally fall into place. Also, don’t be afraid to fail, especially when you’re first starting out. After all, adopting DevOps approaches is about setting up the ability to fail fast and fail often, but in a safe environment, in order to enhance the protections we’ll be providing for our production systems.