Grant Fritchey

18 November 2020

Challenges to Implementing Database DevOps: Dealing With Hotfixes

I recently took part in a panel discussion at the PASS Virtual Summit 2020 entitled: Challenges to Implementing Database DevOps. The interaction with the attendees was stupendous. We received all sorts of questions. Most of them were answered during the event, but a few were not. I’m going to create a series of blog posts answering the questions we couldn’t get to, and probably a few of the ones we did.

The first question I’m going to try to answer is:
Once you have a good database DevOps process in place, how do you recommend handling “hotfixes” to production when development activities are also in process in lower environments?

Source Control

Before we get going, understand that there is more than one way to make all this happen. I’m going to walk through one possible path. Depending on how you’re doing things, there are other ways to solve the same problem.

In order to address this question, I’m going to start with source control. Presumably, if you’re automating all your deployments, you’re likely using some version of source control. Most source control systems support the concept of branching. Basically, this is a way to create a copy of all the code. Then, a second process called merging is used to move the changes from that copy of the code back into the main branch.

So, when you recognize that moment, ooh, we need a hotfix, you look over at your code and spot that it’s all in flux because development is going on all the time. However, hopefully, as part of your initial setup, you also recognized that you need to maintain the production code as a unique resource, separate from the stuff that’s actively in development. If not, well, that’s our first step. However, likely you spotted this, so you already are working with at least two branches, stuff you’re changing and what’s in production, usually called the main branch. You’re going too take the main branch and create a new branch, just for the hotfix.

Make the necessary changes. Put them through your testing and deployment processes (more on that in a moment). Once those changes are thoroughly tested, deploy them. Now, you’re sitting with three branches (at least), the main branch, which no longer looks like production, the hotfix branch and the development branch.

First, we’re going to do the easy step and merge our changes from the hotfix branch back into the main branch. Now, some people might actually work off the concept that you only deploy FROM the main branch. In this case, we would first merge the hotfix branch after it was done testing. Either way, we need to get the hotfix branch merged back into the main branch. This should be an easy step because the only changes are the ones we introduced in the hotfix. An added wrinkle could come from the fact that more than one hotfix is heading out the door at the same time. I would suggest keeping them in the same branch as much as you can. However, you may have multiple branches to merge, and this can be a non-trivial task.

Once that’s complete, the next step is to also merge the hotfix into the development branch (or branches), in order to ensure that you are developing against the latest copy of production. Depending on the changes, what’s in development and the changes there, this may be either a very simple task or, again, a non-trivial one.

As long as we’re focusing on only creating limited life branches and small sets of changes getting done to production, the merge processing should generally be fairly simple and straight forward. However, sometimes, especially if we’re aggregating large sets of changes, this task becomes harder.

This is what is known within your source control system as a workflow, or usually, just flow. Git, the most commonly used source control system, has a number of different flows defined. Git Flow is one. Github Flow is another. That one is more closely aligned with what I’m showing here. With some research, you can figure out what methodology of branching and flow will work best for you and your environment.

Speaking of environments, we need to talk about how do deal with branches and databases.

Environments and Databases

To put it as simply as possible, databases don’t branch. It’s just not possible to create an in-place copy of a database. Instead, you have to create a literal copy of a database. This means, as you begin to implement branching within your system, you’ll also have to think through how you’re going to deal with branching your databases. This is further complicated as you have to deal with all the different environments you’ll have in support of your DevOps-style process.

Let’s just talk about Continuous Integration (CI) for a moment. You’ve probably set up your CI process so that you have a particular trigger that fires the CI process. That trigger may be code committed to the repository, a pull request, or a schedule. Whatever the cause, the trigger fires and code is retrieved from your repository and you run that code through your CI process. When you run that process, it’s retrieving the code from a specified location within your repository. Likely, not the new branch you just created. Yet, you’re going to need to test this new code, through your CI process and all the rest of your environments, in the same way as you test new functionality under development. Nothing changes because we’re looking at a hotfix.

This means, for CI, as well as your other environments, you’re going to need to build a mechanism that lets you deal with branches. In general, that means creating multiple copies of your databases. Maybe it means creating whole new servers or new instances on your cloud platform, or whatever method you’re using to cross environments and do testing. You will have to build out some type of Infrastructure as a Service, or Infrastructure as Code, for each environment in order to support the branching methodology.

Conclusion

If all this seems complicated, well, it is. However, as you implement these methods to solve your hotfix problems, the more you’re going to be adding to the protections you have already built for your production systems. Implementing branching will mean you can support multiple development streams, as well as hotfixes, much more easily. Automating the creation of environments and databases in support of hotfixes will make it easier to do the same in support of new projects, different kinds of testing, and all the rest. The goal is simple, automate in support of your people and your processes in order to add protection to your production environments.