Grant Fritchey

10 December 2020

Challenges to Database DevOps: Which Do you Deploy First, the Database or the Application?

Challenges to Implementing Database DevOps: Should You Deploy Database Changes Before or After Application Changes?

Another great question came out during the Challenges to Implementing Database Devops that I’d like to try to address here. This question is pretty simple, although the answer might be a little complicated:

Should you deploy databases changes before or after application changes?

I don’t think there’s a single correct answer to this question. Instead, I think the answer is completely dependent on the application in question, the database, and the processes you have built to automate your deployments. I believe more often than not, you’ll find that you deploy your databases first. However, there are likely to be all sorts of exceptions to this. Let’s first discuss why I think you’ll mostly deploy the database first.

Database First

This question actually comes up very frequently. So often in fact that I joke about it:

Why, the database first of course. Because it’s more important.

OK, not much of a joke. There are real reasons why you’re likely to deploy the database first though, so let’s talk about them.

First, and probably most important, database rollbacks can be problematic. To understand this better, let’s lay out a few of the ways you can roll back database deployments. The first method is to run a restore from a backup. This is always a method of getting a database back to a moment in time. However, it’s also a slow process. Restores and recovery take time. Further, did changes to the data occur which will need to be retained? Then the restore might not work for you.

You could of course take a snapshot of your database prior to deployment. Either a snapshot through your RDBMS functionality, or a snapshot through your storage technology. Snapshots provide a fast way to get back to moments before the changes you implemented. However, the same issue around data retention will come up. Did you discover that you need to rollback your changes immediately after the deployment, or a few hours, days, weeks down the line? If the latter, you won’t be able to use the snapshot to rollback.

You can also build rollback scripts. However, if you’re going to rely on rollback scripts, a couple of things have to be taken into account. First, you should be testing all your rollback scripts the same way you test your deployment scripts. You don’t want to be in an emergency situation where you’re running that rollback script for the first time ever. You could make that emergency even worse. As with the other two mechanisms for rollback, you do have to sweat new and changed data as a part of your rollback process.

As you can see, database rollbacks can be a challenge. Because of this, it’s best to ensure that the database deployment is completed and successful before you deploy your application. That way, if something goes wrong, instead of needing to rollback two different deployments, you only have the database to worry about.

In additions, most of the time the application is dependent on the database being in place, so any changes to the data structure and code must be done before the application deployment can be completed. All these reasons are why I think it’s likely that most people will be deploying the database first.

However, there are likely to be exceptions, so let’s talk about them.

Application First

I’ve seen situations where the application had to lead the way ahead of database changes. Without getting into the details of the implementation, it was necessary that the application be changed before the database because the code for manipulating the data and the database had to be in place before we could make database changes. The application was literally a driver for the database changes. With this in hand, obviously, the database changes were deployed after the application was in place and ready to support them.

I’ve also seen a lot people developing using feature flags such that they want to have the application successfully deployed, with new code, before they enable the feature flag switch. You would ensure that the application code was successfully deployed. With testing complete, you would then move to deploying database structural changes and then follow up with whatever changes were needed to enable the feature flag, changing behavior of the application.

There are probably a number of other choices in how you code your application that will result in the application being deployed first. You’ll still want to think through a rollback process that’s going to protect your data, and ensure that you can successfully rollback the application. Frequently, the best way to do this is to simply provide a roll forward mechanism. Meaning, instead of trying to rollback changes, you implement a new set of changes to fix the problematic behavior.

Conclusion

As I said at the start, there’s not a single true answer for which to deploy first, database or code. As you can see, the choice is driven much more by the needs of your system and your ability to rollback the deployment. The dependencies of the behavior of your system should inform you how best to get your database deployments accomplished.