A bright future with Deployment Manager

Ben Adderson is a test engineer and project manager in DevOps at Red Gate. Ben is a member of the Product Services team which works on the Product Release Application, licensing, "check for updates", and the "early warning system" (a tool that aggregates different data streams to show how many users are on a particular build, and how that build is performing in the wild). The majority of what his team does is developing and deploying .NET applications.

The "bad old days" of manual deployments

Back in the days before automated deployment, we could spend months on one release. Each deployment was the result of weeks of time and effort, and had the potential to be hugely risky.

Once we would finish our testing cycle, we'd decide which build we wanted to ship and put it in a zip file containing the compiled application. If the project we were working on contained changes to the database, we'd also use SQL Compare to prepare change scripts. If the changes to the database were considered especially dangerous, we'd get SQL Compare to prepare a rollback script as well, just in case we needed to bail out.

The zip file, change (and rollback) scripts, and a set of thoroughly detailed manual steps were all thrown over the wall to Operations. We'd arrange a time when one of them (for access to the servers), and one of us (who would be able to identify what was missing if things didn't go as planned) to get together to do the deployment some morning.

We're talking multiple hours per deployment

We lost a lot of time taking database backups right before deploying in the event that we were making changes to the database that we couldn't easily recover from if something went wrong. We need exclusive access to the database in order to back it up, which meant taking down all applications that might want to use it. Within the architecture we have, this includes all the applications we maintain.

Taking the backup would take about 15 minutes. The physical deployment took 30 – 45 minutes, so all of a sudden our deployments are taking an hour; and that's assuming that everything goes correctly. Then there's the validation after the fact to make sure everything's okay, and monitoring the data to make sure it's still being collected correctly for our BI reports. And of course, you'd expect to spend at least as much time preparing for the deployment in advance, writing the scripts of manual steps, and checking everything.

I would write the deployment script and do it in test to make sure that I hadn't missed something out – so that would be another hour. All in all we're talking about multiple hours per deployment when we're looking to ship.

A workflow without manual labor

We introduced automated deployment about eight months ago. I started experimenting immediately, and after some successful experiments we quickly rolled it out to our other projects. We now use Deployment Manager for all our team's deployments.

One of the things that we successfully gained from automated deployments was repeatability. Instead of a series of manual steps, we've managed to instigate continuous integration and automatic deployment to Test. Here's how it works:

Our team uses Subversion for our version control system. When development work is committed on trunk, continuous integration is triggered. TeamCity notices there's a commit, builds the application, and runs the unit tests. Assuming these stages are successful, it packages the application into a NuGet package and publishes that to our internal NuGet server. Lastly, we remotely call the Deployment Manager API and tell it to deploy the new build to a named test machine.

We iterate and iterate until there's a whole series of commits and we're ready for a build to go out. We open up Deployment Manager and find that particular build in Test and deploy it to Production. It's so much faster that we've gone from one or two deployments a week to Production, to ten deployments a week to Production. Physical deployments used to take 30 to 45 minutes, now it's 30 to 45 seconds. There's no comparison.

Once you've automated deployment, nobody ever wants to go back

The biggest areas of improvement have been around time and repeatability. We love being able to deploy the same way everywhere.

Testing the deployment plan is just as necessary as testing the application works, so making sure we're deploying the same way everywhere was a real driver. We also love that we can now automate Deployment Manager. For instance, we've automated Deployment Manager so that it deploys every single new build from TeamCity into our test environment. Previously, I had to log on to the test environment, browse to the network share where the build zips would land, copy it across, unzip it, and manually edit the files. Now, I just see that there's been a build, I browse the test machine and see that it's already there.

Now that the automation has saved us so much time, we get the follow-on benefit of being able to deploy much more often. This means we can deploy small builds which are less likely to cause a major problem with each deployment. Less risk means we don't need to take a full backup before deploying (the nightly backups are enough), so we're saving even more time.

I can't remember the last time I did a manual deployment. It must have been February. Since introducing automated deployment we've extended the approach to pretty much every project that's seen active development since that time. Once you've automated deployment, nobody ever wants to go back to doing it by hand.

Get the free Starter edition of Deployment Manager:

Pricing

Free Starter edition
(for up to 5 agents and 5 projects)

Pricing then starts at $995 for the
Professional edition

Got a question?