Do we need Release Management systems? Or rather, do we always need them? Do we always benefit by separating Release Management concerns from build and test?
Why not deploy changes directly from a build system such as TeamCity?
After all, build systems have many of the same capabilities, such as connection to an artifact repository, the ability to connect agents to multiple target servers to push updates for databases, web services etc. They can look after versioning and environment variables perfectly well.
A database deployment pipeline using a dedicated Release Management system
Perhaps it’s worth stepping back for a moment to consider what we want to achieve.
There are obvious reasons most organizations carefully control how software is applied to the Production environment (perhaps all higher environments).
Are the business users ready? Do the customers need to be told? Are monitoring, backup, and disaster recovery configured correctly? Most of all, does it work as intended? (Or at all?)
Some of these things can be automated but there’s still co-ordination required, so normally someone needs to give the ‘ok’, hit the button, send the email.
In some industries, segregation of duties is also important; just as the cashier can’t validate their own cash float at the end of the day, the developer can’t directly change Production code (‘tweaking’ and other out-of-process changes being a major source of operational risk, and potentially fraud).
But even if promotion to Production is different, is it different enough to require another system?
Well, I’d say it depends.
Release Management as a term implies greater control than merely putting your software somewhere it can be used. Systems that specialize in providing such control often do excel in the visibility of deployed versions per environment. Separating the concerns can also make it easier to revert changes, aid time-to-resolution when faults occur, and assist in providing a clear audit trail.
I remember maintaining a spreadsheet of software versions per environment. With a dozen or more services, databases, and applications across five environments, it quickly became a fool’s errand. So seeing Octopus Deploy’s main interface, with its clear visualization of each environment, was an “Oh, wow” moment.
But you also can do all these things with mature and highly-customizable systems like TeamCity, or a combined build/test/release tool like Atlassian’s Bamboo. Moreover, software teams have – by definition – been solving these problems since software delivery began, using hand-built or generic systems for much of the time.
In this post, I’m going to look at using TeamCity for release management, rather than a specialist system like Octopus Deploy.
I’m going to use as my test case a database deployment using the Redgate Database Lifecycle Management (DLM) tools.
What am I trying to achieve?
I first thought about what concerns I wanted my release management process to address. They broke down as:
- Ensure everything deployed has been built and tested (separately, ahead of time).
- Force a manual intervention to make sure higher environments have appropriate protection. We want faults to surface early in the pipeline, and for team members to control (and take responsibility for) software promotion.
- Provide easy-to-understand information at the point of approval, so it’s not just box-ticking.
I’m imagining a situation where control is important, but recognizing simplicity is also valuable.
What features does TeamCity have to help me?
There were a few ways to address my concerns, but the following were sufficient for my scenario:
- Tagging of builds
- Artifact dependencies
- Environment variables
- Build report tabs
Step One – Separating Build from Release
Some unified systems have Release components; separate subsystems for addressing the Release concerns (Bamboo, coming soon in Visual Studio Team Services). With TeamCity, a simple project split is the way I’ve seen it done. Then it’s clear the CI project can be run at a different cadence (on each commit perhaps) to any of the deployment projects.
My deployments in this case are pretty simple; I’m using the PowerShell capabilities from Redgate’s DLM Automation to deploy the database contained in the NuGet package created by the CI steps to a given environment. I’ve created the environment variables env.DatabaseServer and env.DatabaseName to deploy to different targets, but otherwise the deployments are the same.
I won’t go into detail, but the commands look like this;
# Ensure we break on errors
$ErrorActionPreference = 'Stop'
# clear the output folder
Remove-Item -Path %teamcity.build.workingDir%ReleaseArtefacts\* -recurse
# create connection to target
$TargetDB = New-DlmDatabaseConnection -ServerInstance "%env.DatabaseServer%" -Database "%env.DatabaseName%"
# prep release artefact
$Release = New-DlmDatabaseRelease -Source "%teamcity.build.workingDir%ForexDB.0.%dep.ForexManager_DatabaseBuild.build.number%.nupkg" -Target $TargetDB -TemporaryDatabaseServer $TargetDB
# export to working dir
Export-DlmDatabaseRelease $Release -Path %teamcity.build.workingDir%ReleaseArtefacts
Use-DlmDatabaseRelease -InputObject $Release -DeployTo $TargetDB
# report finished
write-host -ForegroundColor Magenta "Database deployment completed."
Before anything can be deployed, I want to ensure the changes have been built and tested, and I want a releasable artifact, in this case a NuGet package, which contains what I need to deploy.
Only when the package has been created do I want to make it available to my Deployment project. So I’ll create an ‘artifact dependency’ in my Deployment project, which requires the package, and that I have tagged the build that created it with ‘Promote’.
Step Two – Tagging for Promotion
In my workflow, I’m happy for the dev team to control deployments to all the environments except Production.
By configuring my Deployment steps to have an ‘artifact dependency’ on my Database Build project, I can ensure deployments will only pick up their artifacts from successful builds which I’m happy to promote. That setting is available in Build Configuration – Dependencies.
Step Three – Tagging for Release
For a Release (which I’m defining as a deployment to Production in this case), I have an additional requirement.
I want to enforce a review of the changes as they’ve been applied to the UAT environment, and I want the reviewer to record their approval. They’ll be reviewing the changes, build logs and any other artifacts I can make available to help them understand the change.
I’m using DLM Automation to deliver a database, within which SQL Release creates a difference report (exported using the Export- DlmDatabaseRelease cmdlet in the script above) which I can surface within TeamCity on a Build Report Tab. I’ve customized the build tabs to make the database changes easy to understand.
The change report then becomes visible as with any other artifact, without requiring the reviewer to switch context. So the goal – of understanding what exactly has changed – is essentially achieved by working across the tabs in the Build Report.
After the tabs have all been reviewed, I want the approver to tag this UAT build for Release.
And for the Production Release step, I’ve added the requirement as a dependency.
And that’s enough. Now I can run my Production Release, knowing:
- My deployment package has been built and tested
- The package was approved for Promotion
- It was deployed to the higher environments, up to UAT, successfully
- The same version, following a UAT sign-off which reviewed the specific changeset, was applied to Production
- An audit trail exists for all the above
Not perfect perhaps, but I’ve got a reasonable proof of concept for my project, and a workflow that has the right amount of governance for my requirements.
If I had segregation of duties concerns, I could add permissions to the process easily enough by enabling permissions-pre-project (off by default) and setting up groups.
If I just wanted to protect against Fat Finger Syndrome, I could add a review screen for the final deployment (Build Configuration Settings-Parameters-Configuration Parameters);
For my requirements, I think TeamCity has given me a perfectly workable solution. The features for surfacing artifacts, tagging, and creating dependencies gave me a level of protection and oversight appropriate to my scenario.
The visibility of what version is in each environment is not as good as with Octopus Deploy, which might become an issue if the number of components grew. I’d also have to do some more work to be able to revert to specific versions easily.
My audit trail in this setup depends on interpreting some fairly detailed logs, so it might also be trickier to communicate quickly.
In an architecture with multiple dependent sub-systems, and especially if I had high levels of scrutiny and governance, I think I’d still consider a separate Release Management system, or a system with a segregated Release Management component. But it’s not necessarily a done deal.
How do you do your release management? What does governance mean in your team? I’d be fascinated to know.