The best decisions are never made at 2 AM…
“Some say, if you have a good process, you can deploy whenever you want. But I am more of the opinion that if it’s 5 o’clock, or a Public Holiday weekend, don’t take the risk — with the best process in the world, something could always go wrong.”
Ryan Hird, Technical Lead at Bennetts
Joining us from his home in Yorkshire, UK, Ryan shared his experience of the stress that out-of-hours database deployments can cause, and the change in approach that has helped his team reduce this pressure.
We used to do out-of-hours releases, but it caused high stress because we were often up against the clock. If the release went well, great. If it didn’t, we had to roll back and get everything running again before business hours resumed.
It wasn’t just the ticking clock that was making these deployments so tense. Working out of hours is expensive, which means there are fewer staff on the ground. “You’ve only got a skeleton crew, they haven’t had their first coffee of the day and are feeling a bit groggy — so when something goes wrong, it’s often a chaotic scramble to put it right.”
These 2 AM traumas led Ryan and his team to switch direction. Rather than doing bigger, riskier changes out of hours, they began releasing smaller changes far more often, and during working hours when everyone was available and alert. “You’ve got the full team, everyone’s in the swing of it, so if something goes wrong, there’s support to fix it fast.”
This isn’t just a tactical change, it’s a shift in the team’s understanding of what responsible, effective engineering looks like, even under pressure. You do your best work, when you’re at your best.
“I am well versed in that 2am phone call. You feel a responsibility and want to get to the bottom of the problem, but the best decisions are never made at this time of night.”
Instead, Ryan advises pragmatism. If the lights go out in the middle of the night, don’t try to fix the wiring. Hand out torches and go back to bed. In other words, document a sensible interim measure the overnight crew can follow, then put in a proper fix the next morning, when more people are in and at their best.
So, what is Ryan’s out-of-hours release valve after a high-pressure day? He lets off steam to the sound of loud, clanging guitars. “It sounds counterintuitive,” he laughs, “but I relax through heavy metal music — it’s an outlet for me.”
Get the next stories straight to your inbox.
Sign up for the Redgate Update and we’ll send them as soon as they’re published. You’ll also get industry news, Redgate announcements, event invites, and more.
Read next
Blog post
Rollbacks, Red Eyes And Unreliable Deployments
This series is about the stress of database deployments on the people behind them, and the small, steady changes that help relieve it. – Felicity Questier, Redgate Software. We spoke to data professionals from a range of industries about the impact of unreliable database deployments — not just on their systems, but on their workload,
You may also like
Blog
Rollbacks, Red Eyes And Unreliable Deployments
Blog