You’re not delivering DevOps to the database

I’ve read through a number of the industry thought leaders to get an understanding of how DevOps is being communicated out there. As with so much else in life, you can start at Wikipedia to get a general understanding:

DevOps … is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes.

This is the opening sentence and neatly sums up my understanding of the approach. However, I know there are haters out there that have zero trust for that source. Let’s go to another from AgileAdmin:

DevOps is the practice of Operations and Development engineers participating together in the entire service lifecycle, from design through the development process to production support.

While I think these are pretty well informed people doing good work, that might be an overly obscure source of information. Let’s try Gartner, who everyone trots out when trying to make a point (when Gartner supports the point they’re trying to make anyway):

DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between Operations and Development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a lifecycle perspective.

Typical Gartner, very wordy. However, when you peel it apart a little, you can tell that it’s largely in sync with the other definitions. In short, DevOps is meant to represent a crossover between Development and Operations (the infrastructure and management part of IT, including DBAs).

I have successfully worked within an environment that implemented a DevOps approach to development, deployment and maintenance. I regularly teach classes and provide consulting on how to approach DevOps from the Ops perspective. Heck, I’ve written books on DevOps.

All of this is just a setup to complain that you’re letting me down. It’s not that I’m not providing you the information and methods needed for you to implement DevOps within your environment. No, it’s you.

Developers

Please, developers, don’t get smug. I’m starting with you because you’re a core part of the problem here. Go back and re-read the definitions of DevOps … I’ll wait … You have to notice one salient point. Nowhere does it say, “Developers rule the world”, or “Developers have ‘SA’ privileges”, or “We get to ignore the Operations side of IT and do anything we want”. No, instead, it talks about cooperation.

I get it, you’ve put up with DBAs and all the headaches they create for decades (they’ll get their turn). However, the fix to the problem is not attempting to eliminate Operations. You still need the specialists within Operations to get your job done. Eliminating them will, if nothing else, put you into the on-call rota (and I know from the number of times I’ve attempted to give developers my phone, only to have it summarily rejected, you don’t want that).

I’ve read the treatises on the full-stack developer, only to find that more than a few of them leave out little things like high availability in the systems they build. They don’t worry about disaster recovery too much while programming because it’s going to slow things down. In fact, all the issues around things going just horribly wrong are written off. I even saw someone arguing that they were a full-stack developer because they could code the front-end and could work with the back-end and the rest was just: “Operational details increasingly handled by service providers.”

Ha! No kidding. Operational details? You mean security, backups, HA, DR, all the stuff that your Operations team does? In short, you’re not actually full-stack but instead are relying on others to provide you an operational space, let’s say, within which to do your development. In short, Operations specialists, like DBAs maybe?

So you do need DevOps. You’re going to build code that has to live within the operational space provided for you. For you to deliver better quality code faster, wouldn’t it help for that operational space to ‘shift left’, to move closer to you and your development so that your code works when it ultimately lands in Production? If all that’s true, then you want to, need to, communicate with the Operations people and the DBAs. Which brings me to the DBAs.

DBAs

Hey guys! Said “No” to someone this morning, or is it a slow day?

You’ve spent many years developing the knowledge necessary to set up a well functioning database, the server it sits on, backups (and backup testing), your Availability Group, and that off-site DR location in New Jersey. And yes, all those trips to the middle of nowhere were pretty awful. Then, out of nowhere, this development team wants to do some crazy thing with Entity Framework (yeah, I’ve seen some of the queries) and all you can see is another trip to New Jersey (I’d suggest looking at Azure). The answer is “No”.

Automating deployments? No.

Shifting Production databases to the left? Left? No.

Source control? Tried that in 2002. No.

Every single time I point out how you guys say “No” to everything, I get the litany of reasons why, well, of course, you’re perfectly reasonable creatures and you’re more than open to what the developers want, but they’re wrong, so, NO.

It’s time to learn a new word. Code is going to change faster and faster. New technologies are not slowing down, they’re speeding up. Further, as service offerings get better, aspects of your job (not your whole job) are going to go away. You need to get on the bandwagon and start to communicate with your brethren in the Development community. I’m saying this because they need you and your knowledge. If you take part, in a DevOps way, in what they do, you can exert positive influences. Where? Let’s talk about that.

Why DevOps

There have been, and, in our increasingly connected and interconnected world, will continue to be, a large number of spectacular failures with technology. I want to point the finger at all of us for these failures. It’s not Development. It’s not Operations. It’s us. We’re in it together. If we don’t communicate with each other, share our knowledge, we’re going to mess up horribly.

For example, I know that you know that you should both take backups and validate those backups. However, have you worked with your other teams in development to ensure that they know? I think the short answer is probably no based on what happened to GitLab.

How about the fact that you should have some pretty stringent controls on when you’re in Production versus when you’re in a Test environment? We use completely different logins right? We use different tool settings and all the different ways we can to ensure we don’t accidently drop stuff in Production, right? Or, maybe this hasn’t been communicated as thoroughly as we thought, as evidenced by the recent AWS outage.

And if you have an outage, you’ve done the work to ensure that you have a disaster recovery process (even if it means a trip to New Jersey)? Well, the problems above exposed more than a few companies that hadn’t quite thought through all those operational details increasingly handled by (in this case non-existent) service providers.

It gets even more fundamental than that. Backups are frequently ignored by startups and small organizations that are overly focused on change. Fundamental behaviors of information storage such as ACID properties are dismissed, incorrectly it turns out.

I’m not saying that all these problems are solved by DevOps. I am saying that more, better, faster communication between the specializations involved in Development and the specializations involved in Operations will lead to fewer of these types of problems. DevOps provides that communication mechanism.

It’s not me, it’s you

I know the whole break-up thing is supposed to go the other way (to my shame, I’ve used that line). However, we’re not breaking up. I refuse to let this go. I know that DevOps as a communication mechanism and process not only works, but it makes everyone’s lives better. Developers deliver higher quality code, faster. DBAs have safer, more stable Production environments. Most of all, the business gets new, better functionality in order to make money. After all, that’s what it’s all about, supporting the business. We have a common goal. We have mechanisms for automation and communication. Let’s start using them and deliver DevOps.

Share this post.

Share on FacebookShare on Google+Share on LinkedInTweet about this on Twitter

Related posts

Also in Blog

How advanced is database DevOps in financial services?

Whether you’re exploring the advantages of DevOps or already fully immersed in the journey, including the database brings additional advantages. But how are you performing compared to the competitio...

Also in DevOps

The real origins of the Agile Manifesto

In February 2001, 17 people met at the Snowbird ski resort in Utah. They were the leading exponents of Extreme Programming, Scrum, and Adaptive Software Development, and they were seeking a set of com...

Also about Database DevOps

Can the database be included in DevOps in financial services?

I’ve had a lot of conversations with Database Administrators (DBAs) who work in financial services and, for many of them, deploying database changes is the most taxing part of their job. They’re o...

Also about devops

What are the leading DevOps drivers in financial services?

DevOps is gaining ground everywhere. By offering a route to releasing software faster, with fewer errors, it can give companies which adopt it an immediate advantage. Unfortunately, it’s not somethi...

  • BrentO

    Small error in there – the “on what happened to GitHub” link is about GitLab, different company. GitHub wasn’t involved in that outage.

    • Matt Hilbert

      Thanks, Brent. That’s now been corrected.

      • BrentO

        Thanks!

        • Grant Fritchey

          My bad. I even knew better.

  • Derek Colley

    I agree that devs and DBAs need to work more closely together. But the problem is, as Sean says below, often the working practices of the dev team and the need for a long-term view of the DBA team are mutually inconsistent. In a nutshell, I think dev teams need to “hurry up and slow down”.

    It’s difficult for me to do any kind of capacity planning if the dev team are banging out release after release and working to sprints rather than to good practice. It’s not uncommon for my clients to have problems like no relational integrity, or heaps everywhere, or versioning within the table (Kimball-style), or EAV (key-value store in the table, urgh), or crappy indexes, and so on, and so on. These problems are symptomatic of working to tight deadlines, knocking out shitty code after shitty code without a longer-term view. To some extent it’s the responsibility of the DBA to say, woah, guys, wtf?

    My favourite tweet, which perhaps sums this up. “I SEE YOU HAVE A PHD. HOW MANY SPRINTS DID THAT TAKE YOU?’

    • Sean McCown

      How many sprints did it take you… I love that.
      And yeah, you hit the nail right on the head. What we should be doing is pushing an industry-wide movement for companies to slow down. Instead, everyone’s speeding up and developing new ways to make that ok. To me, Agile is just a way to formalize shitty practice. We know we write crap, but since it’s on purpose now, it’s ok.

    • Grant Fritchey

      There’s no reason though why we can’t both go fast, and to pick on one thing, have referential integrity. We’re only choosing to do something stupid and then being equally stupid to apply a reason for it “because we go faster” through ignorance. One issue I have with some dev teams is that they get seriously stuck on the concept of premature optimization. They start to think that any kind of design or planning or even thought itself, is wasting time on premature optimization. Again, that’s just ignorance about what design is (which is something you’re still supposed to do, even in Agile processes including using sprints and all the rest) with attempting to start optimization. These are two different tasks with two different processes. It’s all about getting the education out there to ensure the difference is clear.

      I’m such a strong advocate for DevOps because I’ve seen it work and work well. It really does require both sides to surrender a little. It just can’t be “We can’t do this because Devs are stupid.” Again, both sides are failing to communicate here. We have to recognize that there’s a pretty good chance that we’ve failed to communicate the issues appropriately to the dev teams. It’s also a possibility that past failed communications are poisoning current communication (also known as ghosts in the communication). There are just tons of issues around all this, but it’s not all on the devs any more than it’s all on the DBAs. I’d just argue we have to find those ways to communicate so that we can arrive at better processes (and better code, faster, safer, etc.).

      For communication stuff, I recommend reading “Critical Conversations”. Good book.

    • Peter Schott

      I think that sort of thing goes back to communication. We had a devops practice @ one of my places that worked well. The ops team was involved enough to push back against new tech that would cause problems or to work with the devs to find the appropriate tech for the job (or if that was the appropriate tech, to make plans to implement it properly). The DB team was involved pretty early to start modeling the database _before_ the devs would write code. Neither of those caused a huge delay, but did involve a little forward-thinking. This was in an Agile shop so the planning often came when we would do the bigger picture stuff and try to figure out how long different things would take and what sorts of approaches we might use. We didn’t get deep into details, but would have a pretty good idea of what we were doing and how we’d tackle it.

      With the DB design mostly done prior to the devs starting their code and questions about whether or not to use which new tech server-side, the company managed to release new features at a pretty good pace without too many bumps along the way. Devs, DB people, and Ops people all collaborated pretty well and there were a lot fewer issues or surprises when it was time to release.

      To Sean’s point, that still means saying “no” to bad design or ideas, but we had just enough forward planning that this could be done without interrupting current work. The communication was a big factor in each team feeling much more positively about the others. We had a really good thing going and it would have continued were it not for external factors coming into play and “improving” it. But that’s a sad story for another time. 🙂

  • Grant Fritchey

    Hey Sean!

    I absolutely hear what you’re saying. That’s why there is a section up there specifically talking to the Devs. I’ve seen teams actively eliminate ANY DBA involvement in the project. This was done in order to make the project move faster because the DBAs did nothing but slow things down (according to them). Said project was then finally delivered several years late, extremely over-budget, and nearly non-functional (in fact, making it work required bringing in the DBA team to build several different additional databases and data movement processes, none of which would have been needed if we had just been listened to at the start). So yeah, I agree. There are places where “No” is the only appropriate word.

    On the other hand, and I’m sure I’ll pay for this, you may not always be right. The devs might have a good idea sometimes (broken clocks & all that). My argument is that DBAs tend to become very close-minded about far too much stuff. However, I’m clearly not arguing that DBAs must accept everything. I point back to the issue above with the Devs. In any failed communication between two parties, both parties have some share of the blame (discounting, of course, when one of the parties is actively abusive, psychotic, what have you). That means both parties have to take responsibility for fixing the communication. The reason I call out both sides is because it’s going to take both sides to come to an agreement and get things done.