DevOps, Continuous Delivery & Database Lifecycle Management
Culture and Organization
- A sad tale of unnecessary conflict
- The lessons
- Concluding thoughts
A sad tale of unnecessary conflict
A recent editorial on Simple Talk described how a senior IT figure, midway through a presentation on Continuous Delivery, broke off, sighed and let loose a diatribe against DBAs, their attitudes and working practices and how they were a blocker to progress.
I abhor the behaviour that lead to the quip that DBA stands for “Don’t Bother Asking”. Such an acronym flies in the face of everything that events such as SQL BITS, sites such as www.sqlservercentral.com or www.simple-talk.com, and user groups seek to promote.
Not so long ago, when working as a DBA, I would have used these examples in defence of DBAs before giving an impassioned and well thought-out argument trying to put the case for the DBA. No doubt a member of the developer community would have risen to the challenge with an equally impassioned and well thought-out counter argument. Neither one of us concedes ground, both of us absolutely correct but both of us failing to resolve an ancient conflict.
Now I am in a more senior position I have a different perspective on this battle. I do not have all the answers but I should like to share some of the lessons I have more recently learnt as a slightly less impassioned observer.
Call out bad behaviour early …
No-one owns a monopoly on being a jerk though it hasn’t stopped a tiny minority from trying to obtain one!
With a dispassionate eye I can think of very few occasions when I witnessed either a developer or DBA deliberately being obstructive and unhelpful.
I have seen people on both sides whose passion for their chosen profession acts as a pair of blinkers to a wider perspective. I’m afraid I have to plead guilty to this.
Whether the behaviour is intended or not, the problem is that it can sow the seeds of discord and those seeds can grow like Knotweed, having a corrosive effect on the DB/Developer relationship.
In any case it is incumbent on all of us to calm things down; to help channel the passionate and be especially merciless with the protagonist of the wrong. Nip problems in the bud; don’t leave it for an annual performance review.
It’s not just a DBA/Developer thing
I found it tremendously exciting to enter the world of employment. My first pay cheque, my first car, and my first responsibilities. I also witnessed one hell of an argument between a salesman and a marketing executive. Given the creativity and verbal dexterity that these two business disciplines require, it was highly amusing right up until the point where it got just that little bit too heated.
I learnt that, in addition to dripping honey, a salesman tongue can also spit venom.
I can summarize the argument in the following sanitized statements: –
- Salesman: You marketing types have no idea what it is like to face an actual customer. Your strategy is just not practical. I have to shift ‘x’ thousand units by the end of the month.
- Marketeer: You salesmen just don’t see the bigger picture. Yes you can flog product ‘x’ like that today to meet some short term goal but that will scupper everything we are trying to do to build our brand and grow the market.
Now substitute a developer for the salesman and a DBA for the marketeer.
- Developer: You DBAs have no idea what it is like to face an actual business user. Your approach is just not practical. I have to deliver a system with ‘x’ thousand lines of code by the end of the month.
- DBA: You developers just don’t see the bigger picture. Yes you can hack the schema like that today to meet some short term goal but that will scupper everything we are trying to do to build the data asset and get long term value out of our data.
No-one else knows (or cares) what the fuss is about.
Before I witnessed the argument I hadn’t really understood that there were two separate business functions. “Sales” and “Marketing”. To me they were simply “SalesAndMarketing”. I dare say that many business people just see “IT” and not “Developers” and “DBAs” or indeed any of the other functional titles that are so important to us in the tribal world of IT.
What I do know is that the beam of benevolence from our commercial benefactors can change to a glare of malignancy when the delivery of their pet project is jeopardized by fractious infighting
The DBA/Developer conflict solves nothing, but can draw unwanted attention that rebounds on the protagonists.
Immaturity in governance causes friction.
No public organization has much leeway in its financial or trading conduct, or in the way that it manages the data related to it. The governance and stewardship of data should be part of normal day-to-day operation of any financially-responsible company. There should be specific roles that are tasked with ensuring that all requirements for security, audit, regulatory-compliance, and data-quality are met. This will include the management of data quality, metadata, and reference data.
If that does not sound like the organization you work for then you are not alone. I’m sure there are organizations who adopt all the practices required of a contemporary business, but I have never seen one.
The problems arise when the business expects the benefits of such robust organizational structure without that structure actually existing. In the absence of those structures the expectation is that these functions are just something DBAs do!
Although DBAs should play a part in data-governance processes, they are not at a level within the organization where they have a sufficiently broad view of all concerns that would be required to lead and enforce data governance with authority.
In the absence of a clear data-governance role within the organization, DBAs often find themselves with a vaguely-defined or assumed responsibility without the existence of authority or a clear mandate. Even worse, when DBAs attempt to fill this role they find their authority undermined by those very people who originally abdicated that responsibility; short-term goals of a more senior manager trump the governance process of the more junior role.
When a role is undermined, it has the corrosive effect of undermining the authority for those areas where that role has a legitimate mandate.
Talk to the right people
A conversation with the right person can resolve conflicts quickly and painlessly. Personally I have found that a lead developer is the best person to talk to. Just as with DBAs, the path to becoming a lead developer is not a short one. These positions are relatively senior within IT and are the product of hard won and often and bruising experience.
They are best able to describe what the problem really is from the developers’ perspective and have the maturity to listen to the other side of the argument.
When a lead developer asks for a compromise they have almost certainly considered a number of alternatives and their need is genuine.
The worst thing that a DBA can do is to argue with a team of developers. It can put the lead developer in an untenable position with two unpalatable alternatives
- Does he back the DBA and lose face and authority with the team he is supposed to lead?
- Is he forced to back his team and support a bad compromise?
If a conflict occurs during a meeting or daily stand-up, then by all means state your case but suggest that a more technical discussion take place in a less confrontational forum.
That isn’t to say that communication should only take place between DBAs and lead developers. The more that DBAs and developers interact with each other the better for all concerned. It is simply that there is occasional need for mediation and more senior roles have a wide enough perspective to fulfil that role.
Communication is everybody’s responsibility
If DBAs or developers find themselves under pressure, it is vitally important to let each other know that the pressure exists. I can think of one particular situation that, with the benefit of hindsight, two disciplines working together would have produced a better balanced and robust solution.
The situation I faced was where an e-commerce site suffered a ‘site-down event’ which had occurred because of a problem in the databases.
All DBAs were summoned to the board room and we were told in terms that would be recognised by anyone familiar with the gory bits of the Old Testament what would happen to us if such a situation ever occurred again.
We were told that no excuses would be tolerated and that we were to prevent such an occurrence by any means necessary. The steps we took were undoubtedly Draconian and caused resentment in the developer community. If we had explained that to the development community we could have had allies instead of creating adversaries.
I know that only the DBAs received the fire and brimstone treatment. Senior management were more interested in extermination rather than root cause analysis.
This leads us to lesson seven.
Conflict is inevitable, it’s how you handle it that counts
Conflict in this context simply means that the needs of the different disciplines pull them in different directions. This is simply a fact of life, so conflicts are inevitable but can be ameliorated by negotiating a mutually acceptable compromise.
In his book “The five dysfunctions of a team” Patrick Lenceoni had ‘fear and avoidance of conflict’ as one of the dysfunctions. His book is short and easy to read; I heartily recommend it.
We have to recognize that the perfect coding solution probably isn’t the best solution for the database and vice versa. When we feel passionately about a subject it is difficult to listen to another’s point of view but in order to resolve conflict it is vital that we do so.
Remember, you are not trying to win an argument; I must emphasize that you are trying to reach the best compromise that both parties can live with.
Functional silos create more problems than they solve
I believe strongly that the agile community have got it right. A multi-disciplined team is more effective than trying to get teams of specialists to work together.
Functional silos encourage separateness and not cohesiveness. My experience is that this separateness leads to a breakdown in communication which results in misunderstandings and conflict in the negative sense.
Each silo adopts defensive processes and functions to protect it from unwanted effects of external interactions. In total the effect is much like scar tissue. Instead of speed and suppleness, scar tissue gives restrictive movement and slow progress.
I believe the answer is to build cross-functional teams aligned to the products that the business sells. This should mean that the coordination between such teams is part of a commercial strategy and more likely to be coordinated at a more senior level within the organization.
Whether I am correct in my beliefs or not, I do know that a poor choice of borders leads to conflict.
I do not have a good answer for the case where the underpinning components of a system have grown in size and/or complexity to the stage where a specialist team is required to deal with that component. Perhaps such a case should be seen as an indicator that a system is due for a redesign and rewrite.
Developers make the best data zealots.
Nothing beats the zeal of the converted. The best thing that can happen to a DBA is to be assigned a developer to work on a data project downstream of point of data entry.
The mark of a convert is when they start to rant about the appalling data quality, how easy it would have been to prevent and how it is jeopardizing their deliverable.
Of course the counter to this is when a DBA realizes that source-control, continuous integration and test driven development techniques can be used to give more reliable and robust deployments.
Imagine a situation where a DBA will only practice test driven development on projects where developers are present, or where developers will only practice good data husbandry when a DBA is standing over them. Very little has been achieved in such a situation other than compliance.
When each discipline adopts and acts as a champion for the practices of the other then a business transformation has taken place. When transformation is achieved then heavy weight governance become unnecessary
From experience I know that, when developers and DBAs work well together, they don’t just add to each other’s effectiveness; they multiply it.
If there is one overarching lesson that encapsulates the nine lessons I have outlined, it is to focus on communication. This is something that IT people in particular seem to find very hard. It is much easier to look inward than it is to reach outward.
I do wonder whether the consequence of this is that some of the features and facilities in the tools we use remain underutilized and underdeveloped. Friction between IT factions prevents a feature requiring cross-discipline engagement in order for the adoption of the tool to be successful and widespread.
- Do DBAs avoid SQLCLR, source-control and test frameworks because they are a developer thing?
- Do developers avoid tools like Service Broker because it is a database thing?
DevOps, Continuous Delivery & Database Lifecycle Management
Go to the Simple Talk library to find more articles, or visit www.red-gate.com/solutions for more information on the benefits of extending DevOps practices to SQL Server databases.