The Customer

As Verizon continues to modernize, the organization has been focused on expanding its reach beyond core telecoms and wireless offerings. These business areas include Edge Computing, providing a broad array of managed services to corporate and state clients, and its highly successful IoT product line. Verizon has been a leader in the IoT space for many years, harnessing the power of connected business to help organizations “enhance customer loyalty, keep tabs on high-value assets and boost efficiency”.

The heart of Verizon’s IoT product line is its Fleet Management Solution, developed by Verizon Connect. Created in 2016 by the acquisition and integration of Fleetmatics and Telogis to Verizon Telematics, Verizon Connect offers the most comprehensive fleet management products and services in the industry.

We sat down with three technology leaders from Verizon Connect to talk about their journey to reduce technical overheads by implementing a standardized deployment automation process using Flyway, from Redgate Software.

Introducing:

• Dan Bowyer – Senior Data Architect
• Dave Murphy – Senior DevOps Engineer
• Enda McCallig – Tech Lead

The Challenge

The integration of three industry-leading SAAS companies created a formidable combination of talent and technology. However, it also presented a significant set of challenges associated with the need to manage a wide variety of delivery processes. One example of this was connected with the use of Microsoft Dacpac. Some dev tasks were considered too risky, and so those options were disabled. This meant that certain changes would need the involvement of a DBA to manually run some scripts pre and/or post deployment of the Dacpac. This created a dependency between the development teams and the DBAs, resulting in delays in releases.

Another example lies in the multiple RDBMSs and deployment patterns. Between the three companies there were 4 different database engines in use, and 7 different deployment patterns. Naturally, this presented a huge overhead for the DevOps teams, but it also meant that 120 Full-Stack Engineers had to learn multiple deployment patterns, rather than focusing on delivering new capabilities.

To compound these challenges, Verizon’s focus on leveraging AWS services meant that there was a desire to use databases such as Aurora and Redshift. As many of Verizon Connect’s legacy products were built on Microsoft SQL Server, teams were familiar with Microsoft’s Dacpac deployment model, but, unfortunately, that couldn’t be used across these platforms.

The Solution

In 2019, Verizon Connect’s DevOps and Data Architecture leaders set out to identify a solution that would enable them to standardize their database deployment process across the organization. Their key criteria were:

• Ability to provide a consistent process across platforms
• Minimal complexity during onboarding
• Provide a platform for local automated testing

According to Dave Murphy, Senior DevOps Engineer, there were only two realistic options on the market that could meet their criteria – Flyway and Liquibase. After an evaluation of both tools it quickly emerged that Flyway was the option that best fit the criteria, making it the final choice for their database deployment automation solution. Flyway’s simplicity and focus on native SQL meant that the barrier to entry would be minimized for adopters while giving them total control over how changes are deployed.

The next step was to define a process that could be replicated across the division. Dave and team got to work defining a versioning, build, test, and deployment workflow compatible across all of their target db platforms:

• Amazon Aurora PostgreSQL
• Amazon RDS for SQL Server
• Amazon Redshift
• Amazon RDS for MySQL

Their implementation enables developers to build and run unit tests against their changes in a Docker container. Flyway’s open framework provides transparency to what the build is doing, so developers get instant feedback on their work.

The process works like this:

Because the automated testing is complete before anything is checked into source control, developers have a very high level of confidence that once changes are pushed they will work as intended. As this was a big change for the teams involved, there was some reluctance to embrace the requirement to test so thoroughly, but once they saw the benefits, most quickly got onboard.

The Results

According to Dave, Dan, and Enda, the benefits of the transformation have been felt in several different areas:

• Reduced overhead on the DevOps team
  As of July 2023, all databases at Verizon Connect have been onboarded to Flyway pipelines. Every deployment follows the same process, meaning developers can move teams with no re-training required for database releases. Additionally, working with standardized pipelines means that if something does go wrong, troubleshooting takes far less time.
• Eliminated need for production rollbacks
  The implementation of local builds and unit testing means that developers catch issues far earlier in the development lifecycle, which has eliminated the need to rollback changes in production so far this year. While this is always the goal, to be able to achieve it in an environment of such scale and velocity of change is a fantastic achievement.
• Ability to meet regulatory requirements via deployment automation at scale
  Over the past couple of months Verizon Connect have had to make some large changes to all of their systems to comply with changing CPRA regulations. The consistency of their Flyway implementation meant they were able to drop the code from a template database into any of their repositories, test it thoroughly, and with a few small metadata changes, deploy it out to hundreds of databases.

From the team’s perspective, it saved them from missing hard deadlines. Dave put it this way:

“The organizational benefits are huge. To be able to have consistency across all projects means you can automate a lot more. That makes these sorts of large changes much more straightforward than they would have been, especially if we still had all the previous deployment models in place”.

Dan added that: “Without Flyway we wouldn’t have hit the hard deadline for the CPRA compliance piece. I’d also add that it’s helped us to shoehorn a lot of best practice changes into the migration to Flyway, which is great. From the database side it’s got to be the most successful project that we’ve taken on board in the 12 years I’ve been with the company”.

And finally, some last thoughts from Enda. When referring to the constraints on DBA availability and the impact of introducing Automation, “..this has been a MAJOR win for the team”.

Next Steps

While the project has successfully delivered against its initial ambitions, there’s room to make more improvements. Decoupling DBAs from deployment is a big step that is within scope. As Flyway requires the developers to write the actual migration scripts, it means those that were previously considered risky in Dacpac are now less likely, and the need for the DBA to be involved in the deployments has greatly reduced. It will allow the development teams a lot more autonomy, while also relieving the demands on the overstretched DBAs.

The longer term goal is to establish full confidence in the deployment process used by developers so that fully automated deployments become a possibility. The idea is that once the dev team has completed the required changes on their application and database code, and it has been reviewed, approved, and merged, then the deployment to production could be fully automated. This would only be possible with tooling such as Flyway in place, but would also require more comprehensive testing, both unit and integration, to allow dev teams to fully trust the process.

As the process improvement progresses, and more reporting on unit testing results and static code analysis is integrated (Sonarcube is being considered), then the team are looking at shifting a large portion of integration testing to the feature brand level. This further increases the level of confidence in code delivered before it is merged into a shared branch, and deployed onto a shared database. In theory once the code is merged then a developer should not have to touch it or even think about when it will be deployed. This creates more time for developers to focus on their chosen profession, coding, and solving technical challenges, rather than having to drop feature work to supervise a deployment as it progresses through the various environments. The team are not ready for this yet but have now set a solid foundation upon which they can now seriously consider working towards this goal.

In future, there is also a plan to evaluate the benefits of providing their developers with data containers populated with masked datasets, to provide further validation their changes will work as anticipated when they hit production.

Conclusion

Implementing Flyway across Verizon Connect helped to remove the barriers to unlocking value from their target cloud database platforms posed by managing 7 different deployment patterns.

The core benefits were seen in three key areas – first, a reduced overhead on the DevOps team, enabling them to focus on enabling speed and quality of delivery. Second, the elimination of the need for production rollbacks through shifting testing left. Finally, the ability to respond at pace to changing regulatory requirements by deploying changes to hundreds of databases with minimal effort.

If you read this and thought – my group or team have struggled with those same challenges, we’d love to talk to you. Luke Porretta is Verizon’s Redgate Account Director and has been working with various groups around the organization since 2019. You can reach him at luke.porretta@red-gate.com.