Inside Perspectives: Database infrastructure – from your data center to the cloud

The Redgate icons for databases and the cloud

When I look back at my career in technology, I’m amazed at the rate of change, but also by just how much has remained the same. My first introductions to database technology were FoxPro, SQL Server 2000 and Oracle, and the applications that ran on top of them. Today, while FoxPro has been lost to the mists of time (thank the Gods…), I’m still working with SQL Server and Oracle, as well as many other specialist database systems, even if the infrastructure they run on has changed dramatically. Gone are the days of racking and stacking servers and storage, replaced by infrastructure-as-code and on-demand resources!

Infrastructure through the ages

25 years ago, virtualization was still a new concept for many organizations who weren’t running mainframes. VMware was barely established, Microsoft Virtual Server and Xen didn’t exist yet, so most workloads were run directly on the physical hardware with no abstraction layer. We had to make assumptions about capacity over a three-to-five-year lifespan for equipment on a workload-by-workload basis. Often this resulted in excessive unused hardware resource.

As virtualization technology advanced, we were able to run workloads at a higher density and get better utilization of our hardware. This still required us to plan and over-provision capacity to allow for growth. In this timeframe we also saw advances in software-defined networking and storage layers from the likes of VMware and Nutanix, giving us hyper-converged infrastructure. We could do more with automation for provisioning servers and services on demand, but were still constrained by the resources we had deployed in our data centers.

In 2004, AWS brought its first service to market in the form of Simple Queue Service (SQS), and while we didn’t realise it at the time, the ‘cloud’ was born. In late 2006, EC2 hit the market, and we got proper cloud infrastructure in a form we were familiar with. Microsoft Azure launched in 2010, and the cloud age was firmly set in motion.

The rise of cloud computing

In the early days of cloud I was very skeptical of this new wave of technology, and viewed those pushing it as snake oil salespeople! But looking back now, I was wrong, as cloud is an incredibly valuable tool for businesses.

Thinking about cloud infrastructure in the context of databases, it has really given us the opportunity to be far more dynamic and cost-effective in the way we operate. The easy button to press when getting database workloads to the cloud is simply to use IaaS and run the engines we want in things like EC2 VMs. It’s like a warm fire on a cold night, something comforting and familiar.

Another option is PaaS, including, for example, Amazon Aurora or Amazon Redshift – managed services using a database engine without needing to manage the operating system and all that goes with it to achieve a resilient database platform.

In addition to the relational database offerings, there are now PaaS offerings for specialist database engines (for document database, key value, graph, and distributed database
offerings). These allow us to go beyond relational and build systems, and mean we can pick the right database engine for the service, rather than being constrained to relational only.

More recently, running database engines in containers on Kubernetes has started to come to the fore – a middle ground providing a higher degree of control at the cost of more
management overhead. As the management tooling continues to improve in this space, we will continue to see more adoption of databases in containers where portability is a key requirement.

Organizations should be taking a PaaS-first approach to databases. Use the right engine for the task at hand and leverage the on-demand flexibility of cloud systems.

The big lie – cost saving in the cloud

One of the main reasons that many organizations adopt cloud infrastructure for database and other workloads is the prospect of significant cost savings. However, most fail at this.

Don’t get me wrong; it is possible to save on infrastructure spending by making use of cloud computing. But effort is required to achieve this, and many organizations don’t bother. Either there’s a naivety about what’s involved, or someone makes a call based on numbers and sets an arbitrary deadline. In both cases, this can lead to the same or possibly higher costs, because corners are cut to just get it done by lifting and shifting. We’ve all heard tales of organizations moving back to on-premises because they couldn’t realise the savings they expected.

If you want to save money by adopting cloud, there are two key steps to take:

1. Put in the work to assess whether the workload is right for the cloud, and which of the seven Rs1 are an appropriate pathway to dealing with it.

2. Take a long-term view of the cost over the projected lifetime of the workload you’re moving.

I commonly see an old way of thinking when it comes to provisioning capacity. Many people persist in over-provisioning, rather than sweating their resources and planning resource increases based on historical analysis. Every compute cycle, MB of memory, and GB of storage costs money, even if you’re not using it. Make use of the tools for observability, and assess where cost savings can be made through minimizing waste in the infrastructure.

“One of the main reasons that many organizations adopt cloud infrastructure for database and other workloads is the prospect of significant cost savings. However, most fail at this.”

How does the future look?

Given that I was wrong when I initially thought cloud wasn’t going to take off, what are my current thoughts on the future for cloud infrastructure?

In the short term, we’ll see a rise in organizations taking workloads back on-premises to save costs after they didn’t achieve what they hoped they would. As IT departments better understand what it takes to successfully use cloud, these workloads will ultimately move back into the cloud. However, in order to succeed, closer collaboration will be required between IT and business functions, which has to be led by culture change.

Cost optimization will become a core metric in the development of databases and their applications in the cloud. I can see code efficiency becoming a commonplace part of test suites to ensure that changes don’t incur excessive costs when they’re released to production.

As a former DBA and developer, it’s clear that current and future data professionals have exciting careers ahead with interesting technical challenges to overcome. And at the heart of this will be a much better toolkit to build the data platforms of the future.

1 https://aws.amazon.com/what-is/cloud-migration/#seo-faq-pairs

Redgate’s Inside Perspectives offer a selection of insights from industry experts on current hot topics. 

See how the cloud fares, plus other insights, in Redgate’s The State of the Database Landscape report.