Maxime Drobot

11 August 2023

How to mitigate the challenges of data growth

Over the last decade, I’ve rarely met a data professional whose organization wasn’t experiencing data growth and making more demands of their data. We build and deploy new applications faster than we retire old ones, and new data is accumulating dramatically faster on our existing systems than our ability to decide to delete older information. Additionally, the ever-growing number of users and devices interacting with that data increases the strain on the infrastructure underpinning it.

The growth of data is well documented, with IDC’s Global DataSphere Forecast, 2021-2025 predicting that global data creation and replication will experience a compound annual growth rate (CAGR) of 23% over the forecast period.

While not all this data will be saved, IDC also expects a five-year CAGR of 19.2% in the installed base of storage capacity over the same period. That’s a lot of infrastructure and servers to accommodate and, whether it’s on-premises or in the cloud, all the data will need to be managed and monitored.

While for some organizations, the growth in data looks like it will be explosive, for others it will be more gradual. Invariably though, the direction is always up, and with this comes a relentless increase in the size of the infrastructure to manage.

Alongside size, complexity is growing too

It’s not just the growth in data being stored that’s an issue, it’s also where it’s being stored. Behind the scenes, the number of different databases and platforms being used across every business sector has been steadily increasing.

Flexera’s 2023 State of the Cloud Report, for example, shows that 87% of businesses now use multiple cloud providers, while still having on-premises infrastructures. And, as revealed in the Stack Overflow Developer Survey 2023, developers are now using an average of three databases, the most popular being PostgreSQL, MySQL, SQLite, and SQL Server. So they’re moving from working with on-premises to cloud, with flavors like Azure and AWS, and across database types as well.

As this complexity grows, so do the challenges. In addition to security and compliance concerns, growing infrastructures place demands on teams, estate performance, and staffing and recruitment. As a result, one of the toughest jobs for team leaders is judging when to expand the size of the team, when to try absorbing the change through reprioritization, or when a more significant rethink of approach is needed.

Budgetary pressures have taken the first option of adding capacity to a team off the table for many organizations, at least in the short term. Reprioritizing is also easier said than done. Real care must be taken to move forward in a way that meaningfully frees up capacity in the team, rather than just creating the impression of doing so. Getting this option wrong can lead to the team becoming overworked or burning out.

Many leaders are therefore looking to a third option as the way to scale their team – improving the fundamental estate management approach so that the same outcomes can be achieved more efficiently.

The appeal of introducing a third-party monitoring tool

There are several opportunities here, such as introducing a more advanced and comprehensive third-party monitoring tool to take the daily strain off DBAs and help them focus on the more complex problems faced by their organizations. Embracing greater use of automation also saves time, and tasks which might once have been performed server-by-server can be achieved in a single, scalable operation.

Cloud databases, particularly Platform as a Service offerings like Microsoft Azure SQL Database, Managed Instances, or Amazon RDS, offer more help by handling mundane operational concerns like backups or updates for DBAs, freeing them to manage larger environments.

Additionally, taking a more proactive approach is key: the more infrastructure you manage, the more systems there are to go wrong. Reacting to these incidents can easily swamp a team, so understanding and improving the health of your growing environment in a timely manner is essential to ensure performance and security. Look to leverage tools that allow you to see any critical issues on your servers that need addressing at a glance, with the ability to quickly drill-down into detail and analyze performance metrics.

An important aspect of this is having a tool that can monitor databases across platforms, whether on-premises or in the cloud, and which can present an overview of every server and instance from a single pane of glass. This gives data professionals the opportunity to see the real-time status of all of their databases instantly.

Finally, make sure the tool enables teamwork and cooperation to further increase the efficiency of your workforce. For example, web-based server monitoring tools make information easily accessible to everyone on the team, regardless of location, device or time zone, enabling problems to be identified early and solved quickly and collaboratively.

You should also check that your monitoring solution can integrate with collaboration tools for sharing alerts and collaboratively solving issues. Tools like Slack, Microsoft Teams or Jira support quick communication and reduce the confusion which can come from people working from email inboxes.

Summary

While data professionals are becoming accustomed to the growth in data and the increasing size and complexity of database estates, the consequences are sometimes hidden away. The ever-rising workloads on DBAs; the need to maintain performance SLAs; the obstacles to staffing and recruitment; the increasing need for teamwork.

That said, it’s worth taking stock of how your situation has changed at least once a year. Has the infrastructure your team is responsible for grown over the last 12 months and by how much? As you’ve taken on that extra workload, have you done less of something else and reduced the quality of the team’s overall output or contribution to the business?

Considering how to proactively and sustainably support a growing environment has never been more critical. As your infrastructure grows, someone needs to make it work. Investment in a team through the provision of tools can increase efficiency and mitigate the ever-increasing demands from a business.

This will allow you to shift from being reactive and constantly firefighting, to being proactive and able to fix issue before they even become issues.

To discover more about how Redgate Monitor can help you face the challenge of growing database estates, try the online demo, or visit the solutions page.