Wibke Carstensen

6 October 2016

Developing the Redgate SQL Toolbelt

There have been a few changes recently in the way Redgate develops software. In a deliberate move to address ongoing requests and requirements from users, the product teams who develop each of the products have been joined by a team dedicated to developing the SQL Toolbelt.

To find out what they do, how it affects our software development process, and the advantages it brings to users, I sat down with Technical Lead, Richard Mitchell, Portfolio Manager, Elizabeth Ayer, and Product Manager, Roger Hart.

Q1: What is the Redgate SQL Toolbelt, what does it do, and what are its key benefits?

RM: I’m a keen DIY’er, so the obvious analogy for me is my DIY toolbelt. When I go up a ladder to fix the roof, I sling on my toolbelt. That way, once I’m perched precariously on the chimney, I have to hand all the tools I know I’m going to need, like the drill and pliers and so on. Plus, I’ve also got that extra drill bit I’d probably forgotten I need, so I don’t have to go all the way back down to get it.

The SQL Toolbelt is just like that: we try to stock it with everything you need to work with SQL Server. We’ve done 15 years of research into what problems our users need to solve and which tools they want us to build, so there’s not a lot missing.

EA: Yes, each tool in the SQL Toolbelt tackles a specific development or administration task, and will help you automate it, make it quicker, more repeatable, and less error prone. SQL Prompt will help you write and format T-SQL code faster; SQL Compare and Data Compare will compare and synchronize database schemas and data; DLM Automation helps build, test, and deploy databases; SQL Monitor is great for DBAs who need a simple way to monitor SQL Server performance issues. And so on.

RH: And the tools are all backed by years of experience and are not overcomplicated. They’re simple to use and suit teams with a range of training and experience. However, what seems simple, such as a straightforward database comparison, is actually a very difficult job under the covers. This apparent simplicity can be misleading. Until people assess it against alternative ways of doing the same thing, they may believe the tools are doing simple things.

RM: That’s right. I don’t believe a database developer can work without a good database comparison tool. If you need to compare, sync or deploy a database, you need to have that job done properly. SQL Compare is one example of a tool in the SQL Toolbelt you’ll use every day, because it allows you to do a task that is actually more or less impossible to do properly otherwise. It means people who are smart but not necessarily experts can deal quickly and effectively with SQL tasks that used to be reserved for a few gurus.

EA: SQL Prompt, on the other hand, is an example of a tool that just lets you work quicker. With it, you don’t have to stuff your head full of SQL. And providing good IntelliSense for SQL is another simple-sounding task that is actually a very difficult problem to solve. We’ve worked hard over many years to make sure SQL Prompt does a good job of it. It’s another tool many people will use every day.

RH: The SQL Toolbelt has a lot of useful tools you might not need so often, but when you do, it will save you a lot of time knowing you can just pull them out.

RM: Yes, like I said, the SQL Toolbelt is a bit like a DIY toolkit. You use the screwdrivers and hammers every day, spanners most weeks, but once a year you need a soldering iron and nothing else will do.

Q2: How has the SQL Toolbelt changed over time?

RH: When Redgate started out, we were producing tools to solve a specific problem, filling the gaps in Microsoft’s SQL Server offering.  Over time, we created more and more tools like this, and eventually offered them both standalone and bundled up in the SQL Toolbelt at a discounted rate. Since then, our thinking has evolved in that we generally think about the SQL Toolbelt having a set of capabilities instead of just a number of different tools. These features were developed as separate tools over time, but they actually approach a range of problems in a consistent way.

Consistent in what way?

RM: The tools all work in a similar manner and feel alike, so there isn’t the learning curve associated with having tools from different companies and learning how each one works. We’re striving to make them more consistent too. Over the last year, we’ve started working on smoothing the transitions when switching between tools. For example, the product switcher allows you to switch from one tool to another from within the UI, and bring the context of the work you were doing with you. All of this is part of treating the tools in the SQL Toolbelt like a set of features rather than standalone tools.

EA: That’s right, although we don’t want an integrated environment that excludes the use of other tools. We want to make sure that what you learn from one of our tools can be transferred to another one, like the way they’re automated, and the way you attach to servers.

So as well as being point tools, they also work together?

RM: They do. You can happily use a tool like SQL Compare on its own, but start using it in conjunction with other tools like SQL Prompt, SQL Source Control, or DLM Automation and the benefits multiply. For example, to create a test database you can create an empty database, use SQL Compare to create the empty tables and schema objects, fill in lookup data using SQL Data Compare and then use SQL Data Generator to add plausible test data. If you add the command lines, you can create any number of databases on any machine you choose with just one command.

RH: That’s an important point. With the SQL Toolbelt, rather than using one or two point tools, you can suddenly start to solve much broader database development and deployment problems. This is why the Toolbelt appeals to teams moving more toward an Agile or DevOps way of working. It helps them adopt good practices as early in the development cycle as possible – to format code consistently and clearly, document it, get that code into version control, integrate and test every day, and practice their build, migration and deploy methods.

Q3: How is the work of the SQL Toolbelt team different from other development teams at Redgate?

RM: In the past, we’ve had lots of teams working on specific products. There’s been a SQL Monitor team, for example, and a SQL Compare team. That’s still the way we do a lot of our development, but we’ve always had a gap with no one worrying about things that go beyond one product.

EA: We’ve also struggled to devote development teams to smaller tools with limited appeal.

RM: SQL Multi Script, for example, hadn’t had much love before we picked it up and the team is working hard to get through a backlog of historical requests. For SQL Multi Script alone, we’ve now pushed out seven releases and, across all of the tools we’re working on in my team, we’ve released 32 times since March.

EA: Before Richard’s team kicked off and proved us all wrong, we always thought this kind of team, working on so many different things, was impossible. Actually, having a team that has the broad perspective across all our tools has been a great benefit.

RH: It takes the two types of team working together cooperatively. There must be teams dedicated to each of our most significant tools, who learn the tool thoroughly and have a certain mindset. The team must be determined to keep shipping frequently, making improvements our customers want to see, and have a strong commercial focus directing their efforts. This is good, but it isn’t enough. There also has to be a team that looks after the SQL Toolbelt as a whole, and who are dedicated to doing the right thing for the customer. We want to genuinely improve all of the tools because most of them are very useful to our customers.

RM: There are a lot of tools in the SQL Toolbelt, so they can’t all have full-time dedicated teams. This is where the SQL Toobelt team can help. We’re responsible for SQL Backup, SQL Multi Script, SQL Data Generator, SQL Test, SQL Search, SQL Doc, SQL Dependency Tracker, SQL Index Manager, and SQL Scripts Manager.

RH: On top of this, we’re also responsible for the common systems that cross product boundaries, such as the Check for Updates mechanism, licensing, the installer, and the product switcher.

EA: Yes, which is also one of our big challenges. Sometimes, it’s hard to decide what to do next and what’s a priority.

Q4: How do you divide your time between feature requests, bug fixes, and long-term strategic changes across the tools?

EA: Generally, I’d say our approach has shifted more towards user interaction, driving changes based on user feedback and away from planning out huge strategic pieces of work. We trust developers to make calls based on their customer interactions, and on whether a particular change will make a real difference.

RM: We also have to make hard decisions about where to focus, and which feature requests we’ll tackle, and I’d be lying if I said we love all the tools the same. We will probably retire some tools with very low usage at some point. If I had to point out a naughty child, SQL Index Manager, for example, would be very high on the list. It has hardly any usage and, putting it kindly, the way it was written and architected is very interesting.

Q5: Was it easy to move from a culture where every team worked on a single tool, to this mixed approach?

RH: We had to become a lot less insular, and our code had to be much more accessible. By far the biggest factor in helping us do this was that Redgate as a whole now has a much more coherent approach to version control across all development teams.

EA: Yes, we’ve used a lot of version control systems at Redgate, like Vault, Mercurial, and Subversion. In the end, someone made a central decision and we all adopted Git. Teams were moving in different directions before then because it was difficult to share code and ideas.

RM: Adoption of Git across the Redgate teams has hugely improved the way we share changes and get them out quickly to as many customers as possible. Git is fast, distributed, and we get the benefits of Github, Gitlab and VSTS. Redgate now has several hundred repositories in Github to which all of our developers have access. Anybody can develop on anything. Just create a branch, create a pull request and when you’re done, the team who works on it reviews it.

It’s changed each team’s perception of code ownership. Before, teams tended to be protective of their code and suspicious of changes from outside the team. Now, when our team, for example, commits to master branch a fix or change that other teams need, we simply create a pull request. The other team can now merge and test on different branches and review changes in one very nice UI. We can have multiple eyes of testers and developers on it for code review. All teams, including our own, are now more easily accepting changes from outside that team.

Q6: So what are the most recent benefits of all this to SQL Toobelt users? What’s new or coming soon?

RM: One obvious benefit is that all the tools users pay for with the SQL Toolbelt license are updated regularly with bug fixes and new features. As I mentioned earlier, useful but recently neglected tools are now getting regularly updated. SQL Multi Script, for example, has much better support for projects and distribution lists, which makes it easier to configure for multiple deployment scenarios and allows sharing of settings and projects.

EA: Some of the big tools like SQL Compare have recently been updated to support SQL Server 2016 features such as in-memory tables, dynamic data masking, and row level security. But there will be more coming in that regard for other tools.

RH: We’ve also made a lot of improvements to the common back-end systems mentioned earlier – the Check for Updates mechanism, licensing, and the installer.

RM: Yes, it’s maybe not the sort of thing we talk about much but they’re important too. Previously, no single team really took responsibility for the installer. It was stable, but had a lot of significant issues that had never been fixed. Our team is now responsible for the installer and the new version will have a completely new code base. You’ll be able to install, update, repair, and uninstall our tools from the same brand new UI. The installer will now also be about 5MB, whereas the old installer was 220MB. Users will download a bootstrap installer, which grabs installers for individual tools as and when the user wants. It’s very easy to select which tools you want to install and roll out across teams. Longer term, we’re also going to integrate it really neatly with the new licensing system.

RH: The new licencing system is a big change. It’s a move away from an old-fashioned serial key model to an account model that will give users a lot more flexibility to manage their own licenses. It will make it a lot easier for teams to see what tools are activated on which machines and switch licenses to different machines.

EA: The new Check For Updates mechanism is cool too. It lets you know there are new features and new improvements for you. So you’ll quickly get to hear about big features, such as SQL Server 2016 support, rather than just a long list and bug fixes. Users can now click a button and update right there and then.

RM: Yes. It basically just makes it a lot easier for people to get the features they’re entitled to because they paid for them. And that’s a good thing.

With 15 tools, the Redgate SQL Toolbelt provides everything you need to work with SQL Server. You can find out more about it, or view our infographic on how it can improve your IT performance.