To Automate or Not To Automate…Is that the Question?

We all know that test automation i.e. the ability to run a bunch of tests automatically, not manually, and then release based on the results, is a good thing, don’t we?

Effective automated tests provide insurance that the areas of your product that are tested are not broken from one build to another. It doesn’t sound that sexy but, realistically, most projects must have an effective and reliable automated test suite.

To put this into some context, in a recent development project at Red Gate (on SQL Source Control), our automated tests ran, every night, the equivalent of two weeks’ worth of manual testing. In so doing, we freed up a significant amount of our test engineer’s time to do what they are good at, finding nasty, obscure bugs, rather than checking what worked yesterday, still works today. It also gave us an immediate indication of the quality of a build that anyone could look at, by using a dashboard.

Automate Everything = Sensible goal?

So it’s a no-brainer then…let’s automates the crap out of everything, right?

In a word…NO!

Test automation is just one tool in the arsenal of a Test Engineer (TE) just like exploratory testing, acceptance testing and so on, and we should treat it as such. It certainly isn’t the silver bullet or the answer to life, the universe and everything!

Sure, test automation has many benefits, and they are not hard to write…but it is very hard to write them well. Ten badly written automated tests will require more maintenance and give more headaches than 100 well-written ones.

The other major consideration when planning your test automation is the investment required to set up your automation system. Writing a solid, reliable and maintainable test automation system is a major undertaking, and it may be more prudent to use the Software Engineers on your team. After all, they are the coding experts! This doesn’t mean that, as a TE, you hand all the code over to the software engineers, but you use their expertise to design it, and get it stable. Pair with them and learn better coding techniques; I’ve found that invaluable.

If your TEs devote too much time writing or maintaining automated tests then you have the same problem as you do when you have little or no automation…you’re removing them from what they do best, which is beating the life out of a product looking for devious bugs!

The triangle of automated testing

Having test automation on your project is a no-brainer but the sensible question is to what extent to automate? The triangle of automated testing shows how you should prioritize your automated tests.

  • triangle.pngUnit Tests – these should form the bulk of your automated tests. They are fast, reliable and test discrete bits of code. The Software Engineers should be writing them for any new code; it could even be part of determining whether a task is “dev-done”. Encourage your developers to do lots of these!
  • Integration Tests – Where something cannot be unit tested, or external sources are required (databases, file systems etc.), then write an integration test. These are typically slower, and require a lot more coding. Do a fair amount of these.
  • GUI Tests – For testing functionality that is only present through the UI then do GUI tests. These should be the last resort. Yes, they are the ‘sexy’ tests, but they are much more expensive to write and maintain. Do few of these!

Summary

Test Automation is a valuable milestone to achieving frequent, reliable releases, but it can also be a burden, and a drain on resources. The key is to balance the value of adding automation with the time it will take to do so. Some tests just aren’t worth writing and are quicker and more reliable to perform manually.

Plan your automation carefully, use your software engineers to help you craft a core test framework, and try to push your tests down the stack (towards the bottom of the triangle) as much as possible.

Finally, don’t be afraid to delete tests. Unreliable or broken tests that will require significant effort to fix aren’t usually worth the effort.