Republished here with kind permission from the author.

UI Automation: Before You Even Start

Welcome to the first of two articles on being successful with user interface (UI) automation. The goal of these articles is to help you start asking the right questions to begin building high-value, maintainable automation in place for your projects. Once you’re past the basics, I hope to help you find the right questions on how to evolve your testing over time to continue adding value to your project.

You’ll notice I’ve mentioned “questions” a lot; answers, not so much. UI automation is a difficult domain to work in, and there aren’t any magic “best practices” other than one: use your brain. This short series will hopefully give you some ideas on how to look at your specific environment, and how to create a new (or tailor an existing!) automation strategy.

Why do UI Automation in the First Place?

Your first question about UI automation is one that’s too often missed: why do UI automation at all? Aren’t unit, integration, and manual/exploratory tests enough?

We in the software industry don’t ask “Why?” anywhere near as often as we should. UI automation is a significant cost in your overall development cycle. It’s important you understand if it’s worth the cost before jumping in!

UI automation ensures you’ve got tripwires around your most critical business value features in your system. Hopefully you’ve got a solid amount of test automation at other levels, but while unit and integration tests are critical to a system’s overall automation approach, by definition they don’t cross all boundaries of a functional slice.

I often refer to a line my friend Adam Goucher uses when discussing why teams should do UI automation: “Show me the money!” he says, quipping a line from the film Jerry McQuire. Creating solid UI automation ensures you’re guarded against regressions around business-critical features, like your customers adding items to a shopping cart and checking out-in other words, giving your company money.

Teams should not look to UI automation as a replacement for the 4,942 manual test scripts they have in place. That is a painful path that’s guaranteed to lead to failure. Those manual scripts should be evaluated to understand what critical “Show me the money!” features and use cases they’re guarding.

Setting Your Team up for Success

Once you’ve gotten a good grasp on why you should be doing UI automation for your project, it’s time to move on to setting yourself up for success. Getting a solid set of expectations from management and within the team is critical if you’re going to move forward.

Setting Expectations with Management

Before you jump off the UI automation cliff, you need to ensure your management chain understands the costs associated with getting this sort of automation in place. You many also want to ensure they understand the benefits, too… (By “management” I mean your leadership, stakeholders, and customers.)

UI automation is a software development task, and it takes time. Some of the impacts will include:

• Time on the schedule to evaluate what tools to use
• Time on the schedule to get the tooling set up in your environment
• Time on the schedule to get your team’s skills up to speed on the new tooling
• Significant time (and patience!) on the schedule to adjust your development processes
• Time on the schedule to get your UI automation work done

You’ll note my overly repetitious repeating of the phrase “on the schedule.” Too often management expects that UI automation effort won’t impact the schedule. This is a recipe for frustration, missed deliverables, and ever-worsening team morale. There’s no way you’ll be successful in the long run if management doesn’t acknowledge there will be, at times, significant impacts to the velocity you’re delivering at.

You’ve got to balance this decrease in velocity with understanding the wins you’re getting in other areas: time freed from repeating senseless manual regression tests, clearer information on your system’s overall risk to business value, and an increase in confidence around your automation safety net.

Setting Expectations with the Team

No UI automation project succeeds without involvement from the entire team. I’m a firm believer that teams throwing software over a fence to some “automation team” isolated from the rest of the group are bound to fail, or at least see only mild successes with an extraordinary amount of pain.

Starting a UI automation effort means you’ll need to get your entire team involved throughout the entire process. Developers and testers will need to work together to get the best automation in place. That means getting discussions in place early in the process-before a single line of code is written or updated-to talk about how automation will be handled for the particular set of work items being considered.

Developers and testers have to get the expectation in place, from the outset, that they’ll need to be working closely together to create testable user interfaces. They’ll need to agree on small things like strategically placing static or pattern based ID values on certain elements; creating backing APIs for setting up and tearing down test data; building switches to turn on and off system configuration options like CAPTCHA or rich text editors; all these and other similar tasks have to be accomplished by pairing up the right people on different team.

The Hard Work’s Before You Start

UI automation can bring great value to your projects, but you’re doomed for an incredible amount of pain if you don’t lay the groundwork for success before you write a single line of code.

Ensure you know why you’re getting into UI automation. Ensure you know what you want to accomplish with it. Above all, take a great deal of care in setting reasonable expectations with your management and team.

As I said some time ago on Twitter:

UI Automation is hard, regardless of what toolset is used. The core problem domain is hard. You WILL struggle, you WILL fail, you MUST learn

-Jim Holmes (@aJimHolmes) July 31, 2013

Preparing for Automation: Selecting the Tools

Selecting the right toolset for your automation is a critical part of your project’s success. Choosing an ill-fitting toolset doesn’t guarantee failure, but you’ll spend an extraordinarily frustrating amount of time on the mechanics of creating automation: learning the tools, writing tests, running tests, fixing broken tests, adapting tests to system changes, etc. All of that takes you away from your primary value as testers: Actually testing the system!

Getting the toolset right, or quickly changing away from an ill-fitting toolset, means you’ll spend less time struggling with the mechanics. Your team will be better integrated with your testers, your stakeholders will be getting better information, and your testers will have time to test versus chasing broken tests.

Goals for Automation

At the start of this article I asked “Why do UI Automation in the First Place?“

It’s a perfectly valid question, and one your team needs to answer. As part of answering that question, get distinct goals in mind for your automation effort, such as (for example):

• Increasing communication about testing across your team
• Helping cut down the number of regression bugs that escape to your customers
• Helping your team build what your stakeholders actually need
• Freeing your testers from rote regression tests so they can do more important testing

Note I never once mentioned metrics like “ensure 100% test coverage,” “create 5,000 automated test cases,” or “convert 95% of manual tests to automated.” Those are incredibly awful, horrible, nasty, smelly metrics that in my view have no place in any project. They’re extremely shallow metrics that don’t indicate the value of tests. Make your goals something that’s more effective in helping your team deliver true value to your customers.

A Tangent: Nomenclature

Before we head off much further into the depths of selecting tools, let’s take a moment to lay out some terms that will help us keep clear the various moving parts involved in automating web browser tests.

At the lowest level, naturally there’s the browser itself. The browser navigates to web resources, loads and renders content, executes JavaScript, etc.

The next level up is the driver. This is an automation tool such as Selenium WebDriver, Microsoft’s Coded UI, or Telerik’s Testing Framework. The driver is responsible for communicating commands to the browser (navigate, load, etc.) and inspecting the page inside the browser. Generally speaking, drivers don’t actually execute the automation tests themselves. Additionally, most drivers don’t even know how to do actual checks or comparisons (“Is this heading block correct?” “Is the button visible on the page?“) Drivers simply pass instructions to the browser and pull back bits of information from the browser. Frameworks reside on top of the driver to handle these functions.

A testing framework fills the test execution and comparison/checking gaps. Frameworks such as JUnit, NUnit, or MSTest provide containers in which driver code can handle the browser operations. Testing frameworks also provide comparison and assertion methods for validating conditions on the page.

Specification or language frameworks enable teams to write specifications in a more natural grammar. Drivers and testing frameworks don’t communicate intent particularly well, especially to analysts or stakeholders. Frameworks like Cucumber, Saunter, Fit/Fitness, and Robot offer up ways to express intent using grammar similar to “As a registered customer, I want to add bacon to my cart so I can purchase it.”

Commercial tools such as HP’s QuickTest Pro, Microsoft’s Visual Studio tools, or Telerik’s Test Studio often abstract away some of the direct browser interaction at the low-level driver. Good tools in this vein still provide you direct access to the driver and browser via unfettered control at the driver or code level.

Who Will Use It?

With nomenclature out of the way, clearly understanding the users of your testing toolset is critical when selecting the tools.

If you’re looking to involve your stakeholders in creating specifications that tie directly to tests, then you’ll want to look to a tool that gives you something akin to plain language. Cucumber, Fitness, and Robot are examples of specification-style frameworks that fill these needs.

Teams that will rely more on developers and coding testers to get their tests created may be more interested in staying straight at the driver level (with a test execution framework), or perhaps a BDD-style framework such as SpecFlow or Saunter.py.

What’s the Environment for the Tests?

Your system’s environments play a huge role in deciding what toolset you’ll select.

Are you planning on running your tests only at your developers’ or testers’ systems? (Please say “No!“) If not, here are some things to take in to consideration:

• Do you have multiple environments for development, testing, pre-production, and production?
• Which environments will you be targeting for testing?
• Who owns deployment to those environments? Do they require a manual push process, or can you tie your automated tests to some form of automated build process?
• Are you able to deploy tests in an open source tool to your pre-production and production environments?
• What’s the workflow for getting things like execution agents and platform requirements deployed to those environments?

Many larger corporations have multiple environments, each with different deployment requirements-especially when those corporations are under regulatory compliance in the financial or healthcare domains, for example. If you’re working in these environments you’ll likely need a toolset that support easy configuration for things like base URLs, database connection strings, filesystem resources, etc. Note that by “easy configuration” I really mean “the ability to write custom configuration libraries in as simple a fashion as possible” because few toolsets have out-of-the-box support for complex environments!

What are the Current Technologies in Use?

Choosing an automation tool isn’t wholly dependent on the technologies your team’s using to develop your system; however, it often has an impact. For example, if you’re working on a Windows Forms project, you’ll have fewer options than if you’re helping test a Ruby on Rails web application. There are a number of technology factors you’ll need to understand about your environment:

• What is the basic architecture of your system? Thick client? Stand-alone desktop application? Web application with mobile support?
• What language is your system being developed in?
• What devices will your users access your system through?
• What components are running in your application’s architecture? (Web server, business layer, database server, distributed content network servers, etc.)

Good automated UI tests don’t rely just on the user interface; they use test oracles to probe the underlying system to ensure databases are properly updated or system configurations are altered. This means you’ll need to understand how to interact with your low-level system’s APIs. Are there appropriate web service endpoints you’re able to access? What drivers are available for your database if you need to create direct connections?

Just because a system is written on one platform doesn’t mean you’re restricted to the same language or platform for your tests! This is particularly the case with web applications. Use Telerik’s Test Studio to cover Rails applications. Use Adam Goucher’s Saunter.py Python framework to automate tests for .NET applications. Use Jeff Morgan’s Ruby Pages gem to drive out functionality on Java applications. Even in the WinForms domain you can look to Ruby code that will exercise your Windows desktop applications.

While the technology stack does put some constraints on your automation approach, more critical is your team’s makeup and skills.

What’s the Team Structure?

How your team is structured has a tremendous impact on your automation efforts. I’m a firm believer the best chance for success in any automation project requires involvement from the entire team: stakeholders, designers, developers, business analysts, and of course testers.

If your team suffers with a much more constrictive, stovepiped structure then you’ll want to avoid tools that encourage more collaboration across those boundaries. Instead, you’ll want to focus on tools that will boost your testers’ productivity, perhaps at the cost of some overall flexibility.

Perhaps your developers can’t or won’t assist in actually writing the tests, but they may be available for writing backing infrastructure to handle setup, teardown, and configuration. They may also be able to write appropriate test oracles for your testers. If this is the case, then your testers will be able to take advantage of just about any toolset they’re comfortable with. Being able to rely on developers for the “heavy lifting” of writing calls to web services or the database can let the testers focus their main skills of building great tests.

What are the Team’s Skills?

You’ve got to allow for your team’s skills when selecting an automation toolset-and not just the current skills, but those that can be learned while adopting a new toolset.

Are you choosing test tools for a developer-centric group comfortable with .NET? Will you have time to learn a new language like Python or Ruby in order to use an interesting toolset like Ruby Page Objects or Saunter.py?

Has your team had previous automation experience in the technology stack you’re currently working in? Do they understand things like the page Domain Object Model and asynchronous operations? Does your team understand how to build well-designed software-and that those same design principles are critical to well-crafted test suites, especially if you’re using a completely coded approach like WebDriver or Watir.

How Long Will You Need Your Automation?

It’s not enough to just consider your team’s skill as you’re building up your automated suite. You have to take into account how long your system will need the automation you’re creating. If you’re relying on a smaller open source project, will it be around in three or four years, or is it likely you’ll need to change to a new toolset? If you’re looking to a commercial tool, will you have budget in the out years to continue getting support?

You also need to consider the time it will take to onboard new team members. Will they have to learn new development languages in addition to getting up to speed on all the other testing activities in your organization?

Try Things!

Once you’ve worked through the long list of questions I posited above (you’ve likely noticed I gave few answers!), you’ll need to put in time exercising a few of your top options for toolsets. Part of your selection process absolutely needs to include time on the schedule doing real work with the tools to see how they “work in anger.” An extended prototype period or proof of concept is the only way to get a good feel for how the tool will work for your teams.

Get time on your schedule for working through your tools, and make sure it’s adequate time for a serious evaluation. Ensure you get everyone, EVERYONE, on the team involved in the trials. You need to understand which of your team will have difficulties with what tools, and which of them loves the other tools.

Selecting a tool is a significant effort. Your team needs to make sure they’re comfortable as a group that the chosen tool will help bring value to your software efforts, not just turn into yet another technology time sink.

Keep in mind the goals for your automation that were discussed near the start of this article. Every one of your selection criteria ought to align with those goals. Otherwise your team’s in for a long, frustrating experience!

A Smattering of Tools to Consider

Here are a few tools you may consider during your evaluation. The list is far from complete, so don’t rely on my article for your single source of potential toolsets!

Summary

I’ve laid out some factors I’ve learned are critical to address before starting your UI test automation efforts, or at least as quickly as possible once you’ve started! Thus far, I’ve likely left you with more questions than answers. That’s OK, because the process of getting any good testing activity in place should leave you with lots of questions!

We’ve dipped into some of the most essential questions you and your team need to be asking, and I’ve suggested some factors and considerations you might need to bear in mind as you assess your UI automation needs.

Finally, to get you started, I’ve given you a non-exhaustive list of testing tools and frameworks for you to consider as a starting point, but I strongly urge you to investigate based on what you and your team need! New tools are being built all the time, so it’s always worth doing your homework and then assessing what’s right for your project.

Thus far we’ve been tackling User Interface automation at a fairly conceptual level, so in my next article we’ll dive down into some of the practical mechanics!