Test automation

Six Challenges in Automation Testing

By David Geer

The global test automation market was valued at nearly $16 billion in 2016, according to Zion Market Research, and is expected to reach $55 billion by 2022, growing at a rate of 23 percent each year. I think we can safely assume that the enterprise is adopting test automation.

While test automation has clear benefits, most automation efforts fail because they lack proper planning and preparation upfront. By automating software testing, you can scale and accelerate tests, reduce errors, and buy back time on software projects. That’s good. But developing a test automation strategy that suits your business needs can be like navigating an obstacle course, with scripting languages and traceability issues forming the hurdles and hedges that could stand between you and success. That’s bad.

To ensure the good outweighs the bad in your implementation, learn where six common test automation hazards lie, and how to leap them.

1. Scripting expertise: A high bar for testing talent

Companies can get stuck if their QA teams don’t have the coding skills to write test automation scripts. The test automation frameworks that enterprises employ require testers with the skills to compose test scripts using various scripting languages and frameworks.

Selenium, for example, supports different types of scripting languages such as Java, Python, and C#. And inside each scripting language, there are more testing frameworks available that also require expertise, such as TestNG for Java and Pytest for Python. While this opens up a world of possibility for those with the skills, this complexity means test automation strategies can quickly become complicated. It’s easy to see why advanced programming skills are in such high demand.

One solution for testers that want to build their technical chops is to pick a broadly applicable language like Python and then learn the fundamentals, including command scripting and code reuse, which cross multiple scripting languages. Every testing team should have at least one or two individuals who are interested in learning new skills, and what they learn can benefit the whole team. Testers who have exposure to coding practices will be better equipped to communicate with the development team.

Another option, for teams who need to transition quickly to automated testing with no coding required, is to adopt a model-based test automation solution like Tricentis Tosca.

2. When you can’t find a trace of traceability

Running automated tests doesn’t fix software any more than starting a chainsaw clears a forest. Automated tests are tools you use to identify errors so you can correct them. If you stop at generating test results, what have you accomplished?

When you see a failed test result, you need to trace it back to get the full value of the test. Is the failed test linked to a particular software build? Is it tied to a particular business requirement or feature?

If you can’t trace it back to where the application’s business logic failed, you can’t fix it or meet business goals or customer expectations. As Tricentis’s Director of Product Ryan Yackel wrote in a recent article for InfoWorld, using behavior-driven development enables you to refer back to business-readable language that tells the business which requirement the automated script is testing. Moreover, for teams using BDD, the scenarios can be easily translated into automated test scripts, which saves significant time in the testing process, Yackel wrote.

3. Quickly scaling test environments is a challenge

QA teams that don’t provision test environments in the cloud will face scalability challenges when they introduce test automation. It’s a challenge to rapidly provision the differing test environments that automated testing requires, scale them up, run the tests, tear them down, and do it all again just as fast if you’re not testing in the cloud. On premises, teams typically have a limited number of environments they can use, which means fewer tests they can deploy at any given time. As a result, testing takes much longer.

Even if your organization does not plan to move applications that have traditionally been on premises into cloud environments, you can make a case for moving test environments to the cloud. Not only will this make it possible to scale test automation, it will also reduce the QA team’s operating costs. Deploying a test environment in the cloud means you can quickly build it, deploy the software, run the automated tests and then tear it down, paying only for the time the environment was used.

4. Too many UI tests can break testing

You shouldn’t rely too heavily on testing the UI, says Yackel. CSS and XPath locations in the UI change often. If you target attributes like these in automated tests, it can lead to false positives and continued maintenance when the changes weaken or break the tests.

You need a bottom-up strategy that includes unit level testing for the API, as well as UI testing, says Yackel. Because the API is not as subject to change as the UI, your overall testing will be more consistent.

5. A lack of transparency can inhibit automated software testing

Test automation can lack visibility when different teams are using different, disconnected automation strategies, says Yackel. If each team is using different test automation frameworks from Eggplant to homegrown solutions to Worksoft, it can be tough getting insights into total software quality because no central portal reports on it all.

You need a centralized view to get a good cadence of tests that are running, tests that aren’t, and what you have delayed, says Yackel. This transparency enables leaders to see across automated testing.

6. There’s no tool for shifting culture

Adopting test automation requires a culture shift, an evolution of behavior and thinking. Often, team members and stakeholders look at throwing tools at a problem, which doesn’t fix the underlying mindset, says Bryan Osterkamp, Technical Architect Principal at USAA. A people-first approach works much better.

USAA, a provider of insurance and financial services to members of the military and their families, moved to test automation by seeding development with smaller teams who saw the potential of automated tests.

We helped those teams succeed with automated testing so they could be examples for other teams while showcasing results such as the defects that testing caught, says Osterkamp.

The road to test automation can be winding and unwieldy, but it is worthwhile. Get some useful automation testing metrics under your belt, like more bugs found and fixed, and I’m sure you’ll agree.

About the Author

David Geer writes about cybersecurity and technology including DevOps for national and international publications. David’s work appears in publications from IDG in the U.S. and around the world in several languages. ScientificAmerican, The Economist Technology Quarterly, IEEE Computer magazine, and many magazines and companies have used David’s content. Visit David’s Google Scholar Page here.