Report – Optimizing test automation efficiency in rapid release cycles

Author of the post: Bas Dijkstra

For decades, test automation has been seen as crucial for development teams in supporting their testing efforts and retrieving the product feedback they need. However, many of these teams still report challenges when it comes to test automation: it is expensive, it takes a long time to get the feedback they’re looking for, and the feedback that they do get is often not reliable enough to support critical business decisions.

In this report, we will zoom in on these challenges and present steps towards a holistic test automation strategy that gives teams what they’re looking for: valuable feedback, fast.

The business cost of inefficient test automation

Test automation tools have existed since the 1970s. However, while the tools since then have without doubt become more powerful and sophisticated, many teams still struggle with their test automation implementation.

Test suites that take hours or even days to run. Large amounts of false positives and false negatives. Teams that have to spend much more time than they want, or than they even have in the first place, just to keep the tests running.

While there is very little factual evidence of the direct impact on businesses, research indicates that there is plenty of room for improvement in test automation (all numbers from the 2024 State of Testing survey ):

• 9% of organizations who responded still have no test automation in place
• 42% of respondents indicate they do not feel confident in writing test automation code
• 60% of organizations have test cases that are not well-written or well-maintained

It is safe to conclude that a lack of efficient test automation leads to either or both of these outcomes:

• Teams spending a lot of time performing tasks manually that would benefit from automation
• Critical but repetitive testing tasks not being performed due to a lack of time or priority, leading to defects leaking to production

In this report, we will present a number of recommendations for teams that do want to improve their test automation efforts and get (back) on the road towards test automation success.

Typical root causes of slow and inefficient test automation

If test automation is all about getting valuable product feedback, fast, then why does it too often fail to live up to the expectations?

There are a number of root causes that contribute to test automation that is slow, expensive and unreliable:

Poor product testability

In my experience, the most important contributor to suboptimal test automation is a lack of product testability, or the degree to which a product enables us to uncover information about its state and behaviour in an efficient manner.

Some examples of how low testability manifests itself in products are

• unstructured code that is hard to unit test, leading to a need for tests with a larger scope to test code behavioura
• lack of clear separation of concerns between components, layers and (sub-)systems, leading to teams having to test behaviour of these individuals components using the system ‘as a whole’
• a lack of (access to) APIs that expose data and backend logic, leading to teams having to resort to other, less efficient means to test backends
• graphical user interfaces without unique identifiers for key elements, leading to teams being forced to use suboptimal locator strategies

All of these examples will lead to less efficient automation, and in some cases, even to automation being virtually impossible to implement in the first place.

Insufficient (technical) skills

Another factor that plays a role in inefficient test automation is a lack of skills in the individuals and teams responsible for the implementation, execution and maintenance of that test automation.

Test automation is a software development activity intended to support testing, meaning it requires both significant software development and software testing skills.

Only when those involved in test automation have good software development skills will the test code be readable, maintainable and generally well-structured. If the test code is of poor quality, it will take longer to maintain it when the product under test changes, to find problems with the test code in case of failures, and to onboard new team members in the test code base.

Using ‘low code’ tools and solutions (AI-powered or otherwise) is not the solution to this challenge, despite what the marketing efforts of some of these tools would like you to believe. These tools, too, require solid engineering skills to be implemented successfully.

Test automation as an afterthought

Another problem that often leads to poor test automation results is treating test automation as an afterthought instead of an integral part of the development process.

Seeing test automation in this way effectively means test automation is a ‘nice to have’, and something that can be postponed when time is scarce (and when is time ever not scarce?)

Also, starting your test automation efforts only after the product has been designed and developed means that the moment to have a conversation regarding how to test the product and how to use tools to do so efficiently has passed. The right moment to have a conversation about testability is before you start development, not after it has been completed.

Test automation truly shines when it is created, executed and maintained in lockstep with the product.

⇒ Bottom line

Typical root causes of slow and inefficient test automation are:

• A product with poor testability
• A team lacking the skills required to implement test automation at scale
• Test automation being treated as an afterthought
• Leadership moves to remove testing and automation bottlenecks

So, what should leadership do to remove these bottlenecks and free the way for test automation that actually delivers on its promise? Let’s take a look at a few suggestions.

Think and talk about testability

As we discussed before, testability matters when you want to succeed with test automation. In other words, your product should allow for easily observing the behaviour of the system or parts of it in a controlled test environment.

Reverse engineering testability into an existing product (the dreaded ‘codebase refactoring’) is not something teams will look forward to. Also, it presents us with a catch-22 situation:

• to be able to write tests for a code base, we need to refactor it to improve testability, but
• to be able to refactor the code base without the risk of breaking things, we need tests to act as a safety net

Therefore, the best time to talk about testability is generally ‘as early as possible’, or more practically, when the behaviour of a new product, feature or bug fix to be implemented is discussed. For many teams, this will be during some form of refinement session.

During discussion of the intended behaviour of a feature, make it a habit to also ask:

1. How can we verify that the feature behaves in the way we expect it to behave?
2. How can we repeatedly perform that verification in the most efficient way possible?
3. Does our product allow us to perform that verification in the way we want to? If not, what can we do to improve the situation?

Generally speaking, test automation will be most efficient when it is written as close to the code it verifies as possible, both in terms of scope (don’t write integration tests for something that can be verified with a unit test) and in terms of time (don’t write your test automation too long after writing the code it verifies).

Make test automation a whole team effort

The spectrum of skills required to do test automation well is very hard to find in any individual, which is why so many companies struggle to find the elusive ‘test automation engineer’ or ‘software development engineer in test’. In my own 20 years of experience, I have only worked with a handful of people who were really good in both development and testing.

Rather than spending a lot of time and effort to find or grow individuals that can ‘do it all’, make test automation a whole team effort, instead. Find people who are great at software testing and people who are great developers, and have them work together on your test automation throughout the development process.

This collaboration ideally should be synchronous, with people with complementary skill sets pairing up in real time to work on a specific task. Synchronous collaboration provides a number of benefits over asynchronous work (like through a pull request process), such as:

• Instant reviews of work being done
• Frictionless knowledge sharing
• A much lower chance of misinterpretation of any part of the work

But what if your team does not yet have the range of skills needed for test automation success? Well, it’s high time to support them to obtain those skills through training and deliberate practice. Also, consider strategic hiring of additional engineers, either temporary or permanent.

Automate test execution from day one

Tests are only valuable when they are executed, because that’s when they inform development teams about the current state of that product and any potential problems with it.

It’s generally a good idea to make the process of executing tests and reporting on the results thereof an automated process, too, for example using build automation tools and Continuous Integration (CI) platforms such as Azure DevOps or GitHub Actions.

The sooner the process of executing your automated tests is automated, the easier this is. If you only have a few tests, they will not take up a lot of time yet, and the code base is small enough to change to allow for automated execution on demand. If you wait too long with automating the execution of your tests, on the other hand, there is a significant risk that it will take a lot of time and effort to achieve, and maybe even require rewriting (part of) your tests to allow for automated execution in the first place.

Strategically select tools that align with scale and business goals

There is a lot of conversation happening online about what is ‘the best’ tool. Often, these discussions quickly degrade towards people advocating for their personal favourite, rather than discussing differences between certain tools in an objective manner. Personally, I’d rather steer clear of these discussions altogether, and I recommend the teams I work with to do the same.

In my opinion, the most important question to answer during tool selection is

“Which tool or set of tools is the best fit for our specific context?”

There are a number of factors that determine whether a tool is fit for context, including (but not limited to):

• What does the product technology stack look like? If your product is built in Java, for example it makes sense to choose test tools and libraries written in Java, too.
• What kind of tests are we looking to write? Which tool or collection of tools offer(s) the best fit? Keep in mind that some tools claiming to be able to ‘do it all’ tend to support many types of testing only in a perfunctory way.
• Do we prefer to write tests in code or are we looking to use a low-code solution? The answer to this question lies not only in the current skill set of your team, but also in the flexibility that a tool provides to build and maintain test suites at scale. Most tool vendor demos do not answer that question for you.
• Do we prefer open source solutions or commercially licensed tools? Here, too, there are pros and cons to both. With open source tools, you need to make sure the tool is secure and well-maintained, and that there is plenty of community support in case you need it. With commercially licensed tools, support is often guaranteed and included in the license fee, but updates to the tool and customization might be more challenging.

There are more factors that come into play when it comes to tool selection, but the ones mentioned above tend to come up with every team I talk to and have worked with over the last 20 years.

Smart use of artificial intelligence (AI)

These days there is no avoiding discussing the use of AI to enhance and support software testing and software development efforts, and the topic of test automation is no exception.

Much has been said about AI already, both by its advocates and by its critics, and the general consensus, and a view that I subscribe to, is that AI can be a very powerful addition to your test automation efforts, if it is used with prudence, yet it can be outright dangerous if it’s not used carefully.

Here are some examples of what for me constitutes prudent use of AI in test automation:

• Generating suggestions for boilerplate pieces of code for a test automation framework
• Analyzing test failures and providing suggestions for possible root causes and solutions
• Suggesting test code refactorings and optimizations

In short, I recommend seeing AI as a team member that is very good and lightning fast at making suggestions on how to improve your automated tests, but that does not have the ability to make critical decisions by itself. I recommend keeping a human in the loop to scrutinize the suggestions made by AI before adoption.

That means that I do not recommend the following types of AI use:

1. Blindly accepting suggestions made by AI, be it for additional test cases, code refactorings or otherwise
2. Using so-called ‘self-healing’ capabilities of AI without a human in the loop

When used with prudence, AI can be an excellent accelerator of your team’s test automation efforts. Without the oversight of skilled human beings, however, there is a significant risk that you’ll quickly find yourself with a test suite that might look good, but that no human understands or trusts.

⇒ Bottom line

Here’s what leadership should do to increase the probability of test automation success:

1. Ensure testability is discussed and included in the design of your product
2. Make test automation a team effort and responsibility, not a role-bound task
3. Automate execution of tests as soon as possible
4. Select the tool(-set) that fits your long-term testing and development goals
5. Be prudent when using AI to support your test automation efforts

Conclusion and key takeaways

In this report, we have seen what it takes to be successful with test automation, and we established that doing so requires much more than selecting the right tool for the job – teams need a holistic test automation strategy tailored to their specific context if they want their test automation to deliver valuable feedback, fast.

• Testability of your product is essential for test automation success, but this is not something that happens overnight, nor is testability something that is easily reverse engineered into your product.
• The probability of test automation success is highest when automation is embedded in the overall software development process and when it is a whole team activity.
• Make sure that the team has the skills required to succeed with test automation. This includes both software testing and software development skills.
• To provide maximum results, make sure that the tool(s) used for test automation fit the context and align with the goals set for software testing.
• Artificial intelligence (AI) can be a valuable accelerator for your test automation strategy, but only when used with prudence and when put in the hands of skilled engineers.