The Myth of Automated Exploratory Testing

Software Testing 101

Get started with Exploratory Testing

Free Trial

As more and more companies move their software development and testing efforts into the realm of DevOps and Agile methodologies, the question continually arises: "How much testing can we automate?". 

It's a very good question, and one that doesn't have a one-size-fits-all answer. The amount of testing you can automate varies depending on your testing environments, business needs, and requirements. After having worked with thousands of companies over the years however, we have averaged it out into the following illustration: 

Using the right tools and methodologies, we have seen companies grow their automated testing from roughly 20-30% to 85% or higher. The 70-80% of manual testing they were doing before shrinks down to a very manageable 15% or less, with an important caveat: that manual testing turns into exploratory testing. 

In the hunt for extreme automation, some people make the mistake of thinking that you can automate exploratory testing. This is a myth. 

Exploratory testing is the process of learning the product, designing test cases, executing test cases and interpreting the test results at the same time. It is a method that requires the tester to think critically through the entire process, making decisions based off factors such as the test scope, charter, and goals. 

This means that exploratory testing, by definition, cannot be automated. 

That being said, there are aspects of exploratory testing that can be automated. One of the most important (and time consuming) tasks of exploratory testing is documentation. The tester is required to document each of their test actions, along with it's outcome and additional notes, so that any defects discovered by can easily reproduced by the development team.

Completing this process manually is easily more time consuming than conducting the test itself. There are a number of exploratory testing tools on the market that help to automate the documentation process by taking annotated screenshots with each test step, enabling audio and video recording, and more. We've put together a feature comparison on some of the top Exploratory Testing Tools on the market - check it out here

While exploratory testing itself cannot be automated, it does provide a perfect complement to automated testing.

Automated testing is best used for checking. Once you design and automate your test, that test will only be capable of giving you a Pass or Fail result. Outside of Pass or Fail, it cannot uncover anything new for you. In that sense, a single automated test covers a relatively narrow scope of risk. That is why you build out entire automated test case portfolios, where each test targets a specific aspect of your business risk. 

Exploratory testing is able to go beyond the confines of automated tests. As illustrated in the graph above, automated tests can perform the majority of the "heavy lifting", freeing up your testers to apply their considerable expertise and business knowledge to discovering new defects, testing usability, and exploring the program in a way that is deeper and more insightful than what an automated test can accomplish. 

When a tester uncovers a new defect in the course of their exploratory testing, it is very possible that their test actions can be automated for future checks. That way, you can ensure that the defect does not return in future builds, while the tester is free to continue exploring the program for new discoveries.