Agile testing

How exploratory testing and formal testing fit into the agile equation

What is exploratory testing? Defining exploratory testing on its own is a tough job. Exploratory testing is easier to understand if you compare it to formal testing.

What is Formal Testing?

What is formal testing? Formal testing is “testing as artifact creation.” It’s a very mechanical approach to software testing – especially when it comes to test automation. Automation is simply processing a certain amount of pre-defined test steps in a particular sequence. Executing an automated test does not require mental engagement from the tester.

What is Exploratory Testing?

On the other hand, exploratory testing is “testing as performance.” It is a more intelligent approach to testing. Exploratory testing is all about learning the product, designing the tests, executing tests, and interpreting the test results all at the same time. You do all of these things simultaneously, which means that in exploratory testing, your next test is always influenced by the results of your last one.

Exploratory Testing vs Formal Testing

Formal testing, in contrast, is designed to monitor known risks. Automated tests help you to confirm what you already know, whereas exploratory testing helps you to analyze potential risks. The purpose of exploratory testing is to focus on what you don’t already know – to question your preconceptions and assumptions.

This implies that any exploratory test has a high informational value associated to it, because you learn something new about the application. This is not the case with formal testing. Formal testing has a low information value since its objective is to confirm what you already know about the application.

Formal testing does have the advantage of offering high risk coverage. By repeatedly monitoring known risks, you decrease the chance of those risks becoming active bugs in your system without anyone noticing. Exploratory testing, on the other hand, offers low risk coverage. Because each exploratory test is unique and informed by the results of the previous test, it cannot provide the type of high risk coverage that formal testing delivers.

The bottom line is that formal tests are a bit like change detectors, whereas exploratory tests are more like problem detectors. As James Bach once stated, exploratory testing is all about “Testing is an infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous.”

The Agile Testing Equation

At the end of the day, both exploratory testing and formal testing are about spotting problems. The purpose of testing any program is to find problems in it. Individually, both exploratory testing and formal testing approaches have their weaknesses. However, if you apply them in tandem as different perspectives towards solving the same problem, you will find that formal testing is strong where exploratory testing is weak, and vice versa. This is the reason I regard something as properly tested when it has fulfilled the Agile Equation:

Checked (efficient formal testing) + Explored (effective exploratory testing) = Tested (Productive testing)

When a system has been checked using efficient test design and explored by the richness of the human intellect, the end result is productive and effective software testing.

[Learn more about Exploratory Testing and the Agile Equation in Ingo Philipp’s on-demand webinar]