How to Measure Exploratory Testing

Software Testing 101

How to Measure Exploratory Testing

When you’re measuring exploratory testing, consistency is the key. Don’t expect perfect results immediately. It’s a journey, especially when the concept of exploratory testing is new to you. 

It’s like going to the gym. You go to the gym to get in shape, but it has no effect if you go the gym once a week for 12 hours at a time. You need to work out every single day for about 20 minutes to succeed. It’s the same with exploratory testing. Long-term consistency beats short-term intensity. 

What metrics should you measure? 

Production Defects

Monitor your production defects. Monitor how the number of defects in production evolves over time. Does it decrease when you perform exploratory testing (given the fact that your formal testing approach remains the same)? If not, it might be due to the fact that you didn’t find the right balance between the effort you expend on exploratory testing and formal testing. So adjust it and monitor again. Or maybe you are using the wrong technique, or are using the technique wrong to translate exploratory testing into practice. Again, adjust it, and monitor again how the number of production defects respond.

Defect Severity

Another measure of success is defect severity. This gives you a measure of how well your testers understand the product. You don’t care if your testers find gazillions of cosmetic defects, you want them to detect the ones that really matter – the ones stopping your customers from achieving their goals. The critical defects! In most cases these defects are the most obvious defects, but are not necessarily obvious to the testers. If you see that happen, reserve more time for learning. Give your testers the chance to talk to developers, product owners, or even customers. Give them the chance to improve their skills, and keep your patience. In the best cases, your testers are also using the product for their private purposes. The goal is for your testers to also become stakeholders.

Defect Variety

Another measure is defect variety. Do you find just functional issues during the testing phase, but in production a lot of usability issues are filed? This would imply that you didn’t really manage to diversify exploratory testing well enough. So, diversify and change direction.

Formal Testing Leak

You could also measure what we call the formal testing leak. This quantity measures how many defects you found that would have been missed by formal testing (e.g. test automation, manual testing). One way to measure that is to apply some kind of A-B testing approach to testing itself. One team (A) just focuses on formal testing, and the other team (B) only on exploratory testing (at least for a limited amount of time, e.g. two sprints).

Mean Time to Feedback

In addition, you could also measure the mean-time-to-feedback. How fast (e.g. minutes, hours, days) did it take you to provide feedback to your development? I am not saying that this is easy to measure, but it’s worth measuring. Maybe you could ask your developers to support you in monitoring this quantity.

Learn More – Exploratory Testing Webinar

It’s clear what Agile requires from developers—but what about testers? How do you ensure that testing keeps pace with rapid, iterative development?

Watch the Transform Your Agile Processes with a New Approach to Exploratory Testing webinar to discover how exploratory testing removes testing bottlenecks for agile teams – and how to get started.
You’ll learn how to:

  • Introduce exploratory testing into agile sprints and continuous delivery processes
  • Apply exploratory testing in a way that complements and accelerates your existing testing strategy
  • Choose among exploratory testing strategies (session-based, tour-based, polychrome, etc.)
  • Plan and coordinate exploratory testing involving a variety of team member perspectives

What Is Exploratory Testing

What is exploratory testing? What are the challenges of exploratory testing? What are the benefits of exploratory testing?” Read more.

Transform Your Agile Processes With A New Approach To Exploratory Testing

discover how exploratory testing removes testing bottlenecks for agile teams – and how to get started. Read more.