When most people think of Agile testing, performance testers don’t usually come to mind. Despite the trend toward cross-functional teams, performance testers tend to remain siloed. Yet, you could make the case that exploratory testing–a key part of most Agile testing processes–was actually pioneered by performance testers.
Much like exploratory testing, each performance tuning iteration starts with a defined question and an explanatory hypothesis that can be tested in a reproducible manner. Also like exploratory testing, it involves using the scientific method to correct and integrate previous knowledge via an iterative process. This process is essential to helping the tester measure, evaluate, and improve system performance.
What can “regular” testers learn from performance testers’ interactive tuning approach? Our recent webinar with Ingo Philipp and Tim Koopmans, Optimizing Agile Testing: What Functional Testers Can Learn from Performance Testers, begins to answer this question. Ingo (a functional tester) shares what he has learned having watched Tim (a performance tester) in action.
In his overview of exploratory testing, Ingo began by stressing that good testers are comfortable in the grey areas, because you will never know if you have tested every possible circumstance. There are time, resource, and budget constraints that mean that testers will only ever be able to check a subset of what can be known about any given piece of software.
Testing can broadly be divided into confirmatory and exploratory testing.
- In confirmatory testing, testers want to achieve depth of knowledge. Confirmatory testing is done mechanically, with pre-defined data being put through pre-designed steps to confirm what is already known. This makes confirmatory tests great for detecting change, and well suited to automation.
- In exploratory testing, testers are trying to learn more about the product and expand the breadth of their knowledge by designing, executing and interpreting tests over multiple repetitions. As they learn new information, exploratory testers can detect potential risks and problems, something which requires a touch of human creativity.
To achieve high levels of business risk coverage through testing, both approaches must be used together.
Taking this equation, it is clear that the quality of testing is not helped by increasing the number of test cases in play. Better quality needs better test ideas and test design, which needs a tester with a tirelessly curious mindset.
To demonstrate, Tim walked us through the approaches he uses in his work, saying that “exploratory testing is like exploring the grey boundaries of your knowledge of the system under test”. He broke testing objectives down into the quantifiable (how likely is this going to happen in production?) and qualifiable (what are the consequences if this risk becomes an issue), and stressed that only by considering both the probability and impact can the different outcomes of risk be understood.
Tim defines load testing as simulating demand on a system and measuring it for performance, but load testing should also look for qualities beyond performance. Performance can easily be measured in confirmatory testing, which can lead to an overemphasis on performance metrics and dashboards crammed with numbers. Exploratory load testing asks ‘what if’ questions which allows testers to exercise their information-finding skills and unearth other areas of concern like reliability, scalability and availability.
To hear more and find out why testing is like mapping a route on Google Maps, wrestling a million pigs, or washing a hippo, watch the rest of Optimizing Agile Testing: What Functional Testers Can Learn from Performance Testers, available on-demand now.