Ever since the rise of Agile testing, there have been people saying that “manual testing is dead”. This is true – sort of. It would be more accurate to say that “manual testing has changed”. With test automation taking over the mundane checks that eat up tester’s valuable time, testers are becoming more and more free to exercise their considerable expertise to more valuable types of manual testing. The one that tops the list is exploratory testing. In this talk, Ingo Philipp addresses how to apply exploratory testing to Agile in a way that is measurable, scalable, and effective.
Here’s the full transcript
This session, as already mentioned, will be all about exploratory testing, and in order to safely dive into that topic let me start with a short story. A story about myself. In August this year I enjoyed four weeks of holidays, and guys I tell you – it was just great.
Now, whenever I go on holidays I find a theme. That means a topic, or a set of topics I’ll focus on during my holidays. For this year’s holidays I selected the theme “creative thinking”. Now, the reason why I came up with the topic was because I’ve read quite a lot about this guy here in the last couple of months: his name is Erwin Schrodinger, and he’s one of the founding fathers of quantum mechanics. By the way, he’s the man behind Schrodinger’s Cat.
Now, in one of his numerous letters he wrote to his friend, the genius Albert Einstein, he said “the task is not so much to see what no one has yet seen, but to think what no one has yet thought about”. This statement immediately blew my mind. I just thought, “How can it be? How can I make myself able to think about something nobody has ever thought about?” I did what everybody would have done. I asked Google. I searched for “Think Different”, which was obviously a bad choice because all the search results I received were all about Apple.
I had to change directions. I went back to Google, and I searched for “Think from different perspectives”. In the very first search results, Google suggested this book to me. It’s called Lateral Thinking, written by the thinking guru and author Edward De Bono. In this book, Edward De Bono makes a compelling statement. He says, “You cannot dig a hole in a different place by just digging the same hole deeper.” Now, what the heck does that even mean?
Well, imagine there is some problem floating around, and that problem approaches you. You need to solve the problem. You are asked to solve the problem. What do you do? You apply all the problems solving techniques you have learned so far from your friends, from your parents, and from your colleagues to get the problem solved. But now, imagine none of these problem solving techniques actually works. Then it doesn’t really matter if you apply all those techniques over, and over, and over again because none of these techniques will solve the problem.
This then implies that you have to solve the problem from a different viewpoint. You have to attack the problem from a different perspective, and that is what lateral thinking is about. It’s a collection of different problem solving techniques to solve problems in more creative ways. Now, to be more concrete let’s do a simple example. I hope everyone knows this guy here. It’s Mr. Roger Federer, and in this picture he is at Wimbledon. Now, imagine there are 128 players playing a singles tournament in Wimbledon on an elimination basis. That means a traditional knock out system.
Given that set up, the question is: how many matches must be played to determine the winner? Guys, I’ll bet everything I have that every one of you, including me, would have solved the problem in the following way. You would say, “Well, in round number one there are 128 players. I have to play 64 matches.” In the next round 64 players will remain, and so I have to play 32 matches, and so on, and so forth until you reach the final, which is two players fight for the title, and play for the Wimbledon title.
If you then add up all those matches you will see that you will have to play 127 matches to determine the winner. Now, that way of solving the problem, or that way of even thinking about the problem is called vertical thinking. The solution is derived by pure logical deduction. Now, the big question is, is there any other way? Is there a more efficient way to get the problem solved? Indeed, there is, and that is exactly where lateral thinking gets into the game.
Now, one technique in lateral thinking is called factorization where you break the problem down into smaller chunks. For example, you focus on the tournament, the knock out system, or the players, and you can break those chunks down further – for example the players, break down to the losers and the winners. Now, we’ve already focused on the winners, so why not choose focusing on the losers? What we know is that one player wins the tournament, so there has to be 127 players who must lose.
Each loser just loses once, and for that reason 127 matches must be played. That’s a completely different way to solve the problem. It’s a creative way to solve the problem. Please remember, creativity is nothing more than a capacity to challenge the existing order of things. It’s just a capacity to break out of your established thinking patterns, and that is exactly what exploratory testing is about. That is exactly how exploratory testing distinguishes itself from any formal testing techniques – especially when we do test case design. They’re requirements.
Test cases are derived in a methodical and structured way by pure logical deduction. If the way we think when we do exploratory testing isn’t any different to the way we think when we do formal testing, what’s the use? We see it day in and day out, that it’s the way of thinking that differentiates these two testing disciplines. There’s so many different types of issues to find, and some of those issues will be found using exploratory testing techniques, and some by using formal testing techniques. But you will never find all of them by using only one technique or another.
The bottom line here is that exploratory testing and formal testing (including automation) are just tools. With just one of these methods you can only do so much, but by using them together you can discover a lot more.
We regard something as properly tested when it is being checked by efficient formal test automation, based on solid test case design, after it has been explored by the rich intellect of human beings. This is our age-old “equation of motion” and “definition of done” – the reason why we pay so much attention to these testing disciplines in all of our projects and product development.
Now, the good news is that lateral thinking, and therefore exploratory testing, is not a talent. It’s a skill that can be learned. Let’s see how we translate the approach of exploratory testing into practice and translate exploratory testing into reality.
Our technique has five components.
The first one is session based testing. Session based testing takes exploratory testing and puts it into a straight jacket. It puts all the imagination, which comes along with exploratory testing into those constraints. By imagination, I just mean the ability to form new testing ideas. Now, you might say, “Well, that’s bad.” Right? That really depends on how tightly you tie up that straight jacket. We do that in order to make exploratory testing structured.
Why do we need to structure exploratory testing? So that it can scale for larger implementations. The challenge is to scale exploratory testing for 30, 40, or even 50 teams – or even more – in a truly agile enterprise. You need to have some structure at hand to do that properly. Only if you have structure can you perform exploratory testing consistently, which allows us to measure its success at the end of the day.
The core object of session based testing is the session, and the session is a chartered, reviewable, and uninterrupted unit of testing effort. This testing effort usually is time boxed – let’s say between 30 and 90 minutes or so. We time box the session in order to curb our perfectionist tendencies during testing. It’s to avoid over-committing to a testing task, which usually happens when there is no deadline. The core object of the session itself is called the session charter. The charter is just a mission statement. It precisely and concisely describes what you are looking for in your test session.
Now the big question is, how do we find the scope of such a session? How do we mark the territory of a session? How do we set its boundaries? In order to do that we apply a traditional requirements-based approach, and I know – even mentioning the term requirements in an agile world at a continuous testing company, is bad right? It’s so bad. But for me, a requirement is just a service we provide to our end users through our products so that these end users can achieve their goal. I don’t care if you express those requirements as a system use case, a business use case, a feature, a user story, an app, or a theme, whatever you like. Call it whatever you like. It still will remain a service, and we make use of these services to limit the scope.
We limit the scope of our tests in order to make it manageable (especially for unskilled testers) because the sheer function and non-function spectrum of our systems is often so attractive that you get so easily lost in details. Stay within the scope.
Now, we have seen how to time box the session and how we limit the scope, but now the big question is: what should we do within that specific amount of time? What are our goals? What are we about to find there? In order to set concrete goals we apply an approach, which is known as tour based testing. A testing tour is an exploration around a theme.
There are many different schools of tour testing thought out there. The one I like most is proposed by James Whitaker in his book “Exploratory Software Testing”. In this book, he describes different types of tours ranging from the the supermodel to the museum, to the couch potato, to the money, and so on and so forth. These tours can be used to to provide a clear focus to your session, helping you to set a realistic goal.
Let me give you the reason why this is important. Let’s look at Google Maps for example. Imagine you want to go from Oslo to Stockholm. A few years back, this is what Google Maps suggested as a route. How do we find these sort of bugs? I mean you can get lucky and find these sort of bugs, but who wants to be lucky in this career?
Maybe in manual testing there’s some random neuron in your brain firing, and giving you the insights to find these sort of bugs. But we can’t just hire genius testers with Jedi instincts, and not everybody has the instincts to find these sort of bugs. So how do we do it? First of all, it takes understanding your application from a business and technical perspective. In this particular case, it takes the understanding that there can be black spots, or dark spots in the mapping data. There can be areas where the mapping data is just incomplete or where old mapping data is connected to new mapping data, etc.
It takes preparation to gain that knowledge. But if you use that knowledge, you will find these sorts of bugs almost instantly. But you won’t find those issues if you focus on 10 different things at the same time – you’ve got to focus on very specific issues, which is what tour-based testing helps you accomplish.
All in all, I think we need to agree upon the fact that testing is blind without the mind, because there is no brain in the loop (per se) when it comes to formal testing. That’s the reason why we have to do exploratory testing. We do all that in order to maximize the quality of our software products, or to be more precise, to maximize quality at an ever increasing speed. So then, how do we measure quality?
Gerry Weinberg concluded that quality is just “some value to some person”. This implies that quality is inherently subjective, and the result is that different stakeholders will perceive one and the same product as having different levels of qualities. What’s the implication for testing? Well, this all means that we, as testers, must look for different things for different stakeholders, which essentially means we must diversify our testing. This leads us to element number four of our exploratory testing technique, which is polychrome testing.
Polychrome just means doing art or work in different colors, and that is exactly what we do when it comes to exploratory testing. We explore the product from different viewpoints to maximize the diversity in our approach to testing. In order to do that, we apply a concept, which is known as the six thinking hats. It’s a concept from Edward De Bono. This concept has been used for more than 20 years already to solve business issues in general. The core components of the concept are the “six thinking hats”. They’re literally hats. Each hat as displayed here has a certain color associated with it. Each color stands for a certain viewpoint, a certain type of thinking.
When we run an exploratory testing session, we invite multiple testers, and each tester takes a different hat in order to diversify our approach to testing. We don’t just go for critical thinking, we also go for creative thinking, positive thinking, emotional thinking, and effectual thinking in order to maximize diversity. Last but not least, we also apply what we call a scenario based approach, and this simply means that each and every test idea is documented, and captured, and put into a exploratory scenario. Why do we do that? Well, in order to make it reviewable at the end of the day. Why do we want that? In order to measure its success, and to define the next test properly.
Now, having all that said let me conclude with one single statement. If someone were to ask me what exploratory testing really means in one single sentence, I would say that exploratory testing is, “any testing that machines can’t do yet – because exploratory testing is not so much a thing you do, it’s more a way you think”. Thank you so much.