Your transformation toolkit

Advance your enterprise testing strategy with our transformation toolkit.

Learn more

Software testing

What’s required for good software testing?

Editor’s note: Accelerate 2019—the can’t miss Continuous Testing conference— is coming to San Francisco next month. To give you a taste of what’s in store, we’ll be sharing some of attendees’ favorite moments from Accelerates past.

Ingo Philipp’s amazing lightning talk is the perfect place to start.

Ladies and gentlemen, this is exactly how I see software development. And this is not just an analogy. This is exactly how I see it. To me, software development is a lot like wrestling in the mud with a pig. And when you do that, you start realizing quite quickly that the pig just loves it. And to make it clear, the pig is the software here. The pig is not the developers out there… sometimes at least.

Now, the question is what the heck is software testing? Well, software testing is all about washing that pig. That can be messy because it usually has no rules, no clear beginning, no clear middle, and no clear end. And sometimes honestly, it’s kind of a pain in the ass because when you’re done, you’re not even sure if the pig is really clean or why you were even washing the pig in the first place.

Now guys, let’s see how we wash the pig. Let’s see how we test software. And now I need your imagination. Imagine that this rectangle represents your software product. It represents the entire functional and non-functional spectrum of your software. That’s all there is. That’s all you could possibly know about your software. Now the green rectangle represents all that you currently know about your software product. And this rectangle is apparently smaller than the previous one because I’m at least convinced that you will never be able to always know everything down to the very last detail about your software.

Now the red rectangle represents all that you repeatedly check. For example, for test automation in your software. And again, this rectangle is smaller than the previous one because I’m convinced that you will never be able to check everything at any time in your software. And here the question is, well, would it even make sense to do that? Would it even make sense to aim to codify, to automate all your knowledge? Well, most people would say yes, that’s the ultimate goal. But I’m saying no because if you would solely aim to codify all your knowledge, then how on earth would you be able to create new knowledge? And you have to create new knowledge about your software because your software constantly grows. You need to constantly increase the knowledge about your software to reveal problems that are far, far outside your checking regime, guys.

This is exactly what testing is about. Testing is all about closing this knowledge gap. It’s all about closing the gap between what we know and what we don’t know about our software. So the goal of testing is information, not automation, guys. And the reason for this is simple. A test is just a question you ask your software. The point of running that test is to gain information. And so testing is, always has been, and always will be a search for information. But don’t get me wrong. The information sought often requires automation, but not always.

Here’s the proof, guys. Now imagine that these dots represent all the test cases in your regression test portfolio. And now imagine that all these test cases pass, and the last regression tests run before you decide to ship your product into production. And then you might conclude, well, that looks good. Ship it. Ship our software products. And that is what you then most probably will do. But just a second later, you will see something like this. You will see that repetitive test case execution is by far not enough to see that iceberg coming, guys. So the question that arises now is How do we avoid scenarios like this?

What does it take to do good testing? Well guys, good testing takes the ability to post useful questions, to observe what’s going on, to describe what you perceive, to understand what you observe. Good testing takes the ability to interpret what you find, to draw the right conclusions fast, to think critically about what you know, to know that you will never know everything. To keep thinking despite already knowing, to focus on things you don’t know. To target specific issues without losing focus. Good testing takes the ability to recognize and manage bias, to form and test conjectures, to jump on those conjectures, not just on conclusions, guys. Good testing takes the ability to reflect on your decisions, to consider alternatives, to scrutinize illusions you’re holding true. It takes the ability to analyze someone else’s thinking, to reason about cause and effect. And good testing, guys, takes the most important ability, the ability to learn how to learn.

This is what makes a good tester hard to fool, guys. And this is what makes a tester cautious, curious, and critical. And this is what makes good testing, guys. So I can safely conclude that the good testers out there are simply the Peter Pans of the human race because they never lose the holy curiosity to find anything that might threaten the value of our software products. Thank you so much, guys.