Agile testing

“There is No Hiding in Exploratory Testing”: Zeger Van Hese on Exploratory Testing in Agile Environments

Exploratory testing used to be a “black sheep” in the testing world. Used to. The wider software community is finally catching on to what the James Bachs and Michael Boltons of the world have been saying for a long time: Exploratory testing is a crucial and, often, under-utilized tool in your testing toolkit. Some would even argue that Exploratory Testing is at the heart of all software testing.

It says something about Zeger Van Hese that he built his reputation and freelance testing brand around exploratory testing in Agile environments a good while before it became a buzzword.  I sat down with Zeger to ask him to explain to me his take on how exploratory testing fits into Agile and continuous testing. Make sure to follow Zeger on Twitter or on his website, TestSideStory, to track his writing and upcoming speaking engagements.


Chelsea: What is your specialty?

Zeger: I am a freelance tester in Belgium, and I have my own company that I have been working at for four and a half years. I specialize in exploratory testing. I also specialize in testing in an Agile environment. Specifically, anything that requires quick feedback from testing.

Chelsea: So, helping to enable your clients to be functioning within the DevOps cycle?

Zeger: We try to make it faster. We try to speed on work together as a team with lots of collaboration, and not having our testing done in separate departments which I have done in the past, so now I can compare. I prefer working as a team much better. It feels much more alive, and you make more of an impact. You can suggest things and solve problems which makes us much more involved. Testing in different departments take so much longer than testing within one department. You have to give feedback and receive feedback. It takes a very long time and I don’t like working that way.

Chelsea: Isn’t that one of the challenges with exploratory testing? With exploratory testing, you have the challenge of needing to record and report what you have done, and that can get quite laborious as well.

Zeger: That is one of the bigger challenges in exploratory testing. It is hard to trace what you have done, so you have to get quite good at recording what you do. This involves taking notes while you test, but doing it in a light weight way so it doesn’t get in the way of your testing. If you are constantly taking notes then you are context switching the whole time, and it becomes very hard. There are several tools I use that I found over the years that I believe are best for that kind of work. Note taking tools, such as Rapid Reporter, are what I use. It is very simple it looks like a toolbar and has a text field where you can note things down and it saves them to a CSV file and timestamps it. You can take screenshots with it, and it stays on top of all other applications so it’s very easy. Because of the CSV format you can do many things with it such as export it, make reports out of it, or send it out in e-mails. Over the years, you look for ways that work best for you. Other people probably have other things that they use that work better for them, but for me this is what I like to use most. Combined with Rapid Reporter, I use Xmind to prepare my sessions. First I get an idea of what kinds of things I want to test in a product, and I list my test ideas. You could call it a plan if you like. Once I find the areas to test I let the ideas guide me in my testing, at the same time taking notes.

Chelsea: Do you train other testers in the same technique as well?

Zeger: I train people in exploratory testing and in testing in general. This is one of the things I show them, but I’m not telling them that this is the only way to do it. I tell them that it is just the technique that happens to work for me. I like teaching and showing people, because I have done several types of testing over the years and I really have a passion of this. It is easy to teach something if you are passionate about it. When you’re not interested in something, it is very hard to teach and to bring across your points. If I have to invest my time in it, it better be something worthwhile. You don’t always have the choice, but if you have your own company or you are working for yourself, it is easier to focus on that kind of stuff. When you’re working for someone, your employer can ask you to do tasks that you may not have an interest in. So you get it over with and make it work, but you’re not really invested in what happens later on.

Chelsea: That seems to be a main tenant in exploratory testing. It has to be something that you are passionate about, because if your mind isn’t engaged than you are not going to be very effective in exploratory testing, are you?

Zeger: That’s true. You can’t hide when performing exploratory testing. For some people, it’s really daunting, because scripts that you follow act as a kind of safety net. If you’re new in the field or don’t have much confidence you feel comfortable following the steps, and you know that you are doing something the way that other people have taught you.  But if you only follow what is written down, you risk not noticing all the other things that are around or surrounding the script. That is one reason why exploratory testing is more effective. You open your mind up for other things.

Chelsea: It is fundamentally about identifying new risks that have not yet been found before. Regression testing and automated testing is only for existing risk.

Zeger: Yes. Regression testing and confirmatory testing help to confirm that things are still working the way they are supposed to. Exploratory testing is about finding new information and risks. It is not focused on confirming something that worked 2 years ago, that it is still doing the same thing. It is something that you have to feel comfortable doing, but by doing it you are able to develop your skills more, which in turn makes you more comfortable. You have to trust yourself enough to say, “OK, whatever I think is important might be important enough to test.” You write these things down and trust yourself. Often testers don’t get to the call to say something is a bug. We can indicate strange behaviors and that’s it. What might be an important bug or feature as a tester may not be important to a product owner, and it’s their call. They decide if we should invest time in fixing it. You have to get used to not having the final say in those situations. We also don’t have the final say about whether or not something should be released. We can give information and advice about areas within the product that are shaky, but the shots are not called by us normally. In the traditional testing way, testers were kind of the gate keeper where if a tester said “Don’t release” then the company wouldn’t. It is a big responsibility, and it’s what managers get paid for. If they are responsible managers, they have to be able to make those kinds of calls.

Chelsea: With these Agile teams that you are working with and training, do you find that they take to exploratory testing easily or is it something that you have to force them to integrate into their testing?

Zeger: It is coming quite natural for them. You realize that it’s the only really good time efficient way to test, because you don’t have time to write all the scripts up front and continue checking and maintaining them. Everything goes so fast, so having fast feedback from testing is something that these companies need. It might be new to them, but they see that it is a more effective way to test. It is still something that you have to get used to, but the advantages of exploratory testing are very clear.

Chelsea: I’ve heard a lot of debate about where testing fits into the DevOps cycle. In your opinion, where would you place testing?

Zeger: I think that testing is everywhere, in the whole loop. Testing should be involved in all the phases of the DevOps cycle, and not just squeezed in between the development and operations. That is typically where testing is, but we should also be involved and sitting in with the developers while they are writing and doing their unit tests, also helping them write and thinking of different scenarios. The role of a tester is slightly changing in those types of environments. We have to find our new reason of existence there, because the word DevOps kind of says it all: where is the testing? It’s not called DevTestOps, though it probably should be. In some environments, when a problem in product comes up you can fix it quickly with the DevOps people, but in other environments you cannot afford doing that. They are too critical to find just by monitoring in production which could costs lots of money or potentially hurt people.

Chelsea: Within DevOps we talk a lot about continuous integration and continuous delivery, but I guess what they should also be talking about is continuous testing.

Zeger: That is a phrase that I have heard some people also coin. It’s a good thing, because continuous testing fits in with continuous integration. It is testing on different levels and in different ways, like automated testing which is testing, but it is only one side. There is also a place for thoughtful manual testing, or the mind of a thoughtful manual tester. Not for testing manually, but assisting with automated testing.  Also, looking at stuff that is ready and quickly assessing. Continuous testing is definitely one thing that should be in there.

Chelsea: Do you think the idea of continuous testing is starting to become a trend?

Zeger: Yes, and DevOps is picking up as well. My last client was also in a DevOps environment. It’s fun to see all these new things pick up. You have to adapt and learn, but that’s a good thing because it forces you to be on your toes. What I am realizing now that I’m looking for a gig since my last assignment just ended this week, is that all of these companies are asking for a super tester that is good in everything. They seem to want testers that have 10 years of automation experience, know all test methodologies, and at the same time know rapid software testing, DevOps and have all possible certifications. I would not describe myself as a technical tester, although I have done quite a bit of technical work in the past. I have done some automation. I can read and understand code, I’ve programmed code but I have the impression that companies are looking for technical super profiles for their testing now. I’m in the process of learning more so I can add new skills to my portfolio. When you look at developers, most of them are either backend or fronted, or only know one language, but they expect the testers they are recruiting to know all of the languages. I’m honest when they ask me about my skills. I don’t pretend I know it all.

Chelsea: Agile and DevOps has driven a need for testing automation. For many companies that have adopted that need for automation, is there a lack of skill base?

Zeger: Yes, leave the automation to the developers, because that’s what they do best. If they have real testing profiles they can help with automation, but use testing people to help with other things other than automation. In some contexts, it can work but its only half the picture. I like to hear about failure stories to understand how what problems they encountered and how they solved them. The success stories that you often hear sound too polished, and they leave out the struggles that they went through. There are always struggles, so stories that sound too good to be true usually are. Something else that is popping up constantly is the Spotify model. The Spotify model is something that was implemented by Spotify with trial and error, and after a couple of years it evolved to something that worked for them. And now people want to copy exactly what worked for Spotify into their environment and they don’t realize that it is a totally different business with different needs. They adopt it and expect it to work as well for them as it does for Spotify.

Chelsea: It’s just like that with manual and exploratory testing. Everyone wants to have a formula that makes them feel safe. If someone can take the formula that Spotify has been successful with and implement it, they believe it will also be successful for them.

Zeger: It looks like a magic formula! What you hear at conferences is companies telling you that they are working a certain way which is very successful for them. What you don’t hear is the years of struggling that they had to go through to find the right way that works for their company. I’ve noticed that there is a lot of demand for Agile teams and approaches. I don’t label myself as an Agile coach even though I have worked in an Agile environment for many years as a tester. Right now, they are looking for Agile coaches to help people move to the Spotify model.

Chelsea: I didn’t realize that the Spotify model was something that a lot of companies are moving towards.

Zeger: In Europe, it is happening a little bit more, I think mainly because it is a Swedish company. In Europe at conferences, it is kind of a big deal. Big banks are even trying to adopt it, and big banks are usually very slow at keeping up with that stuff. I worked with healthcare companies for a while and I don’t see them adopting the Agile Spotify model. It’s different from a music streaming app; there is patient safety at stake, insurance, and regulations. You want to adopt a new model if you are looking for a problem to solve. Doing it because it is new and trendy is no reason to adopt a different model. As a consultant, people ask for a solution and have the solution already figured out. You have to start first with “What is the problem you want to solve?” and you go from there.

Chelsea: Often, you have to do some work to identify what the problem actually is. Sometime, their problem isn’t what they think it is

Zeger: That’s one of the cool things of working with people. They ask me to do something, and I tell them to hold off a while. Let’s look at what the problem is and try to find the best way to get the solution. I have learned a lot of lessons from Jerry Weinburg. He has written lots of books, and my favorite one is Secrets of Consulting. In the book, he talks about when people call you in to solve their problem, within 5 minutes they usually already mention a solution to the problem. They want you to find a solution, but they already know what’s wrong and how to fix it. It is very easy to pick up on their problems. There is lots of wisdom in that book. One of my favorite quotes from the book is “No matter what it looks like, it is always a people problem”. It might look like a technical problem, but in the end, it’s a person not being able to do what they can or unwillingness to do something. Even in technical environments, they people factor is very important.

Chelsea: What do you think is going to be the next big thing to come in testing in the next 2 or 3 years?

Zeger: That very hard to determine. The trend for testers to need to become more technical is something I am seeing, and experiencing for myself right now. I think people will be focusing more on automation in the coming years. I will always be an advocate for thoughtful exploratory testing even in the automation testing frenzy. It will be hard, because not a lot of people know it well enough to value it. They go to these conferences and see all the new tools and vendors which draw them to technical automation. Also, DevOps will continue to grow in the next years. The last big thing will be an increase in security. There has been lots of hacks and people who are capable of controlling things through the internet, and there will be a lot of that happening. There will always be a new trending thing that people will continue to follow and develop.