Testing is not a monolith: lessons from DevOps Unbound

Do you remember a world before DevOps? Testing was set apart, an island reserved for testers only.  That software testing island at the end of a release wave got swamped, overwhelmed by today’s continuous software improvement across all industries. As testing and development teams are now crisis-averting by collaborating, we are also presented with a variety of testing approaches to choose from.

The many forms of testing have recently been explored in DevOps Unbound, where thought leaders take on DevOps’s toughest topics head-on (no slides, no spin). For this episode, we ended up back where the series began: looking at testing. Is automation the be-all and end-all? Does manual testing still have its place? And what about everything in between, from performance testing to BDD?

Alan Shimel, DevOps.com CEO and Editor in Chief, guided this session on scrutinizing different testing methods with the insights of Christine Fisher (NAIC), Leonardo Melendez (Qualitest), Ken Pugh (Ken Pugh Inc.), and Mitch Ashley (Accelerated Strategies Group). Some of the topics covered include:

• Getting a grip on the different software testing types
• Is better testing just an argument for adding automation
• Everyone should have a seat at the table

This engaging discussion sure leaves an impact on Alan: “It’s fascinating that testing has been so profoundly affected by DevOps. Rather than being left behind, it’s being accelerated through automation, speed, and more focus.”

[Watch the episode here]

Getting a grip on the different software testing types

The session begins by diving into BDD – we’re reminded that, in a nutshell, it centers on understanding the behavior of the end user. Christine points out that we should understand these requirements before testing and use common language across teams to underpin them. Ken explores the importance of the triad – the customer, developer, and tester – which, as Leonardo clarifies, are the “three amigos!” Our amigos should get together prior to implementation to create scenarios; scenarios become the tests, and the developers are given said tests to run against their code.

This sounds straightforward enough, right? Not exactly! Initially Leonardo states that developers are responsible for marking tests as completed but corrects himself by saying “in general you’re supposed to be a team, and everyone is responsible for each element.” As BDD encourages collaboration and a shared accountability, it’s no secret that this is no easy feat, but it is a venture that pays off.

Ken brings the testing matrix to the fore: “The testing matrix shows testing that’s not just about functionality. It describes all the different sorts of tests, from the functionality tests which are on one side of the matrix, to the quality tests which are on the other side – including performance, security, usability, and so forth. ATDD and BDD tend to focus on the functionality, but you ought to have all those other tests already established ahead of time.”

Is better testing just a case of test automation?

Alan recognizes that most of our current testing approaches pre-date DevOps, but it’s DevOps that actually propelled us to adopt many of them. With this in mind, Alan asks our speakers how we can get better at testing.

Ken is a true advocate for “exploratory testing” – using human intelligence and creativity to find the intricacies that functionality tests aren’t going to cover.  Leandro shares that many tools give transparency into the code and allow us to monitor it, catching bugs way before the push to production. Using a ‘man with hammer’ analogy, Leandro hits home that we have evolved way past the one-tool approach – “Prioritize, distribute and make the best effort with each of them.”

Mitch poses a question about whether the ever-evolving conditions we’re in today, where software agility is not an option but a need, impact the way we work. Leandro compares the work of a manual tester to that of caring for horses; it is still essential and can’t be replaced. When we travelled in carts and not cars, horse groomers were commonplace. Today, where cars and automation drive our daily lives, there is still a niche for horse care, so the need for it persists – much like manual testing!

If your QA team sits outside of development, Christine invites you to reconsider. Her wakeup call came after testers on her team expressed concerns about their responsibilities changing dramatically. For instance, manual testers who were reluctant to learn automation. The turning point was “realizing that quality is the responsibility of the entire team, and that everybody needs to contribute to that conversation. The most important part to me is that the testers are at the table and part of that conversation.”

Everyone should have a seat at the table

This DevOps mantra underpins Christine’s contributions throughout – ensuring that developers and testers know where they sit in the process early on has set up her team at NAIC for success.

This has been formalized as a test strategy meeting – “we get the Business Analysts, manual and automation QA in a room and we talk about each of our stories. Should we automate it? What are we looking at manually? They took it further than when we first started, and they came up with a scoring guide to say what is our effort level with this, what’s the repeatability for the user etc., and they’ve realized that those conversations are so important.”

Giving the team room for critical thinking has drastically increased the quality of their tests from all angles. What’s not to like?

Join the conversation

In bringing voices from different corners of the software testing globe to the DevOps Unbound table, it’s a joy to hear a variety of spins on hot DevOps topics.

Alan wraps up with the powerful sentiment – “It’s fascinating that testing has been so profoundly affected by DevOps. Rather than being left behind, it’s being accelerated through automation, speed, and more focus. DevOps has been one of the best things that’s ever happened to testing.”

[Watch the complete “Testing Is Not A Monolith” roundtable on demand]