James Bach and Grigori Melnik: A brief history of software testing

How do we release applications faster, smarter, and with the “right” level of quality and security? That’s the challenge software delivery and IT teams face every day – and the focus of the award-winning new video/podcast series, DevOps Unbound.

Across bi-weekly videos/podcasts as well as monthly roundtables, DevOps Unbound gathers thought leaders on CI/CD, security, cloud, open source, testing, and more for straight talk on what’s required for success. No slides. No spin. Just great guests, candid conversation, and strategies you can apply immediately.

To kick it off with a bang, host Alan Shimel invited two of the testing world’s most respected and passionate “disruptors,” James Bach and Dr. Grigori Melnik. Despite their many differences (proud high school dropout vs. former university professor, a love of breaking things vs. a love of building things…) the two actually have a lot in common. Both have been thinking deeply about software testing for decades, both are tenaciously committed to advancing the craft of testing, and both are willing to shake up the status quo in order to achieve that. Also, neither is terribly shy.

Nothing by way of summary can do this discussion justice. The full session video is below; you can also read the complete transcript or consume it as a podcast.

Here’s a selection of memorable moments from the conversation…

The first book – and book chapter – on software testing [14:44]

[James] Okay, first, do you know what the first book on the subject of software testing was, the first known book? A little quiz for your Grigori

[Grigori] Is it the Myers book (The Art of Software Testing. New York: Wiley, 1979)?

[James] It’s before that. It’s Program Test Methods from 1972 by Bill Hetzel. Now that book was a report on a testing conference – the Chapel Hill conference – which happened in ’72. That seems to have been a historical watershed event in testing. Before that, the first chapter on software testing that I’ve been able to find was written in 1961 by my teacher, Gerry Weinberg, who wrote it when he was 25 years old. It’s a brilliant chapter on testing.

What’s fascinating to me is how differently Weinberg talks about testing in 1961 compared to how they talked about testing in 1972…In the 1961 book (Computer Programming Fundamentals), Gerry talked about testing as a matter of imagination, not automation, as a matter of being suspicious and critical, as a fundamentally unalgorithmic process. And he develops this beautiful story in his chapter about a programmer who’s sure that he’s found the last bug and then finds out that he’s wrong. And then he adds another check for that. And then he’s sure now he’s found the last bug and then he sees something else go wrong. And Gerry’s conclusion is look, we’re working on complex systems and this is always going to be like this.

Agile, DevOps, automation, and tools, tools, tools [19:00]

[James] In the nineties, the urge to unbind ourselves from the world of waterfall and heavy documentation created Agile, but it also created the context-driven school of software testing, which is a humanist approach to software testing.

DevOps and Agile continued to sort of take over the world, except that it seems to me that Agile in a lot of places has lost its humanist foundation and it’s turned into tools, tools, tools. If we’re going to really talk about unbound, DevOps Unbound, it seems to me that we have to be specific about what is it we’re unbinding. I hope what this means is that it is people being unbound, using DevOps as a tool rather than…

[Alan] But not just a tool. I’m sorry, I didn’t mean to jump in. But you know what, James, we don’t want to make the same mistake that Agile made. And you just said, one of the problems with Agile is we went from humanist to tools, right, too much of a reliance on tools. Let’s not make the same mistake in DevOps. That’s something I’ve been advocating for a while too – we can’t lose the human in DevOps.

[James] Actually, I want to hear you say more about that, Alan, if you’re willing to say a little more about that.

[Alan] I’m not shy. I’ll give you my two cents. Look, I started DevOps.com. I cover this space pretty heavily and I have for seven or eight years now. I don’t want to say Agile failed; Agile is far from a failure. But, there has been a reliance on tools, maybe over process. When we think about things like ITIL and ITSM versus DevOps, we tend to point fingers at ITIL and ITSM, saying there’s too much process, it’s too heavy, there’s too much of the whole change management thing.

But there are good things there. Nothing’s all bad and all good. The world’s not black and white. It’s grey. Right? And so there are things we take out of ITIL and ITSM that we put into DevOps. But DevOps – to me, anyway, and it’s just my humble opinion – is fundamentally about people and teams and culture, not the tools.

We could fight about it till the cows come home, but at its nitty-gritty, DevOps is making sure that devs and ops and test and security and these people work with a higher degree of communication, with a higher degree of cooperation, getting jobs done faster. And we do that by adding things like automation to the mix. But automation for automation’s sake alone is not enough. To do automation, yes, you’ve got to use tools sometimes to make that automation work. I get it. But if we lose the fundamental aspect that DevOps is, at its core, about humans, I think we lose what DevOps is. I don’t hold myself out to be as accomplished as either one of you, but I…

[James] I still love you, Alan. I love what you just said. I think if you can hold on to the principle and the feeling behind what you just said, we can do anything with tools and we’ll be okay. We’ll be okay if we’re talking to each other, if we’re caring about each other. I keep seeing people losing that. Of course, any technology and any methodology, when it’s applied sociopathically, will become abuse. Those of us who can should do whatever we can do keep it non-abusive and keep us talking. And then I think DevOps can be a wonderful thing.

Tools + humans: negotiating boost vs. bind [23:40]

[Grigori] It’s super encouraging to hear what you said, Alan. Again, I’m a very big believer in the greatness of the human mind, but it seems to me that the pendulum keeps swinging back and forth in our industry. Remember the whole infatuation with CASE tools back in the eighties and nineties? Then, later on, there was this whole big movement of software factories, where the idea was that you can go and abstract away at some high level and then the tools, will generate all the code and everything for you and make it all highly maintainable and all that. Of course, that failed too.

But I think there’s this desire to come up with boosters, some way of equipping the humans to deal with the inherently complex problems that, for now at least, require human brains. This whole vision of computers writing computer code – that’s been promised since the first era of AI, the sixties and seventies, then it kind of died out. Now it’s the same thing again, having the computers entirely responsible for a test strategy, testing entirely automatically. I think it’s utopian. If you consider what tools are capable of doing today, I think the human will remain at the center of this activity for a very, very long time in the future.

All the tooling is great because tooling gives you additional powers and additional ways of automating the most rudimentary, the most boring elements of the job. This lets you free up time to think about new theories, new hypotheses, new models, new approaches…

[James] But you’ve got to consider different kinds of tools. Consider a tool like a programmer’s IDE. When I use a programmer’s IDE, I don’t feel oppressed. I feel empowered. I feel like I’m riding a magic carpet and I have laser cannons mounted on it. I feel wonderful! But when I use most testing tools, I feel oppressed…

[Alan] Like you’re castrated.

[James] Like I have just a few possible things that I can do and if I have anything that’s outside of what the test tool designer has thought that I might want to do for testing, then I’m not allowed to do that. This is how I feel when I encounter something like Cucumber, which is very restricted, and the Gherkin language, which is very restricted. It’s stopping me from going on flights of imagination. Unsurprisingly, when programmers create tools for fellow programmers, those tools make users feel powerful and free and able to do all kinds of wonderful things. I would like to bring the same sense of freedom to testing. That’s part of what we’re doing with the transformation work we’re doing at Tricentis.

A testing platform for the future [28:04]

[Grigori] There’s a new initiative that we are driving inside Tricentis, a whole new testing platform for the future, something akin to the idea of an IDE. It’s an integrated testing environment (ITE) that places the human being, the human tester in the very, very center. All these other snippets and utilities and services and everything are built around the human tester, to support the individual no matter where his or her line of thought, line of exploration, line of test execution may lead.

We’re envisioning this to be a super configurable, constantly morphing tool—almost chameleon style. Depending on the context and the environment, the different capabilities of the ITE will represent themselves and then the human will actually pick whether this is something that they want to take advantage of or not—as opposed to being forced into a specific kind of mold or one particular way of doing testing. I don’t think anybody has attempted something of this caliber, specifically in the space of test tooling before. That’s why we’re so enthusiastic and so passionate about this project. Aren’t we, James?

[James]: Yeah. I also don’t think that anyone is going to be able to copy it easily because the only way that they could actually compete with us is by hiring someone who loves testing, which test tool companies hate doing because testers are troublemakers. I still can’t really – it’s hard for me to fathom that Grigori really knows what he’s done by hiring me because I’m a constant gadfly.

[Grigori] Laughs

James Bach’s role as Technical Fellow at Tricentis [30:14]

[James] I’m constantly complaining about the state of testing tools. I want things to be fundamentally better, but that means we have to fundamentally change how we think about what a tester really is. This might get us into the role of testing a bit, but to me, a tester is a puzzle solver, a problem finder. I’m not a button pusher and I’m not a verifier. I don’t verify things. Tools can verify things. I’m not looking to prove that something works; I can’t prove that anything works. What I can do is collect evidence and then draw inferences from that evidence with a critical mindset.

Of course, lots of tools are collecting facts about the products that I’m testing and that can be automated and so we’re pulling in these facts. And we can say, “We got this output and we expected to get this output, so apparently there’s no problem here as far as we can tell so far.” But then the tester layers onto that a critical mindset saying, “But how do we know and how can we be fooling ourselves?”

I’m bringing that mentality into Tricentis and I’m really surprised—so far they’ve been quite welcoming of me. For years and years, I’ve tried to help other tool companies with this and basically, you know what reaction I get? They basically say, “Well, our customers just want to have automation that just pushes a bunch of buttons. We don’t really care about test design. We don’t really care about empowering testers. That’s not going to help us make any money, James, so we’re really not interested in supporting the skilled testers that you keep talking about.” Tricentis has a different kind of vision that I think is hard to match.