Agile testing

Continuous Testing Live: What Does QA have to do with DevOps?

Was software testing really left out of DevOps just because it was left out of the name? What’s required of testers in organizations launching DevOps initiatives, and how will their role change as DevOps adoption scales? We asked these questions and more of Patrick Turner, CTO, Small Footprint to learn how he views the current challenges in the testing community, and in software organizations as a whole.

Don’t miss a single episode of Tricentis’ new podcast, Continuous Testing Live—subscribe today at iTunes, Google Play, or SoundCloud!

Noel: So, in your session yesterday, you talked about viewing DevOps as “an extension of agile.” I know everyone’s got their own definition of DevOps and agile, or, perhaps they don’t bother trying to define it. But, I was curious as to why you saw DevOps that way, and what kind of advantages there are if you’re explaining DevOps that way to someone else. Maybe you’re trying to explain to someone how they’re going to be able to adopt DevOps. Why that is the chosen approach you’ve taken for conveying what exactly DevOps is.

Patrick: Yeah, sure. I see agile as a big shift in the way development teams communicate and collaborate together. There are certainly lots of tools and processes that are involved in agile, but ultimately, the whole purpose is getting people to actually not sit in their cubicles and ignore each other. What I’ve seen happen with DevOps is it’s taken that same model, where agile was really focused on engineering and QA teams, and it has driven that downstream to product delivery. So, now, the teams are using tools again, but more importantly, DevOps is creating better communication and transparency down through the product delivery process. And, so, DevOps is really just an extension of agile down through delivery, as far as I’m concerned.

Noel: Do you ever go into organizations to help them with DevOps, and they haven’t even really started an agile transformation? Is it the kind of thing where when it’s that situation, they need to first do agile, or can they do them hand-in-hand? What’s the approach if they don’t already have a strong familiarity with agile?

Patrick: Well, a lot of that’s a technical answer. I think, generally speaking, most of the organizations that we work with either haven’t really implemented agile or haven’t done agile very well. I think that’s the norm, really. And so, as far as whether or not we’ll bring in DevOps best practices, our goal is to certainly do that at the same time if possible. If it’s a greenfield opportunity with the software, and we can architect that into the software from the very beginning, we’re going to do that. Sometimes if we’re working on a legacy product, there’s a little bit more engineering that has to happen to get it to that point.

But our goal is to do it all at once because, again, we feel like it’s bringing that culture shift to the entire organization, and it’s important to do all of it as soon as possible.

Noel: So, that moves right into a question I had about “culture.” I noticed that that Small Footprint‘s goal is to come in and help with culture. The hardest part of doing DevOps is changing culture, I would say. It is pretty well understood that a tool cannot come in and do that for you. In your session, you showed the difference in cultures between developers, testers, and operations, but when you come in and try to help change that culture, is the goal to get all three of those groups to follow a single culture? Or do they each still own their own way of doing things, their own culture, but that culture is better aligned with overall business goals?

Patrick: Yes, similarly to our agile discussion, traditionally, organizations have lived in silos, and it’s been throwing the hot potato over the wall. So, whether it’s development with QA, or development to product delivery, it’s really been people trying to stay out of each other’s business. And, really, part of the cultural shift is changing those roles and the way people see what they do. So, development really needs to take more responsibility for what happens in delivery. And so, as a result, system admins have to give up a little bit of that, of their kingdom to do that. But, at the same rate, what they’re going to appreciate is they’re going to have better visibility of what’s coming down the pike as to what they’re going to have to support.

Likewise, with QA, QA’s generally been sitting in a silo, where they didn’t really know what to test, or how to test it, or even where to go to test it. And, because of some of the tools you can implement in your DevOps practice, it enables QA to really have great visibility of what’s coming. They’ll know where they have to go to test it, as well as actually being able to tell—all the way back to the task level—those features and user stories they’re going to be looking at in that new release.

Noel: Very cool. Let’s talk about testing specifically. There was a really telling moment in the session yesterday. You said, “Some of you in the room may even be thinking, ‘What does QA have to do with DevOps?'” And, out of the corner of my eye, I saw a couple of heads nod, affirmatively. I winced in my chair a little bit because that’s definitely something that has to change. I don’t actually believe that when the term “DevOps” was coined, and those early banner-wavers started rallying around it, I don’t think they ever intended for testing to not be a part of it.

But, that view that testing isn’t just as crucial to DevOps as development or operations, and seeing firsthand that people may still view DevOps as, “Oh, this isn’t about testing,” that that’s got to go pretty quickly.

Patrick: Yeah, it’s interesting. Because some of the feedback I got yesterday was that I didn’t talk a lot about testing. But, I feel like testing is probably the area that had the biggest impact because of DevOps. And most people just assume that it’s the product delivery guys and the release managers that are really heavily impacted. But from QA’s perspective, their impact on the project is changing. Their skill sets have to change. And really, what they do every day is changing dramatically. And so, the obvious example is automated testing, which you could argue is not part of DevOps.

But, from my perspective, DevOps—with continuous integration especially—has enabled us to do automated testing better. Because you can have that CI server kickoff that automated test. In the past, it wasn’t really tightly integrated if you were doing automated testing, which, of course, 99% of organizations weren’t doing that. At the same rate, we’ve had to challenge our QA team to learn some new skills. Some of them are really excited about it. Some of them aren’t, but the great news is it doesn’t mean you have to kick anybody off your team. It just means they have to go in different directions. So, some of them are going to have to learn how to build some automated tests. Maybe do a little bit of coding even.

Others are going to find even better ways to add value, such as user experience testing and some other things like that. But I think DevOps really provides a great opportunity for QA to add a lot more value other than just looking for ways to “break” the code, which has kind of been that traditional role.

Noel: Right. I think shows like these are good examples of ways that testers can get more involved. I’ve been to strictly DevOps shows before where there’s a call for hands from the presenter, and they ask, “Who’s here from operations?” and it’s half the room. And, “Who’s here from development?” and it’s the other half the room. I have never seen a presenter say, “How many of you are testers?” I don’t know if they just don’t expect them to be in the room, or if they think they won’t want to raise their hands in the middle of the room of developers and operations, but I’ve personally never seen an attempt to speak directly to any testers who may be in the session.

Yesterday, during your session, you had a poll, and one of the questions was “What’s the one main thing you want to get out of for your DevOps implementation?” I was so pleasantly surprised to see that “Higher Quality Software” was the number one response. I took a picture of with my phone as proof!

Patrick: Yeah.

Noel: I think that since higher quality software is going to remain a big goal for people, and that testers either have got to start going to these DevOps shows or the shows themselves need to start making testers clearly more welcome since everyone’s all in this together.

Patrick: Yeah, along with your statistic and my rough scientific survey inside the convention center here, was that about half of the software product booths here were somehow related to the QA test automation, which was really interesting and very insightful. I think probably in some ways, it’s the area that’s maturing the fastest right now. And, maybe we didn’t let testers come to these conferences before because they were too busy finding our bugs. But, I think it’s certainly changing the fastest, and I think people are seeing that opportunity to provide better quality code.

So, the traditional parts of product delivery, like where your environment is, because of DevOps, if you’re using containers and you’re seeing that you have the same exact environment in all the different areas, you’re eliminating some of those really simple—maybe not simple, but, fundamental—ways of why things didn’t work in the past. The old “it worked on my machine.”

Noel: Right.

Patrick: So, we’re knocking those things out pretty quickly, and testing, I think, has had the furthest to come in terms of automation. But, again, it’s a huge opportunity. And certainly, we all, as software developers, realize that 85% of projects are seen as failures. I think that is purely a quality issue and it’s because of poor communication and collaboration. And again, back to my first statement, that’s what our goal is with both DevOps and agile is to improve that, so that we really understand what our users and our product owners want out of their software.

And, it’s really important for the quality assurance engineers to get a really good grasp on what we’re trying to accomplish as well. Because they need to be, ultimately, that gate before it goes out to our users, in order to find out that we’re not building what they want, to have that perceived software product failure.

Noel: You also mentioned in your session that there’s “so much education out there to be had, and so much to learn.” You said, “I feel like we’re all just getting started.” I feel like it’s easy to get caught up sometimes in the stories of the people who have been doing DevOps for a long time, and you hear these things about “200 times more deployments” and these just staggering numbers. Yet, you come to these shows, and you’ll find out from people and from speakers like yourself that there are still plenty of organizations that are just getting started with DevOps, or they’re even just getting started with agile.

I can’t remember if it was your session or not, but one speaker was asking people if they even knew what unit testing was.

Patrick: That’s me, yeah.

Noel: It’s sobering sometimes to realize that we’re not quite at the point of needing to talk about whatever’s “next,” because we’ve got plenty of people, including large organizations or governments or other highly-regulated industries where it’s going to take them much longer to apply some of the things that we’re talking about today. However, it’s important that they start looking for wherever even small improvements can be made.

Patrick: Yeah. One of the other surveys at the beginning of my presentation yesterday was asking the one word that people use to describe DevOps in their organization. One of the gentlemen put “infancy” and I squirmed a little bit for him when he said that. But then he said, “What that means is that I’ve gotten started.” I think that was a really great point that he made, and I think it’s really important for us to all understand is that we’re all at different levels, and we’ve been working certainly to become a mature DevOps organization for a few years and we’ve still got a lot to figure out.

Right now, we’re trying to figure out how to do security testing as part of our continuous integration environment, because we know that’s something we need to do. And, of course, in the last couple of days, I’ve discovered a lot of other things that I didn’t even think of as being opportunities for us. But I think the real key is that organizations need to focus on making a move and just getting started. And, as I mentioned, speaking of unit tests, certainly, when we started to really try to build unit tests into our process, that was the hardest thing, just getting started.

And so, we use a little gamification and had a contest just to get people going on that, but at the end of the day, it got them exposed to unit tests. It helped them understand, in some respects how easy it was to do, and in other respects, how challenging it was to do. But the real key is just getting started. And, of course, I hope all organizations are focused on learning and continuous improvement because that’s really the key, but know that you’re never going to be done.

As you mentioned, software is never “done,” and the way we do software will never be done. It’ll always change and always improve. We just all need to be continuously working together better to make sure that we’re moving it along down the road.