Your transformation toolkit

Advance your enterprise testing strategy with our transformation toolkit.

Learn more

Digital transformation

How do you measure testing transformation success?

Christian Plaichner and Wayne Ariola recently sat down to chat about what’s really required to drive a successful testing transformation….and how to communicate your achievements in terms that matter to the business.

Want to hear more from Wayne + Christian? You’re in luck — they’re back for a second round. Catch them as they explore the top challenges that QA leaders face when leading enterprise testing transformations.

Here’s the complete transcript:

Wayne:  Hi, I’m Wayne Ariola, the CMO of Tricentis. And I’m here with Christian Plaichner. Christian, first of all, I’d like you to tell us a little bit about your role at Tricentis. But after you do that, I have a pretty serious question for you.

Christian: I joined Tricentis in the beginning of 2018. I’m responsible for Customer Success Services, which as you know, is a very, very broad function. I think it’s also important to think about customer success in two ways. One would be as a mindset for the whole organization. The other thing is the department and the responsibilities that you have in these different roles, and how we can help our customers to be successful.

Wayne:  Very interesting. Along that theme, though, let me ask you a question. Because what we heard at Accelerate, really consistently, is the fact that it’s beyond this idea of change. The concepts of digital transformation, scaling Agile, or transforming the organization towards more of a DevOps mindset. It’s here. It’s done. We’re over the hurdle.

Christian: That’s true.

Wayne:  And now it’s time to really take the steps to, in some cases catch up, in some cases get ahead. But most people are just learning how to crawl at this time, I think. That’s the feeling we got from the audience. What I’d love to ask you is, what do you think is required for a team or a group of teams to really make the transformation when it comes to testing?

Christian: A very fair question. I think that’s also one of the things that we think about and discuss a lot with customers, and also with industry specialists. One topic is the tools. Are the tools in place? Are the tools flexible enough and have the capability to provide what we need? I think an even much more important and more complicated situation is, how can you make your processes, and especially also, how can you make your organization work to apply to the new world? Everybody talks about the Amazons and the online companies of this world. The reality is, many of the big enterprises will also be testing core systems that are very critical to their day to day business. It’s a very different approach. You also have to always keep in mind, many of them begin from a very, very different starting point.

We spend a lot of time providing help and the experience to our customers. I think the initial thing that we have to be careful about is figuring out where they are today, and also where they want to be. Not everybody can be (or wants to be) an Amazon. It’s always something we have to consider, right? There are various ways that we approach this, but I think it’s very important to understand where they are today and what they really want to achieve. And then it comes back to, tools, people, processes. Very often, especially if you talk to bigger enterprises, the tooling is not the issue. Either they have everything in place, or they can buy it. That’s typically not the problem. The problem is always, how can I transform my organization so that they change their mindset and their approach, and what does this mean for the setup that they have today?

Wayne: And I know that Tricentis has a continuous testing maturity model, and I want to actually ask a little bit about that. But when you go into an organization to do some of these transformation-type projects, can you characterize the level of senior manager buy in? Are you seeing it as a consistent factor, or is that something you guys are looking for when you go into an organization?

Christian:  That’s something that’s also a very good point. Testing was always…I don’t know if red-headed step child is the right word, but I feel like it’s kind of how people perceive it. I think we are now in a situation where you need top management buy in to make it part of the whole thought process, and as I said, part of the whole process at the organization. And it’s also a challenge. We see very often that QA managers are in an unfortunate situation because they have their spot today, but it’s difficult to understand what that spot looks in the future. And that needs management buy in. And looking at the whole picture from a ten thousand feet view, to say where are we? What do we want to do? That requires management buy in. Because otherwise, you can’t change your organization. You’ll never be able to really implement Agile and DevOps concept into these kinds of enterprises.

Wayne: Interesting. And let’s go back to the continuous testing maturity model concept. In terms of actual continuous testing maturity, where do you think, in general, organizations are today? Are they on the lower spectrum? Are there people now moving as fast as possible to the higher spectrum? How would you characterize it?

Christian: It’s a difficult question, but it’s a very interesting question. What we see is, I think there are various approaches to this. One is, if you talk to top management, they feel like they are much further along than they really are. That’s also why we want to talk to top management, the QA team or to QA management, and then also to the testers. We want to see the whole picture. On the other hand, we always have to consider what’s possible for them, and how fast they can move. We go back to the theme of, are they ready to change the way they do stuff? We provide what we call the continuous testing maturity model, five steps. It helps them to measure themselves and to benchmark themselves. And also give them a path forward.

And the other thing is, I’m not sure how many of them are consistently looking at KPIs and metrics. The challenge then is, if you don’t look at KPIs, it’s very hard to make the case about what your role is and how important you are. One of the pieces that we have in our maturity model is to make sure we nail down the KPIs. And we also provide what that means compared to other companies.

Wayne: Can I just ask you about the KPIs just a little bit? Is it the lack of consistently visible KPIs, or is it the wrong KPIs for where they’re going? What is the major issue there?

Christian: I think it’s kind of both. It depends on the organization. It’s really a combination of these two things. The metrics that everybody looks at, especially when they move into DevOps, are very development-oriented. There is not so much about testing. We all know, the problem really becomes transparent when you’re suddenly in the news. Other than that, you’re happy with your QA, or you’re not happy with your QA, but as long as nothing really bad happens, everything is fine, and nobody really worries. And I think we really have to change their minds and also arm our QR managers and QR teams to come back with the information and show the value, and also help the organization to understand how this fits into the new role. Because if you talk about quality at speed —  and ideally, you even reduce the costs by half or whatever —  that’s a very, very challenging approach. And it’s better you are measuring where I am today, and where I want to be tomorrow, and what does that mean? How did the changes that we made help? It helps them to have a discussion based on facts and not based on “how do we feel about the quality of the product?”

Wayne: Exactly. That’s a great point. By the way, just to tie in something, we did a joint research project this year with Forrester. It’s called DevOps Leaders and Laggards. We measured the factors that distinguished a DevOps leader versus a DevOps laggard. And by the way, continuous testing was one of the number one factors. [Download the report here]

That report showed that organizations who were DevOps laggards were more into counting statistics and KPIs. And organizations that were DevOps leaders were doing more risk-based analysis or contextual analysis. And it drew in two pieces of information. One, when there was a bottom-up motion, they were more about counting their artifacts and work. But when it was top-down sponsored, the idea of business risk started to infiltrate into the organization, and it forced a change in their metrics or their KPIs. So I think that’s very, very interesting.

Christian: Absolutely. We have another approach, a business value assessment, where we go exactly into this direction. We at Tricentis typically measure three things. What’s the cost, what’s the speed, and what is the risk? We provide a list of about 25 KPIs that we recommend, and we prioritize them with the customer. It goes back to what stage they’re in, where they want to be, and what makes more sense. At the end of the day, it always comes down to these three things. But you want to make sure, depending on how you fit into the chain or how far along they are, there are different things that are important to visualize and make transparent for management to really show what that means.

Wayne:  So Christian, first of all, thank you very much for spending the time. I know that the Customer Success Services team is imperative to the Tricentis mission to really reinvent software testing for DevOps and Agile. So thank you for your time, I appreciate it. And quite honestly, I’d love to do this conversation once again.

Christian:           Wonderful. Thank you very much