Continuous testing

The Clear Path of the Continuous Testing Maturity Model [Podcast]

On this episode of Continuous Testing Live, Tricentis founder, Wolfgang Platz, discusses why the current state of software testing warrants innovative, lightweight guidance through DevOps and digital transformation initiatives. Learn how the Continuous Testing Maturity Model supports the best mix of both open source and commercial solutions any enterprises’ unique needs, and how the model will be made even stronger in the years ahead.

Be the first to hear each episode of Continuous Testing Live by subscribing to the show on iTunes, Google Play, or SoundCloud!

Wayne: I am here with Wolfgang Platz, founder of Tricentis. It’s been an incredible year for Tricentis, Wolfgang; it’s been pretty remarkable. What a ride.

Wolfgang: Yeah, thanks. I’m super excited as well.

Wayne: As I look across our industry, which is software testing, I see some pretty incredible things happening. First of all, it looks like complete turmoil. Do you agree with that?

Wolfgang: Yeah, that’s true. It’s always the same pattern. Right? There are new technologies arriving, and new technologies are enablers of disruption and transformation, and it seems to always be the same pattern. It’s like a big blast. The market has been more or less homogenous here in this space, and there’s been the big incumbent in HP. All of a sudden, the new technologies force it to break up completely. What happens then is that, first of all, the old things go to die, which, you see right now happening. But, in addition to that, out of this explosion there comes new players, and at a certain point you then see the gravity kicking in again and things are coming together and forming real platforms. I think we’ve right now entered this process of “aggregation” again. And yes, I’m super excited because the continuous testing platform, which we always tell people is about to be built— I think Tricentis will ultimately be the one who provides it.

Wayne: I would have to agree just because from what I can see in the marketplace today, the transformation initiative, such as digital transformation, or Scaled Agile, or initiatives like DevOps is really blowing apart that old-school platform that I’ve seen out there. Which, leads me to a question I’ve been dying to ask you. So, testing has been around for a while and there have been maturity models, and training courses and all this stuff built up around software testing or test automation. Does it apply anymore?

Wolfgang: That’s a great question. Unfortunately, it’s going to take me I think a couple of minutes to talk through this if that’s okay with you?

Wayne: That’s fine. There’s so much education being built up. Conference after conference of people educating people on software testing.

Wolfgang: Actually, because you mentioned it, I think that’s also something we’ve got to learn. You say there’s so much education? Yes, that’s true. We’ve got to make sure our education is kind of lightweight, right? You see all these maturity models, and you’ve been pointing to them, there’s so much stuff out there and it’s so tedious to go through it. It’s so voluminous, it’s endless scripts, and it’s such a heavy load that people shy away from it. Some of them, well, they pretend to implement things. But, come on. There must be a way that takes into consideration that we are in is services-dominated space.

Tools can do so much, but you’ve got to have the right approach. So, there is a need for a model to drive the services, but on the other hand, we want to make sure that this is digestible. Right? The way we live it, is we’ve come up with something on our own that we call the “Continuous Testing Maturity Model.” Now, you’re going to ask me, “Hey. Come on, why another one of those?” No. It’s not one of those. It’s lightweight. The way we do it is we show a very, very clear and easy to understand path to drive our customers alongside our five maturity steps into continuous testing.

Why is that required? Because, what we found out is when people enter the continuous testing space they tend to just jump on UI test automation. Right? The probability that you end up with disappointment and disillusion if you just focus on UI test automation information is very high. So, what we run into is actually a series of challenges that you need to solve in order to really get to continuous testing, and this is what the Continuous Testing Maturity Model talks about.

We do not see that as just a kind of instruction. We have an own department within Tricentis called Customer Success. They help customers to move from one maturity level to the other. It’s not only real consulting that we do alongside that, we have also come up with several, so to say, sanity instruments, which check whether your test artifacts are compliant with the next maturity level you’re aiming for. So, it’s partly tools supported. On the other hand, it certainly points to the services we need to apply in order to get to these maturity levels. I think that’s the kind of mix which our customers love.

Wayne: That’s really interesting. If I heard you correctly in that last statement, this is not just a static measurement of a maturity level which just goes away-

Wolfgang: No. How could it?

Wayne:…This is dynamic.

Wolfgang: Yes.

Wayne: That’s fantastic. So, that dynamic nature allows you to actually understand that you’re going to hit the goal or maybe miss it.

Wolfgang: Yes. Exactly, The way we do it is, we go to our customers and we would start with an assessment so we could find out where they are today. A lot of them are pretty elementary regarding their maturity. Most of them will start on the maturity level M1. But, then we come up with a clear route of steps and initiatives to be taken that are going to lead them into M2. And then it goes step by step even further to get to the full levels of maturity. They will be able to, upfront, have an inspection of what they’ve come up with in the repository to find out whether they’re on a good track or not.

Wayne: That’s interesting. Let me take it from one perspective. So, if I was going to try to characterize maybe a general organization that we would see out there. I know this doesn’t apply to everybody. Maybe they’re doing about 80% manual testing. They might have a small group that’s working with the web front end that has Selenium and they’re building scripts. Now, they’re looking at this transformational process associated with digital transformation or DevOps and they’re trapped. Here comes Tricentis, what’s our first step?

Wolfgang: First of all, we would need to understand why they are where they are. Actually, the situation which you described is pretty typical. It seems to be such an easy entrance into the field of continuous testing—and especially test automation—to jump on open source, right?

Wayne: Right.

Wolfgang: A lot of our customers, they would go and start with a Selenium implementation first. Why? Because, the tool is free. It seems to be such an easy entrance, but what’s happens then is they all of a sudden, they realize, “Wait a minute, we’re running into the same maintenance issues that we had with HP UFT before.”

Wayne: Right.

Wolfgang: Or, “No. It’s even worse.” Because, Selenium wouldn’t even have this object repository. They’re facing the same issues and now they say, “Well, okay. Seems we’ve got to do something about it.” They would now try to get real developers to build an abstraction layer on top of that, a framework. This hurts. Right?

Wayne: Exactly.

Wolfgang: First of all, these guys are rare, they’re developers. And then, they are expensive. So, all of a sudden, you’re removing a very, very valuable capability from your development into the testing space. What we need to talk to them and say, “Where are you? Are you satisfied with your strategy on Selenium? Is that something you want to pursue further or are you willing to shift into a different approach with Tosca?”

If they say, “Yeah. Well, we’re going to stay to Selenium.” Then we have the option to embed what they have been doing and make use of that for the specific technologies that Selenium is able to address, like HTML. But, especially when it comes to complex enterprise applications, there are so many more technologies out there. You can’t just constrain that to Selenium.

We would have a mixed approach. You would have some test automation in Selenium, but the overarching management capability, plus the automation on other technologies—this would definitely come from Tosca. And as soon as we have come up with that kind of a structure, then we’re open to introduce the other elements that make test information successful. Elements like test data management. test system provisioning, meaning, service virtualization, in order to be able to drive it this far. Right?

Wayne: It’s very interesting, because test automation is obviously beyond just the concept of automating the test. It’s enabling that test to actually become continuous in any kind of environment that they face. So, taking those steps towards maturity sounds pretty incredible. I know it’s going be a big release for Tricentis in 2017-2018, any news you can share with us about the release of the Continuous Testing Maturity Model?

Wolfgang: The Continuous Testing Maturity Model is pretty savvy out there. We’ve gathered feedback from a very, very broad variety of customers and we’ve run the model through in its entirety in large enterprises now. So, I think the Continuous Testing Maturity Model per se, is in pretty good shape. You shouldn’t see it as a draft; it’s really proven out. The evidence we have is awesome. So, there’s not too much of innovation you need to expect on that front.

Where you see the innovation, though, is tied to our product more. It’s that we do believe that when you talk continuous testing this is way more than just a functional area. We have expanded our capacities and capabilities into the load testing space. What you now see in 2017, and coming in the beginning of 2018 is a fully new way of doing this. Which, all of a sudden will make load testing continuous. So, all of a sudden you will be able to use your functional test assets for the purpose of load testing.

Given the complexity of interfaces today on a protocol level and the challenges that come with it, we do think this is really a revolution. Our customers are super excited about it and we can’t wait to be able to present that to the outside.

Wayne: So, it’s really interesting. If I can try to sum it up … First of all, measuring testing has been really one of the major challenges, as people have tried to understand really what testing means. But, you’re saying as well as measuring, the process of testing is also critical in actually assisting someone to grow in their maturity as this complexity starts to arise. You obviously get bumps in the road, whether it’s technology, people, or process. Right?

Wolfgang: Sure. I mean, metrics are certainly something very important, and metrics come with the model because whatever we do, whichever level of maturity we want to get to, we’re not doing that for the sake of achieving maturity and being able to tell others about it. Those metrics need to be bound to real KPIs. So, what you’re going to encounter when you get familiar with the Continuous Testing Maturity Model is not just a description of “what’s next” in terms of which process levels to achieve, which cultural aspects to cover. You will also get to the real fact level and see KPIs. KPIs that we want to improve along the way.

Wayne: Wonderful. I am here with Wolfgang Platz, founder of Tricentis and we’re talking about the Continuous Testing Maturity Model. I’m assuming this is going to be something that we’ll see many views of in 2018. Wolfgang, thank you for spending the time with us.

Wolfgang: Thank you, it was my pleasure.