Continuous Testing: What’s Really Required for Digital Transformation?

Agile testing

Continuous Testing: What’s Really Required for Digital Transformation?

All signs indicate that the next wave of innovation will be driven by AI, robotics, big data, and predictive analytics. Testers will face the challenge of testing applications that are simply beyond the scope of our current approaches. But you’ll also be able to take advantage of a new generation of testing technologies—enabling you to reinvent your testing process for the digital future.

Watch Wolfgang Platz, Tricentis Founder & Chief Strategy Officer’s keynote, as he shares his vision of software testing in the digital future. Learn how enterprise application architectures and associated delivery models are changing, how testing must evolve to address these changes, and what testers can (and must) do to successfully navigate the road ahead.

Here’s the full transcript

[/vc_column_text][divider line_type=”No Line” custom_height=”10″][vc_column_text]First of all, thanks everybody for being here in this insane time. We think we’re growing close with these Americans now, but every once in a while, you see that there’s some cultural difference. These guys would have started at 8:00 a.m., okay? And we couldn’t convince them that 9:00 a.m. is really early for Austria. So, nevertheless, you made it and I’m glad about that. The room is full. Thank you for coming. This is great.

Digital transformation, and the impact that it has on testing

What I wanted to touch on is actually three things, three major topics. Number one is the Continuous Testing Platform. I’m going to elaborate on that further because we think it’s time to build this platform, and the time is now. I want to touch on new technology trends, and I want to reflect on what they mean for testing. What they mean from a testing perspective and how we need to react to them, and last but not least, I want to take the time to bring some details about our thinking regarding the chasm between development and test. People are talking about that quite heavily, and I want to share our thinking about that with you guys.

Continuous Testing Platform

You’ve heard from Sandeep yesterday, that we have been subject to a great investment. $165 million have been invested into Tricentis, and if you have such an investment, it is an opportunity. It is a huge opportunity for us to grow, to expand, to innovate. But if you have a look at the market, you find that our biggest incumbent, HP, is gone down, and they’re going further down, even. You heard the term hospice yesterday, right? So, what’s happening is that this opportunity is not just an opportunity, it is even an obligation to us. The obligation is to build daily Continuous Testing Platform. This is what we’re up to.

If you have a look at the space where we are in, the space is DevOps and there is a series of competitors today who try to be the number one as a DevOps platform. And I personally think that probably Atlassian is situated the best. What these guys do is, they need to comprehensively cover the full flow. They need to comprehensively cover all the different stages. Planning, development and so on. Test, and then the operations. What they lack, though, is the depth of functionality, and this is what we are here. In the testing space, it’s going to be us to provide that kind of depth with the Continuous Testing Platform. And when I talk platform, I do not only mean functional testing, which we have been focused on and famous for, for the last couple of years. It needs to extend and go to the non-functional testing, as well.

Before we go into that in more depth, let’s share some statistics because, from my perspective, they’re always stunning. The software testing market today is huge. It’s $35 billion, and if you have a look at that, it includes both tools and services. Anyone of you has an idea to which extent tools are in these $35 billion? Any guesses? It’s like 10%. It’s about 10%, the rest is services. It’s a huge services market we are dealing with. And quite astonishing, this is supposed to grow further, and it’s not a small growth rate. It’s a growth rate, which is like 12% annual. It’s going to go up to $55 billion soon. That’s at least what the market expectation shows.

If we analyze that a little bit more, we’re going to find out that the growth comes particularly from application testing, not so much from product testing. You see these two different colors here? What it means is that the statistics fully confirm what we see out there, that bespoke applications are declining and instead it is packaged applications that go in and take things over. The growth of packaged applications is about 15% per year, 12% per year is the overgrowth of the testing and product testing does not grow that much. So, this is the full substitution that you see happening throughout these stats, and I’m going to come back to that on several positions today.

Let’s have a look at non-functional testing

Let me start with something that is very relevant and very interesting to us. It is with performance testing. Performance testing is also supposed to grow nicely. You see like a 15% pattern, composite annual growth rate, and it’s going to go up to $4 billion until 2021. It is so relevant for us as a tool vendor, since performance testing, quite as an opposite to the functional testing, has a way higher extent and share of tooling compared to services. So, as a tool vendor, when you want to be complete in terms of a Continuous Testing Platform, you have to have performance testing in there, and that’s what we acted on. I’m very, very proud that we succeeded in the second quarter of 2017 to acquire a company from Australia, the company’s name is Flood IO, to be with us and cover the load testing aspect.

The Flood guys are superb tech nerds, but they’re even handsome guys. You have to see them out there on the booth. So, that’s a pretty rare mix, what I found out. We’re super excited about this acquisition, and it’s not only what they have today because today they do have a fully SaSS-based business for load testing. What is even more exciting is where we’re going to take this in the future. We’re going to take this to the opportunity of full re-use of your functional test assets in the load testing space. It means that your functional test definitions that you have come up, you have automated, can all of a sudden be re-used for the load testing. This is not only going to save you a lot of effort. This is nothing less than opening the door to make load testing part of the continuous integration that you have out there.

The continuous integration is going to kick off the continuous testing from a functional perspective in the future, but it’s also going to kick off the continuous load testing. All of a sudden, the dream of continuous load testing becomes true, and we are working on this right now. See it out there on the booth with these guys. There’s another aspect on a non-functional testing that I want to highlight. This is not about performance, it is about security.

We see security growing very, very fast.

The overall composite growth rate is with 18%, which is remarkable is that, of course, mobile is driving it even faster. So, what we do think is that security is an aspect we definitely have to look at. Today, the tooling out there still has some challenges, so to say. We call it static application security testing, I don’t know if you’re familiar with that term.

What it does is it would inspect code to find out whether there are potential areas of asserts and what we find out with this is that today the technology reveals a high number of so-called false positives, meaning you get an alert but there’s nothing into it. Still, this number is very high, but it is maturing fast, and I do personally believe that within the next two to three years, tooling out there is going to be as mature that you’re going to have it and it’s going to bring great value. So, when we talk about completing the Continuous Testing Platform, it needs to be functional. We have extended it with continuous load testing, and we’re going to extend it further to continuous security testing. Then, it’s going to be complete.

Let me go back to the origin of Tricentis Tosca. I actually had the privilege to stand on a table yesterday, I don’t know if some of the guys are here still, where I saw three guys, and they all together had more than 50 years’ experience on Tosca. That’s how long we have been thinking about this, and it comes, actually, from the test automation. Test automation as the original core idea. Now, let’s see where we are on this. I don’t know if you remember two years ago, I was presenting this chart, which was trying to do a prognosis on how test automation is going to increase. Well, let’s be frank, I think it was a little bit too optimistic. Test automation is not catching up as rapid as expected, and when we dig into this further, what we find out is that it is actually a series of challenges that our customers go through.

The three nightmares of test automation

What it talks about is as soon as you have solved the maintenance trap, and everybody who has been within test automation knows about the maintenance trap. It’s one thing to set tests up, right? But it’s another thing to keep them alive and make sure you really benefit from them. With Tosca’s technology, you can escape the maintenance trap. This is one of our core prepositions. And you can escape them even for the most complex scenarios in the E2E world of complex enterprises. We also serve API, which is a way more powerful access to test logic, if you can use it. So, let’s say you escape the maintenance trap. The next thing that’s going to hit you is test data.

For the sake of automation, you need to have the right test data at hand, and you need to have it whenever you need it, and it needs to be stateful. What is means is you need to be in sync with the execution of your tests, and the effect that has on your test data, otherwise, your test is going to fail. And then, as soon as you’ve solved for that, here’s the next challenge, especially true for enterprises. You have to make sure you have the systems at hand in order to be able to test. So, test system provisioning becomes a huge challenge and what we think is the service virtualization is the appropriate solution for this. So, it needs to be three elements together to address test automation and bring the full benefit to it.

The three nightmares of test automation need to be solved, and this is something new to a lot of customers because a lot of them enter the space just through the expectation of, okay, we’re going to do some UI test automation, right? And it’s going to be done, and it’s going to be fine. But what we see is continuous testing is way more than that. Continuous testing is a real change agenda, and since we have noticed the struggle with a lot of our customers, we’ve come up with something that we call the Continuous Testing Maturity Model. Our customers access department, they have a booth out there, see them is here to make continuous testing happen for you as customers. And they’re going to help you to walk through these five maturity levels here, which are going to spread out from the basics of a UI test automation into comprehensive adoption of continuous testing in your organization.

We live through your success. This is all we care about. This is our mission. We are a product company, but in our hearts, our services are there to make you succeed. Here comes trouble. Test data management. You heard about GDPR?

General Data Protection Regulatory (GDPR)

GDPR is an EU thing, right? It protects personally identifiable information of EU citizens. Well, you know me when I say, well, the data always had to be protected, right? That’s not something new, and I would actually agree. When we entered the banking space in testing about ten years ago, this bank just had an ISA 402 audit, some of you know this? ISA 402 audit? So, the outcome of this audit was, you guys use protection data in testing, you shouldn’t do that, right? Back then it was directive. A directive means that all the different states in the EU had the right to interpret these things on their own. So, guess what? It was completely washed away. It was completely diluted, all of a sudden. It was completely optional.

So, people would point their fingers out and say, yeah, yeah, you have to take care about production data, but nobody cared. This time it is different because this is a regulatory. It is not a directive. It’s going to happen soon, it’s going to happen in May 25, 2018, and it’s going to be really painful if you screw up. So, what’s happening is that each and every breaching record is going to be subject to punishment, and each and every breaching record is going to cause one to two K of fee for you, and the overall is 4%. Here comes the stunning thing, 4% not of the local revenue, but of the global turnover, right? It seems that the Europeans have learned something about the Americans, how they punish European banks. So, this is worldwide, and the effect of that is that all of a sudden, this is not just a EU regulatory, this is going to be global. So, GDPR happens, and we’d better be prepared for that.

How can you react on this? How can you react and make sure you comply with this? There’s two ways. Number one is you stick to your production data thing, and you now mask these guys, and really mask them, irreversibly. And deterministic, meaning you make sure that they’re masked in different spots the same way. There’s another way of doing it, which is create the test data you need, which we call a synthetic test data approach. If we have a look at them and compare them, what we’re going to find out is that synthetic test data is pretty powerful. It is not the intuitive way of getting into the test data topic. Intuitively, most of the people would go for production data, and then there come some specific challenges. You would need to make sure you extract this data comprehensively. Imagine you have a super complex landscape. We have a customer who has 5,700 systems on legacy. Imagine the extent of data redundancy when you want to extract them, and you want to mask them.

Synthetic test data is really an alternative. What it can get you to, depending on the industry you’re in, up to 99% of business risk coverage. So, you should really think about that as an alternative. It might not get you to everything, especially for banks, but it’s going to be very, very powerful. Nevertheless, we have to cover it all, so you will probably end up with a mixture of synthetic data and production data. That’s what I wanted to share with you about Continuous Testing Platform. It’s to be built now, and we’re going to be the ones to build it completely, comprehensively. The next thing I wanted to discuss with you is technology overview, and the impact that this has on testing, and what I took as the basic theme here is a structure that you may have seen before. Who has seen this? Can you raise your hands? This curve is called the Gartner Hype Cycle. It tells the story about how new technologies are being adopted in the market.

You see that there is, first of all, it’s becoming public that the technology is out there. You see this as the innovation phase. Then it goes up into a huge peak of expectations. This is what we call the hype. But then, all of a sudden, people need to see what it really is about and then there’s some disillusion happening, and you go into a valley. From there, if there was really substance to it, things would recover and go into what we call a plateau of productivity. So, the three most exciting new technologies regarding IT, which are peaking these days, are the Internet of Things, Deep Machine Learning and Artificial Intelligence, and the Blockchain. And you see, given the color of these dots, what Gartner expects how long it’s going to take them to be at full productivity. We’re going to discuss the three of them.

I just want to mention one here because I think it’s funny, that is autonomous drive. If you would ever look at the hype cycle two years ago, you would have seen autonomous drive a little bit to the left, but the expectation for its full productivity would have been in five years from now. So, it seems that it’s going to take a little bit longer to make this really happen. Anyway, let’s go to the three major things here. IoT, Blockchain, and Artificial Intelligence.


Everybody knows what the Blockchain is? Everybody knows that everybody is talking about the Blockchain, right? It’s not teenage sex, it’s not what Ingo said. Trusted transactions with no third party, that’s the concept. A complete revolution. You do not need a trusted third party anymore to have trusted transactions, and the net is the one that makes it possible.

What we see out there is that there is significant Blockchain projects happening. If you have a look and do market research, you’re going to find 25%, significant. And the most public one, the most known one is probably Bitcoin because Blockchain is the transaction layer behind Bitcoin. You may have also heard from Ethereum. That’s from our fellow friends from Switzerland, from the ETH heart, Zurich, who have come up with this as a project. But I do think is really going to happen, and we see this already is that Blockchain is going to be provided as a built-in service with the large Cloud providers. In the last AWS Conference, they announced that they’re going to have a Blockchain. Microsoft Azure already has it in there. So, bottom line, Blockchain is going to be just a service in the future. You use it just as a service. It’s going to have a nice API and that’s it. What does that mean for testing?

How do you test the Blockchain?

Let me tell you something, the question is stupid. We’re not going to test the Blockchain, it’s here. What we’re going to test is applications built on the Blockchain, applications that use that service. And the way we’re going to test it is like we test any other application. So, just because a new technology is here, it doesn’t mean that testing is going to change. Not this time, not with Blockchain. Let’s have a look at the next one, Internet of Things. Very, very impressive numbers. 15 billion devices out there in 2015, that already communicate with each other, and the applications of the Internet of Things are really impressive. Smart health, smart energy, connected homes, smart health. All of a sudden, you get all the medication you need instantly, all the treatment you need. Connected homes gets warm, you didn’t have to do anything, right? It’s going to increase massively, from 15, it’s going to go up to 75 billion devices in 2025. That’s, at least, what people expect.

So, the Internet of Things will be surrounding us everywhere in the future.

So, what does that mean for testing?

Yes, it does have an impact. The impact is that the architecture layers in the bottom that are set up to serve the Internet of Things will have mini and micro services in it. This is the de-facto standard, and if you want to make sure you can test applications that rely on the Internet of Things, you have to make sure you can deal with mini and micro services. And what we have been proposing out there that you guys take API testing and service virtualization into your consideration. It’s going to become even more evident with the Internet of Things.

Last but not least, let’s go to another field of innovation here. We’ve seen huge innovations in the past. 200,000 years ago, people invented the fire. 7,000 years ago, they invented the wheel, and 2017 invented the iPhone, and it’s really changing mankind. It is really changing mankind. Let me share with you, I have been Monday on the Central Railway Station in Vienna and I wanted to go to Budapest and the train was late 50 minutes. So, I start walking on the railway station and since I was bored, I was looking at the people. Out of ten, how many do you think were just doing this? Ten. Yeah, somebody had two. Eleven. So that’s one thing.

And have you been here with Todd yesterday? Who has been here with Todd’s presentation? I think it was a great presentation. He was speaking about the power of mobile devices to relieve poverty on this planet. He talked about people being able to escape their cages because all of a sudden, you can bring knowledge to the furthest spread out areas in the world through mobile. Then he was talking about a masterpiece to be set up in testing, right? And fortunately, he associated us with this. He was bringing up this picture, and you know what I thought to myself when I saw it, as it reflects?

“Geez, she’s taking a selfie”

Can you believe that? This is a picture on the end of the 19th hundred, okay? 19th century. I see this, it’s Impressionism, and the first association I have is she’s taking a selfie. It’s changed our life completely. Mobile has changed our life completely, and while it is the chance to escape the cages of poverty, it seems that a lot of people are now going to this cage of mobile autism. So, it’s not just all good about it.

Anyway, Sundar Pichai there, great CEO from Google in this IO Conference, what he positioned out there is mobile first. He brought these great stats that Amazon has with more than two billion users on Android and so on and so forth. But then, he changed it and said, this is over. AI first, that’s the next thing. The next thing that’s going to change life entirely is going to be AI, and he pointed out a lot of super cool and innovative things that Google is going to do with this. The stunning thing about that is, it was really AI. It was really Artificial Intelligence because out there today, there’s a huge discussion about AI and there’s a lot of noise. You see, this is the number of Xs in the Internet where people are trying to read about Machine Learning versus Big Data. Big Data has always been up front, but all of a sudden, it has now been Machine Learning taking over.

The impact on testing is very, very similar to the Blockchain

AI is a service. We are not going to test AI, per se. We’re going to make use of the outcome of AI and build applications on it and we’re going to test these applications. The second question we need to ask ourselves is, is there AI going to be in our product? And yes, you have heard about that. We are doing optical character recognition. We are doing pattern recognition, in order to analyze interfaces. This is here. This is already working. This is real AI. But as I said, there’s a tremendous amount of noise out there on AI, and I had the chance to elaborate with a few of our GSI partners on the topic, and they shot a list of use cases on me where they’re going to have AI represented. And I didn’t see AI. Anyway, it’s indisputable that we’re going to become smarter in testing. We need to go ahead and make the testing smarter than it is today, but believe me, it’s not always going to be AI involved. So, I want to just make you aware that there is a lot of noise and let it not kill the signal.

Bridging the chasm between development and test

There is still a chasm, and there is something out there, which we call SDET. Who has heard that term? Okay, a few. Okay, let me explain. It’s called Software Development Engineer in Test. It was Microsoft, actually, that invented that term. It’s talking about software development engineers who are with testing, and we need to ask ourselves what’s that going to tell us? Is that going to be the future? Where is development and testing going? First of all, what we find out is that these SDETS are pretty rare species these days. Apart from these, what we call the GAFAs, they seem to be rare. The question is, should there be more? Where is it going to go?

First of all, I do believe that it is a fundamental difference whether you build a house, or you check if this house works. Or if we drive to exploratory testing, you even try to make it fail. Meaning, building something is a completely different act from trying to make something fail. It is two completely different perspectives. Development and test does need to have different perspectives, so, it is very tricky to unify the roles. Now, you could have developers now being in Dev and developers that are in Test. So, having the same skills, but being in two different roles. Let’s have a look at this. We see that a lot of development is going on these days in test automation to bring up frameworks, and I’ve been to a bank recently, a couple of days ago, right? And I presented Tosca and what we do, and this guy came up and said, hey, why don’t you do it with open source? And the answer I gave to him is, and I quoted a customer, a different customer of ours from Australia, it’s because:

“The total cost of ownership is a complete disaster”

I’ve read a Quora article the other day, and it was from a hardware developer, he said, I am sorry to say that Selenium Test Automation Scripters are going to go out of business. Five hundred answers within the first hour. And then he laid it out why and he said, it’s either you assign your best developers to come up with a framework, which is nothing but developing tests, development project to test, or you let go completely. It’s not working out. So, the total cost of ownership is huge, meaning, I really doubt the SDET thing. I really doubt it, but development and test need to get together. How does it work? How is it going to happen? The conjunction of development and test, from my perspective, is going to be through the artifacts, through code and the test cases. Let’s have a look at this, how it’s going to look like.

Assume we have an application. It’s a black box for us. It has been built through code, and we’re now setting up test cases. We set up the test cases from the perspective of a black box, and we’d better do because the application was built for business and it has not been built for the developers, right? So, we have come up with this set of test cases here. What we now can do is, first of all, we can dig deeper into the code, and what you see here is what we call the tree map. The different areas that you see here, represent areas of code and different granularity. It goes into classes, it goes into functions, it goes in varying levels of details. What we can do now is when we have a test case, when we let it run, we can switch on a tracer, and the tracer is going to find out which areas of code are touched as soon as we run a certain test step. So, we establish a direct link between code segments and test step, and through this we get immediate information which code has not been covered by our test suite, which we call the test gap.

Then, when we re-run the tests and it’s no matter whether this is a Tricentis Tosca test or it is a Selenium test. It could be any technology. When we re-run it and we get to results, given the fact that we have established the links before, we will be able to point directly on the code that is causing the issues. You know where the most effort of fixing an error is within today? More than 70% of the effort of fixing an error is with the analysis of the cause. Imagine the precision you get through this. Imagine the precision of pointing directly from the test sequence to the causing position in the code. What we get is what we call a test map, which visualizes the buggy code. This is how it looks in real life, a little bit more complex.

But the story is not over here. Now, when you submit changes, let’s see changed code here. Since we have established a link between the code and the tests before, we can immediately point to the test cases that need to be run now to secure the change. So, all of a sudden, you had your check-in and your continuous integration. Now, you have your continuous testing spot on to the change, and from there it goes again. You find the test gap. You have find which areas of code have not been covered by your test cases. You get the results, the test maps that directly point to the cause of errors, and this is how it goes around. So, this is super exciting, and from my perspective, this is the way how we’re going to see development and testing going to be linked.

We even may not call it testing in the future. We may call it engineering productivity, like Google Test Automation Conference says, but it’s going to be very close. That’s what I wanted to share with you today. Three things: platform. Continuous Testing Platform. It’s here to be built. We are the ones to build it better than anybody else. It’s going to have continuous load. It’s going to have continuous functional testing. It’s going to have continuous security testing, sooner or later. There are new technologies out there. Very, very exciting new technologies that will change the world. Some of them have an impact on testing and we are all ready here to serve you for that purpose. Some of them don’t. Still, they’re going to change our lives.

Last but not least, yes, development and testing is going to come together. It’s going to come together through the artifacts, and it’s going to come together from mindset, but it’s not going to be the same people doing it. Thank you.

Host:     Thank you, Wolfgang. Highly informative, thought-provoking, and visionary. I think we agree that that was an inspiring presentation. Do we have questions at this point? Does anybody want to get an answer from Wolfgang right now?

Audience One: I have a question. How is test gap compared to the risk-based testing? Because it’s the other way-

Wolfgang Platz: Test gap and risk-based testing. I love you for this question. How long did you give me time to answer it? This is an awesome question. Yes, let me repeat the question. So, the question was we guys, we are here and tell people about risk-based testing, right? And risk-based testing has the perspective of business and now we come up with code, right? So, how does that fit together, okay? Right? So, let me get back to this picture. This here. You need to build test cases from the perspective of business, and the way we try to do that, and I don’t know if you have been within a test case design training of Tricentis. We motivate the way we build test cases with an anticipation of how to cover code underneath, although it is from a business perspective, right?

At the very end, it is the code to be covered, but the perspective comes in from business. We’re going to have a look at this through risk, yes, because the risk is going to show us which areas we need to tackle with higher priorities than others, okay? That gives us the sequence of how to build the test cases. It gives us the sequence of how much to invest in certain areas. Then, we jump into case design. We build test cases from a business perspective, but we build it in order to do a very good job in covering the code. That’s what the methodology is about when you remember these things, right? But there needs to be proof because, as you know, since it is a black box, we have a lot of assumptions there. One of the assumptions is that your developers don’t screw you, which is not always true.

Now, we get the full transparency about which code is really covered, although we have tried to do our job as good as possible to cover it all. So, we shouldn’t get two maps like you’ve seen before where each and everything is great, right? Except for a few errors. So, we try to do the best from the risk and business perspective to make the code tree map comprehensively covered, but you should do it. You should show that it’s really the case, and the huge value comes into play when you, all of a sudden, able to point to the cause of an error. The direct connection between the test case and the cause of the error. That is a huge, huge benefit. Does that help?

Audience Two: I have a question about Amazon. How do they think about testing? What have they already done with testing? And what do you think they’ll do in the future and how could we, somehow, partner with them?

Wolfgang Platz: Amazon, first of all, is very interesting. You know that they make 90% of the revenue still with selling goods? 90% of the revenue, but by far, the fastest growth in Amazon is the platform, AWS, and almost all the profits come from there. So, strategically, Amazon is … They’re trying to get their publicity out there through drones and stuff, right? And selling goods, but in their hearts, they now even more care about the technology underneath. What we’re going to see in Amazon is that from providing just hardware, they’re evolve into providing services on top. More and more of the business is going be software. Specific software that helps you becoming productive when using their Cloud infrastructure.

This is going to be bigger, way bigger, than the platform and the hardware is ever going to be. And when it comes to testing now, we certainly want to make use of what we call the economy of scale. The economy of scale, which is here through the Cloud is opening opportunities for us we haven’t even been dreaming about, and the super prominent example that I’ve shown you today on our load testing is making use of the economy of scale. Without somebody providing infrastructure like an AWS, we would not have been able to go after that. So, the economy of scale is going to open new applications and new ways of test implementation that are immediately here to be nurtured, and we have a very close relationship to both of them. To AWS, on one hand, but also to Microsoft Azure, and you’re going to see us pushing into that direction very, very aggressively.

Host: Great questions so far.

Audience Three: Hi. Thank you and good morning. Nice presentation. I’m from Charles Schwab in Texas. So, I found it very interesting when you said software developers in test probably will not exist. So, are you saying we don’t really need testers with programming experience at all going forward? What’s your thinking?

Wolfgang Platz: That’s a very good question. What I say is that we are tool providers, we as tool vendors better provide tools that do not require software testers to be developers. It’s our obligation to come up with this. We have sold for that. That’s what I say because, have a look at the market out there. There’s supposed to be 600,000, about that, manual testers who do nothing else … It’s their profession. It’s going to be 650,000 testers complete, that do nothing but that for their profession. What do you think, we’re not going into a huge education program and make 650,000 manual testers developers? Is that going to happen? It’s not going to happen.

There are a few companies out there that can afford to have developers in test. There are some out there. I know some. The GAFAS, right? It’s not a hard thing for Google to just get 5,000 more developers. They just go out there, right? They pay any price, they get them. To say, it’s not your job to do testing. And what they can’t test, they’re going to shift into production, right? Tip. Which is a very cool strategy, by the way. This is the one aspect, and there’s Goldman Sachs, they do the same. Money is not an issue, okay? But most of the enterprises don’t have that kind of a luxury. They need to make do with capacity and infrastructure that is here, and 650,000 people, you cannot just convert them into developers. Actually, we have tried it out with one of our GSI partners, that was a great experiment.

We took 100 people, manual testers, and they trained, they tried to train them in their own infrastructure on Selenium. Three months. A hundred. And we went in and said, well, we think we can do better. So, we took another hundred, same skill level and we tried to train them on Tosca, and he said, the head of QA, you are the big mouth guy, I’m not going to give you three months, you’re going to do this in three weeks. After three months, where we had only three weeks, we counted, what do you think? How many of a hundred manual testers made it into Selenium scripters? Any guesses? 15, it’s not that bad. One five, 15. And Tosca? 85, who said it? Yeah, you got it. You heard it? 85%, that’s the huge difference, and that’s the reason why the certificates go up so rapidly.