Software testing

Building a Test Automation Strategy: How Should We Be Automating? | Whiteboard Friday

Welcome to episode 2 of our new Whiteboard Friday series, Building a Test Automation Strategy. In last week’s episode, we explored why many teams are incorporating automated testing into their QA strategies. We also discussed the benefits of test automation.

This week we cover what areas your test automation strategy should be focused on by reviewing two test automation pyramids.

Watch the Video

In this episode you will learn:

  • Why you should focus on unit test automation more
  • Where you will see the most ROI in your test automation
  • Two different approaches to test automation

Here’s the full transcript:

Hey, everyone. Ryan Yackel here with Tricentis, and welcome back to another edition of Whiteboard Friday. And we’re continuing our series on building out a test automation strategy in your organization or even in your small team that’s starting to get started with test automation. Last time, we talked about, was really around why this focus on test automation. You may be hearing from other people saying, “Yeah, you need to get more test automation.” Well, why is that? What are the things they’re driving at? So we talked about that, but today what we’re talking about is really getting into what areas of focus should our test automation then be laser focused on.

And what I have behind me is actually two pyramids. One is an anti-pattern pyramid and one is the correct ideal pyramid we want to get our test automation to. So I want to talk through it as a simplistic way to understand how to actually go about thinking about where you should be focusing your test automation. This Test Automation Pyramid, just so you understand, it’s not something I did. It was actually from one of the thought leaders in the space called Mike Cohn. A few years ago, he built this out. But what I want to do is talk first about maybe what you’re probably doing which is wrong, and then get into more of an ideal state of where you should focus.

And the way this is really set up is that, a lot of times, what we do is we focus a lot of our effort around UI test automation. And the reason why we do that is because, traditionally, if you get into organizations that had done a lot of manual or exploratory testing, especially if they’re testing something like a UI layer, what occurs is that the simplest path that they think, the path of least resistance to get to test automation, is then they go into automated UI testing, right? So that means you’re saying, “Okay. I’m testing on a Chrome browser, and what I want to do is actually scale up some Selenium web browser testing to go against that.” Right? So what happens is, you actually put a lot of your effort into the UI test automation, right? And then what you’re doing, though, is you’re neglecting a lot of the testing that goes on at the integration layer, the API layer. And then the developers, for the most part, when they’re doing their unit testing, they’re not really doing any automated user or automated unit testing here.

So what happens is, you have all this testing focused at the very top that’s really UI focused, and the reason why this is bad is because, if I shift gears here and we talk about kind of the dichotomy that’s going on here, is that you’re really going to see more ROI down below at the lower level test than you will at one of the tests that take a lot of time and a lot of effort, right? So if you think about UI test automation, it takes a long time to run these tests. So what’s going on is that you’re having to provision environments. You’re having to use different browsers. You’re having to use different data. You’re having to use object models as you do your testing. You’re having, maybe, if the test changes, right, or the screen changes, it becomes brittle. So you have all this testing focused here, but you have more time and effort that goes into it, when you can really get…a lot of your bang for your buck is down here as you flip the pyramid and really focus on unit test, API test, and then get into the more GUI test automation, and then you get into exploratory test automation.

The reason why the unit testing, or why we should focus more on the unit test automation is because, one is, you’re getting continuous feedback around what actually is going on in the build. So if you think about a transformation that’s going on in your organization, people are talking about continuous integration that leads into continuous delivery, right? Well, what they’re doing is saying, “Every time I do a commit, right, against the code in my branch, I want to run these unit tests to make sure everything is good.” Right? So down below here, you have a lot of developers starting to do more progressive testing, like test-driven development. It’s where they can get immediate feedback on the tests that have occurred, right? So getting that layer of feedback at the build level is very important so that you can make sure your application and your static code, or doing static code analysis to make sure your code is in a good spot, right?

Then as you move up here, the next part is really getting around your API testing, right? Why do you want to focus on API testing? Well, because that’s the integration layer between, maybe, microservices you have in your organization. It’s the API that, if you do a best practice, that API has a UI component to it, but the API component shouldn’t change, for the most part, when you go build to build, right? So the UI part may change. The page may change. The buttons may change, right, on that page, but the API layer of that testing, or that application, that’s not going to change, right? So you have more less brittle tests when you’re testing the API layer before you get to the UI tests, right?

And then, of course, at the automated UI layer, then you can incorporate things like figuring out, “How can my UI test be optimized where I’m not running thousands of these tests, maybe I’m running hundreds of these tests.” Right? So you can say, you know, “For our end-to-end automation, there’s some critical paths that we have to do. We don’t have to go off and do a full UI regression script every single time we do our testing. Rather, we can focus more on getting immediate feedback from the unit test, focus on the API,” and then move into the UI layer for your test automation.

And then, of course, the last part is the fact that the manual exploratory testing really doesn’t change, in the sense of, if that’s going to be done in your organization, right? So when people are talking about, “I’m going to move to 100% test automation,” right, or move toward 98%, or whatever it is, right, that’s fine, to have a goal to get to a certain level of automation to where you’re not having to do a lot of manual tests and manual progress, right? However, there’s still some effort that needs to be put into place around your manual exploratory testing so that you can do more end user validation testing and find more critical bugs that are going to be up at the top here.

The last thing I’ll say before we close out is just that this pyramid is not something that somebody can just look at and go, “Okay. If I do this exactly right, I’m going to have a full-on test automation strategy that’s perfect.” Right? In your organization, your actual pyramid may look more like a rectangle. You may have an equal number of these tests that are going on in that organization. You may have, even, for example, a lot of UI tests and also a lot of unit tests, but no API layer to go on there. So it’s not something that’s totally prescriptive. It’s more of a descriptive way to look at how your automation and where it’s focused to really optimize where your tests should be going, and where you should be focusing your attention on.

Then the last thing I’ll say is that this transformation from really focusing on UI testing to more things like API and unit level testing, that’s kind of a transformational change you have to make in your organization, right? If you have lots of tests that are going on with Selenium, right, all your Selenium automation engineers may be needing to say, “You know what? We need to actually look and optimize our tests so that we’re not running all these tests, right, and then focus more on manual testing.” You’re going to have to get your developers to say, “You need to start doing test-driven development at the unit level.” Right? So it’s not just something you can just flip and say, “Okay. Now we’re going to start just doing this level of test automation at these certain levels.” You’re going to have to convince your organization to actually start down that path, and it starts with a full development change and every single person who’s responsible for this type of testing that goes on in your organization.

So again, just to recap, a lot of focus is put on the UI layer. We need to have automated UI acceptance testing going on, but we need to transition that more into a focus on the lower level of the code so we can get better feedback, get more stable builds, and then build up into more, or less UI tests, and focus more on exploratory manual tests so we can find those critical bugs.

So that’s been a quick view on kind of where we should focus our test automation. Be sure to tune in next time for another Whiteboard Friday as we continually talk about some of the key things you need to think about for your test automation strategy going forward. We’ll see you next time.