image

Continuous testing

Wolfgang Platz: enterprise continuous testing in a flash

Wolfgang Platz was recently invited to speak at SKILUp Day: Continuous Testing, one of the many great events hosted by our friends at DevOps Institute. In this informal discussion, Wolfgang explored the key elements of Continuous Testing as well as how the role/practice of Continuous Testing is evolving.  

This event has passed, but you can catch Wolfgang’s talk here: 

Note: You can now access Wolfgang’s book, Enterprise Continuous Testing, on the Tricentis web site or Amazon. 

Here are some key takeaways from his talk…  

To drive efficiency in testing, it’s more important to have the right test cases than to worry about optimizing automation. 

There are two main reasons for testing: 1) to find errors 2) to ensure that the application is correct. Exploratory testing is the best way to find errors. Specification-based testing is the best way to ensure that the application is correct. 

There’s usually a gap between what was specified and what developers actually implement. Exploratory testing is the only way to find issues in functionality that was implemented but not specified. Specification-based testing is the only way to find issues in functionality that was specified but not implemented. Exploratory testing is usually the best way to find errors in functionality that was both specified and implemented. 

Most test reports (reporting the number of tests passed, failed, not executed) don’t give you enough information to make informed release decisions. Just knowing that 10% of tests failed doesn’t help. The test failures might be trivial, or they could be showstoppers. Additional investigation is required. 

For fast, accurate assessments of the risks associated with a release, we need a new currency in testing: business risk coverage. You can actually cover ~80% of your risks with ~20% of the effort if you really understand where your top risks lie and you strategically create tests to cover those risks. 

Based on intuition, most testers reach no more than ~50% risk coverage. Test case design helps you create tests that achieve your organization’s target level (e.g., 90-95%) of business risk coverage. Additionally, change impact analysis tells you exactly what tests you need to run in order to test each change and release with confidence. 

Before enterprises start the test automation journey, they usually have a few unit tests, some integration tests and system integration tests, and lots of end-to-end tests. The end-to-end tests are usually executed manually and come at an extremely high cost (with respect to slow execution as well as delays in finding and fixing errors). This needs to be reversed.  

The Agile test pyramid is totally flipped. It should have a solid base of unit tests, some integration and system integration tests, and a very small set of end-to-end tests. You want to automate as much as possible here, but a lot of end-to-end tests, some system integration tests, and a fraction of integration tests should still be manual.  

Beyond unit testing, be sure to consider two flavors of automation: UI and API. If the functionality you need to test is exposed via an API, try to do the majority of your testing at that level. API testing can start earlier in the sprint since you’re not waiting on the UI to be implemented. Plus, API tests are all-around faster: faster to create, faster to run, faster to maintain/update… 

Recent developments in AI/ML eliminate many of the challenges that traditionally plague UI test automation (specifically, that automated UI tests are too late and too brittle). With technologies like Tricentis Vision AI, you can start building UI test automation even before a UI exists. It doesn’t matter what technology is behind the application. If you can see it, you can automate it…with highly resilient tests. With this change, UI test automation will be poised to expand—cutting into both manual testing and API testing. It might even become the dominant form of automation that we see going forward.