What is Continuous Testing?
Continuous Testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release candidate as rapidly as possible. It evolves and extends test automation to address the increased complexity and pace of modern application development and delivery.
Everyone accepts that Agile and DevOps are all about change—transforming Dev/Ops people, processes, and technologies to deliver innovative software as rapidly as possible. In spite of all this change, one thing tends to remain the same: the software testing process. One recent study reports that 70% of organizations have adopted Agile, but only 30% automate testing. A separate study found that while Agile adoption is now near 88%, only 26% of Agile organizations have broadly adopted test automation.
In other words, testing processes remain stuck in the past even as organizations invest considerable time and effort into transforming their development processes to meet today’s and tomorrow’s business demands. Most legacy testing tools and processes are unfit for the type of Continuous Testing that Agile and DevOps require due to:
Inability to “Shift Left”
Tests usually cannot be implemented until late in each sprint—when the UI and dependent components such as back-end APIs are finally completed and available for testing.
Slow Execution Time
Tests are time-consuming to execute, so it is not practical to run the complete regression test suite on each build. This means the team lacks instant feedback on whether their changes impact the existing user experience.
UI tests require considerable rework to keep pace with the frequent changes typical of accelerated release processes. This results in slow, burdensome maintenance and/or causes automation efforts to be abandoned.
Test Environment Instability
Test environment instability (inaccessible dependencies, test data issues, etc.) commonly cause timeouts, incomplete tests, false positives, and/or inaccurate results — preventing you from delivering the fast quality feedback Agile and DevOps require.
To Enable Continuous Testing, Automation Rates Must Exceed 85%
- Align testing with business risk to optimize test execution, implement automated quality gates, and provide critical insight for release decisions.
- Minimize manual testing and shift it to session-based exploratory testing.
- Shift testing efforts to the API layer when feasible.
- Use test data management and service virtualization to enable continuous execution of realistic end-to-end tests—without false positives and timeouts.
- Integrate functional testing into CI/CD so it’s a seamless part of the delivery pipeline.
What’s Required for Continuous Testing?
Continuous Testing & Testing Automation
Industry research indicates that average test automation levels have been hovering around 20% for years. Today, changes across the industry are demanding more from testing at the same time that they’re making test automation more difficult to achieve:
Application architectures are increasingly more distributed and complex, embracing cloud, APIs, microservices, etc. and combining virtually endless combinations of different protocols and technologies within a single business transaction.
Thanks to Agile, DevOps, and Continuous Delivery, many applications are now released anywhere from every 2 weeks to thousands of time a day. In response, the time available for test design, maintenance, and especially execution decreases dramatically.
Now that software is the primary interface to the business, an application failure is a business failure—and even a seemingly minor glitch can have severe repercussions if it impacts the user experience. As a result, application-related risks have become a primary concern for even non-technical business leaders.
As software becomes the key to creating a competitive advantage across all markets, enterprises no longer enjoy the luxury of selecting either ‘speed’ or ‘quality’ when delivering software. Both are critical. Now that agile practices have matured and DevOps initiatives have entered the corporate agenda, Continuous Integration (CI), Continuous Testing and Continuous Delivery (CD) have emerged as key catalysts for enabling quality at speed. Of the three, Continuous Testing is by far the most challenging.
While Continuous Integration is primarily a tool-driven activity and Continuous Delivery is a tool- and team-driven activity, Continuous Testing involves tools, teams, individuals, and services.
Building and integrating code changes is certainly important. However, if the automated delivery process cannot identify how changes impact business risk or disrupt the end-user experience, then the increased frequency and speed of Continuous Integration and Continuous Delivery could become more of a liability than an asset.
Executed correctly, Continuous Testing serves as the centerpiece of the agile downstream process – executing automated tests as part of the software delivery pipeline in order to provide risk-based feedback as rapidly as possible. Mastering Continuous Testing is essential for controlling business risk given the increased complexity and pace of modern application delivery.
Beyond Continuous Testing with Artificial Intelligence
Continuous Testing: Is It All Just a Bunch of Hype?
Forrester Study: Continuous Testing Separates DevOps Leaders from Laggards
Continuous Testing Insights from the Latest DevTest Surveys Blog
Continuous Testing vs. Test Automation: What’s the Difference? Recorded Webinar
Continuous Testing in 2019: What You Missed (Already)
What is Continuous Testing? A Crash Course for Software Testers Blog
Continuous Testing Maturity Model Fact Sheet
Continuous Testing vs. Test Automation Whitepaper
Continuous Testing and DevOps: A Chat with Ingo Philipp and Eran Kinsbruner
Continuous Testing: Transforming Testing for Agile and DevOps
State of DevOps 2018: Continuous Testing is Required for DevOps Success
Continuous Testing Insights from Forrester’s Latest Wave
Get details on how Tricentis helps software testers adopt and advance Continuous Testing through risk-based testing, model-based test automation, service virtualization, test data management, and more.