CONTINUOUS TESTING

What is Continuous Testing?

Continuous Testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release candidate as rapidly as possible. It evolves and extends test automation to address the increased complexity and pace of modern application development and delivery.

Continuous Testing Definition: 14 Key Points

  1. Continuous Testing’s primary goal is assessing business risk coverage
  2. Continuous Testing provides instant insight on whether a release candidate is too risky to proceed through the delivery pipeline
  3. Continuous Testing establishes a safety net that helps the team protect the user experience in accelerated development processes and avoid software failure headlines
  4. Continuous Testing expects testing to be embedded within the development process, not tacked on at the end
  5. Continuous Testing is seamlessly integrated into the software delivery pipeline and DevOps toolchain
  6. Continuous Testing expects a stable test environment with valid test data to be available for each and every test run
  7. Continuous Testing embraces everything from “shift left” (unit, component, coverage…) to “shift right” (monitoring/ APM, Testing in Production)
  8. Continuous Testing involves executing the right set of tests at the right stage of the delivery pipeline—without creating a bottleneck
  9. Continuous Testing delivers actionable feedback appropriate for each stage of the delivery pipeline
  10. Continuous Testing evaluates each layer of a modern architecture at the appropriate stage of the delivery pipeline
  11. Continuous Testing includes end-to-end tests that realistically assess the end-user experience across all associated technologies (front-end and back-end)
  12. Continuous Testing’s tests must be broad enough to detect when an application change inadvertently impacts functionality that users have come to rely on
  13. Continuous Testing reduces false positives by prioritizing robust, flexible modern test frameworks over brittle scripts
  14. Continuous Testing involves continuously reviewing and optimizing the test suite to eliminate redundancy and maximize business risk coverage

Continuous Testing Is Change

Everyone accepts that Agile and DevOps are all about change—transforming Dev/Ops people, processes, and technologies to deliver innovative software as rapidly as possible. In spite of all this change, one thing tends to remain the same: the software testing process. One recent study reports that 70% of organizations have adopted Agile, but only 30% automate testing. A separate study found that while Agile adoption is now near 88%, only 26% of Agile organizations have broadly adopted test automation.

 

In other words, testing processes remain stuck in the past even as organizations invest considerable time and effort into transforming their development processes to meet today’s and tomorrow’s business demands. Most legacy software testing tools and processes are unfit for the type of Continuous Testing that Agile and DevOps require due to:

Inability to “Shift Left”

Tests usually cannot be implemented until late in each sprint—when the UI and dependent components such as back-end APIs are finally completed and available for testing.

Slow Execution Time

Tests are time-consuming to execute, so it is not practical to run the complete regression test suite on each build. This means the team lacks instant feedback on whether their changes impact the existing user experience.

High Maintenance

UI tests require considerable rework to keep pace with the frequent changes typical of accelerated release processes. This results in slow, burdensome test maintenance and/or causes test automation efforts to be abandoned.

Test Environment Instability

Test environment instability (inaccessible dependencies, test data issues, etc.) commonly cause timeouts, incomplete tests, false positives, and/or inaccurate results — preventing you from delivering the fast quality feedback Agile and DevOps require.

Enterprise Continuous Testing: Transforming Testing for Agile and DevOps

 

Even with the most extreme test automation, the “test everything” approach is not feasible—or necessary. If you rethink your approach to automated software testing, you can get a thorough assessment of a release candidate’s business risk with much less testing than you’re probably performing today.

Enterprise Continuous Testing: Transforming Testing for Agile and DevOps introduces a Continuous Testing strategy that helps enterprises accelerate and prioritize testing to meet the needs of fast-paced Agile and DevOps initiatives. Software testing has traditionally been the enemy of speed and innovation—a slow, costly process that delays releases while delivering questionable business value. This new strategy helps you test smarter, so testing provides rapid insight into what matters most to the business

To learn how, read the 132-page book by Tricentis Founder Wolfgang Platz.

Get the book Get the book

Continuous Testing & Testing Automation

Industry research indicates that average test automation levels have been hovering around 20% for years. Today, changes across the industry are demanding more from testing at the same time that they’re making test automation more difficult to achieve:

Application architectures are increasingly more distributed and complex, embracing cloud, APIs, microservices, etc. and combining virtually endless combinations of different protocols and technologies within a single business transaction.

Thanks to Agile, DevOps, and Continuous Delivery, many applications are now released anywhere from every 2 weeks to thousands of time a day. In response, the time available for test design, maintenance, and especially execution decreases dramatically.

Now that software is the primary interface to the business, an application failure is a business failure—and even a seemingly minor glitch can have severe repercussions if it impacts the user experience. As a result, application-related risks have become a primary concern for even non-technical business leaders.

Continuous Testing, Continuous Integration, Continuous Delivery & DevOps

As software becomes the key to creating a competitive advantage across all markets, enterprises no longer enjoy the luxury of selecting either ‘speed’ or ‘quality’ when delivering software. Both are critical. Now that agile practices have matured and DevOps initiatives have entered the corporate agenda, Continuous Integration (CI), Continuous Testing and Continuous Delivery (CD) have emerged as key catalysts for enabling quality at speed. Of the three, Continuous Testing is by far the most challenging.

 

While Continuous Integration is primarily a tool-driven activity and Continuous Delivery is a tool- and team-driven activity, Continuous Testing involves tools, teams, individuals, and services.

 

Building and integrating code changes is certainly important. However, if the automated delivery process cannot identify how changes impact business risk or disrupt the end-user experience, then the increased frequency and speed of Continuous Integration and Continuous Delivery could become more of a liability than an asset.

 

Executed correctly, Continuous Testing serves as the centerpiece of the agile downstream process – executing automated tests as part of the software delivery pipeline in order to provide risk-based feedback as rapidly as possible. Mastering Continuous Testing is essential for controlling business risk given the increased complexity and pace of modern application delivery.

Get the Continuous Testing Reference Card Get the Continuous Testing Reference Card

Beyond Continuous Testing with Artificial Intelligence

We’ve already undergone quite a journey to arrive at Continuous Testing. Nevertheless, when we look into the future, it’s clear that even Continuous Testing will not be sufficient. We’re fast approaching a time when Continuous Testing will be unable to keep pace with shrinking delivery cycles times, increasing technical complexity, and accelerating rates of change.

To ensure quality in an era where software will be processing an unimaginable number of data points in real time—for example, both figuratively driving the internet of things and literally driving “self-driving” cars—we need all the help we can get. Beyond Continuous Testing, we need “Digital Testing” to achieve further acceleration and meet the quality needs of a future driven by IoT, robotics, and quantum computing. AI, imitating intelligent human behavior for machine learning and predictive analytics, can help us get there.

Read the blog Read the blog

Continuous Testing Resources

5 core testing practices that separate successful DevOps from laggards

Read more

Continuous Testing, shifting left, and test automation

Read more

Beating 3 challenges to enterprise Continuous Testing

Read more

A practical guide to continuous performance testing

Read more

Forrester Research: What separates DevOps + Agile leaders from laggards?

Learn more

Risk-based Testing

Design the most efficient set of tests and test data required to achieve the maximum business risk coverage, then deliver test results that report on business risks. This risk-based insight is essential for establishing automated quality gates and making smart release decisions.

Read more Read more

Exploratory Testing

Expose critical defects early in agile sprints as well as trigger formal reviews at appropriate points in the Continuous Delivery pipeline. Sessions that expose risks can be captured for defect reproduction/documentation; they can also be integrated into the automated regression suite.

Read more Read more

CI/CD Integrations

Integrate functional testing into industry-leading software delivery pipeline and DevOps tools. We provide out-of-the box integration with popular tools such as Jenkins, Jira, GitHub, Docker, Puppet, Visual Studio, etc.

Read more Read more

Service Virtualization

Execute tests continuously, without environment instability and test data availability issues triggering false positives or compromising the trustworthiness of results. Testing can proceed without waiting on access to 3rd-party services, not-yet-implemented services, fresh test data, etc.

Read more Read more

Model-based Test Automation

Ensure that rapidly-evolving applications don’t result in overwhelming false positives and burdensome test maintenance by creating API tests and UI tests that are resistant to insignificant changes and easy to update for intentional changes. Extensive technology support helps you test realistic end-user scenarios across modern enterprise architectures.

Read more Read more

Test Data Management

Stop waiting weeks for test data and worrying about test data expiring after a single test run. Test data management helps you design, generate, and centrally manage data appropriate for each and every test run.

Read more Read more

What Next?

Get details on how Tricentis helps software testers adopt and advance Continuous Testing through risk-based testing, model-based test automation, service virtualization, test data management, and more.

Explore the platform Explore the platform

Get started with Tricentis

Get started

Arrow