Your transformation toolkit

Advance your enterprise testing strategy with our transformation toolkit.

Learn more

Agile testing

Continuous Testing: New Research on Why It’s Essential for Agile + DevOps Success

Want further evidence that Continuous Testing has evolved from buzzword to business imperative? New Forrester research—What Separates DevOps + Agile Leaders from Laggards?—found that Continuous Testing, done properly, is a key differentiator between DevOps + Agile leaders and DevOps + Agile laggards.  Here are the key findings and recommendations from that research project.

Key Continuous Testing Findings

1. Five core Continuous Testing practices separate successful DevOps + Agile leaders from laggards

Firms that are more mature in Agile + DevOps do five key things differently:

  1. Allocate proper testing budgets and focus on upgrading their testing skills
  2. Implement Continuous Testing to meet the demands of release frequency and support Continuous Delivery
  3. Include testers as part of their integrated delivery teams
  4. Automate end-to-end functional testing
  5. Shift-left testing to earlier in the development life cycle

For firms that follow these five best practices, testing is a way to gain speed and efficiency, rather than a bottleneck to delivering faster.

While most Agile and/or DevOps firms have implemented some of these five core testing practices, few have actually implemented all of them — only about a quarter of respondents say they somewhat or completely agree with all five best practice statements. This is an important distinction. While many CXOs and high-level decision makers believe they are ahead of the pack, our research shows that only the small fraction of firms that follow these best practices have more advanced Agile and DevOps practices compared to their peers — and their behaviors and attitudes support that assertion.

2. Business risk must be accurately and continuously managed

Automating software delivery without being able to accurately measure software quality is dangerous. Firms must understand the business risk — the chance of a negative user experience — that each software application carries when deciding whether to release. Unfortunately, most firms today admit they have gaps in their ability to accurately assess and manage business risk in testing.

3. Leaders ruthlessly drive end-to-end test automation

With a better indication of risk from their testing metrics, DevOps leaders can relentlessly pursue automation to develop with quality at speed. They do this by prioritizing end-to-end automated business process test cases. They automate test design and test execution, and they orchestrate the automation of the overall dev-test-deploy process.

Key Continuous Testing Recommendations

Realistically assess and up-level your Continuous Testing practices for Agile + DevOps. While many CXOs believe their firms are ahead of the curve on DevOps, our research shows that only about a quarter of firms are following testing best practices. Make sure your firm is implementing Continuous Testing as part of its DevOps strategy.

1. Make business risk the driver for your metrics program

Most firms acknowledge gaps in their ability to manage business risk with the right metrics but are nevertheless optimistic on risk coverage. This is dangerous for extreme automation — and exacerbated as delivery velocity and volume increase. First, you must clearly define risk. Second, you must continuously measure your exposure versus that risk. Next, use that understanding to promote releases. By focusing test automation efforts on processes and transactions that carry higher business risk, you reduce the chance that critical defects slip into production.

2. Ruthlessly automate your testing and QA processes, end to end

If your goal is to deliver high-quality software faster (and it should be), then you need to automate your software development pipeline. More mature Agile + DevOps firms understand that automation is critical to driving release velocity, and they are reaping the benefits. Automating end-to-end testing is a key step in implementing Continuous Testing and a top priority for Agile + DevOps leaders today.

3. Prioritize test execution predicated around business risk

Understanding which tests deliver the greatest risk coverage is a significant advantage when speed matters. The ability to prioritize test execution depends on the implementation of a model that is collaboratively developed among the development, testing, and business stakeholders.

Continuous Testing: Critical for “Quality at Speed”

Delivering high-quality software fast is no longer a luxury — it is a prerequisite for digital success and mandatory for customer loyalty and acquisition. We are in an era where customers are empowered to choose and switch to the company that can provide them the best experience quickly and at minimal cost. Organizations that cannot keep up with their customers’ expectations and adapt to new business paradigms will find themselves losing customers and business to competitors that can.

How can firms deliver with quality at speed? Implementing Agile + DevOps best practices and building in quality is the first step — and this is not trivial. Automation is the lever that development teams pull to go faster. However, if not properly managed, automation can jeopardize quality and increase risk. To ensure that quality issues and unacceptable risk do not negate the velocity benefits of automation, firms must ensure they are tracking the right metrics to measure quality throughout development testing-deployment.

Additional Continuous Testing + Agile and DevOps Insights

For more research findings, read the complete Forrester Research: What Separates DevOps + Agile Leaders from Laggards? report.