Contributed Articles

DevOps Leaders vs. Laggards

Continuous Testing is a Key Differentiator

This article was originally published in

The Definitive Software Quality Metrics for Agile + DevOps, a new report by Forrester, reveals what software quality metrics and practices differentiate DevOps and Agile leaders from DevOps and Agile laggards.

The study, which was led by Forrester VP, Principal Analyst Diego Lo Giudice and commissioned by Tricentis, found that firms with the most successful Agile and DevOps initiatives do a number of things differently than their peers:

  • They are transforming software testing into Continuous Testing by adopting 5 core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.
  • They are almost twice as likely to consider automating the software quality process to be a “critical business differentiator”
  • They are significantly more likely to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).

However, only 26% of firms with active DevOps adoptions are currently practicing the 5 core Continuous Testing practices shown to separate leaders from laggards. And, as the following infographic highlights, many CXOs have a severely inflated sense of their firms’ DevOps and Continuous Testing maturity.

Different DevOps Quality Metrics

The study also identified the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers.  Forrester examined 94 quality metrics that were classified into four categories. Key findings include:

  • Understanding of business risk is the most important determining factor of DevOps and Agile maturity.
  • Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).
  • Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

To determine the “most important” metrics, the research surveyed 603 global enterprise participants responsible for their firms’ Agile and/or DevOps strategies. Participants were first asked a variety of questions to assess the maturity of their Agile/DevOps practices. Next, they were asked what quality metrics they actually used—then asked to rank the value of each metric they regularly measured. Results from the Agile/DevOps experts were later separated from, and compared to, those of the “other” respondents. The most important metrics were determined by analyzing the experts’ value rankings for the metrics that they actually used.

The Risk “Blind Spot”

Risk was a prevalent thread throughout the various parts of the report. Although risk-related metrics did not rank high in overall popularity, Agile and DevOps experts measure them significantly more frequently than the non-experts did. In fact, Agile and DevOps experts consistently ranked risk-related metrics among their top 3 most valuable metrics in different phases and categories.

Interestingly, most firms (80%) believe they deliver within acceptable business risk, but fewer than a quarter state that their QA and testing processes completely cover business risk. Only 15% of respondents say that their test suites reliably provide a good indication of business risk.

From Tricentis’ analysis of Global 2000 companies’ test case portfolios, we’ve found that most test suites actually cover about 40% of the organization’s business risks. However, these test suites have a 67% level of redundancy on average—meaning that over two-thirds of the tests don’t actually increase their business risk coverage. This might be one cause of the disconnect: people assume that more tests means more business risk coverage, but that’s often not the case.

Another likely cause is that most test results focus on pass/fail status, which doesn’t provide the required insight into whether the release has an acceptable level of risk. As companies move towards Continuous Delivery and automated release processes, real-time insight into business risk becomes critical for promoting acceptable releases as rapidly as possible while stopping potentially-damaging ones from proceeding down the release pipeline.

Details and Additional Findings

If you want to learn more about how Forrester arrived at these findings—and see how the 94 quality metrics ranked—read the complete Definitive Software Quality Metrics for Agile+DevOps report.