Tricentis Virtual Summit is back for 2022

This fully online and free-to-attend conference is key to deliver innovation with confidence.

Register now
Your transformation toolkit

Advance your enterprise testing strategy with our transformation toolkit.

Learn more

Agile testing

CXOs are Overestimating DevOps, Agile and Continuous Testing Maturity

Tricentis and Forrester recently partnered to research what separates DevOps + Agile leaders from DevOps + Agile laggards. After analyzing data collected from 603 global enterprise respondents responsible for their firms’ Agile and/or DevOps strategies, we learned that the most mature in Agile + DevOps do five key things differently:

  1. Allocate proper testing budgets and focus on upgrading their testing skills
  2. Implement Continuous Testing to meet the demands of release frequency and support Continuous Delivery
  3. Include testers as part of their integrated delivery teams
  4. Automate end-to-end functional testing
  5. Shift-left testing to earlier in the development life cycle

This was not terribly surprising. Also not surprising was the fact that firms who follow these five best practices experience testing as a way to gain speed and efficiency, rather than a bottleneck to delivering faster.

What caught us off guard, however, was CXOs’ highly inflated estimation of their Continuous Testing, DevOps, and Agile maturity.

According to the report, only 26% of non-CXOs reported that their organization was following all 5 of these best practices—but 57% of CXOs believed that they were:

While most Agile and/or DevOps firms have implemented some of these five core testing practices, few have actually implemented all of them — only about a quarter of respondents say they somewhat or completely agree with all five best practice statements.

This is an important distinction. While many CXOs and high-level decision makers believe they are ahead of the pack, our research shows that only the small fraction of firms that follow these best practices have more advanced Agile and DevOps practices compared to their peers — and their behaviors and attitudes support that assertion

Moreover, CXOs also reported much higher levels of Agile and DevOps maturity across the board than non-CXOs did.

The good news for development, testing, and DevOps professionals is that CXOs are your champions: they believe you’re doing a great job! The bad news is that, at some point, you need to tell them about the gaps that currently exist in the process. You also need to collect and share some of your own data to make the case that actually adopting these 5 proven best practices will help you increase delivery speed while reducing business risk.

Read the Complete Report

This was just one of the many interesting findings from the research project, which was led by Forrester Vice President and Principal Analyst Diego Lo Giudice.

As a high-level summary, the research uncovered the following:

  • Understanding of business risk is the most important determining factor of DevOps and Agile maturity.
  • Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).
  • Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

Detailed results and analysis are available in the complete report: Forrester Research: What Separates DevOps + Agile Leaders from Laggards?

Report Methodology

In this study, Forrester conducted an online survey of 603 enterprise organizations in North America, Europe, and Asia Pacific to evaluate current software testing practices and metrics tracked during survey development. Survey participants included decision makers and individual contributors responsible for their organizations’ Agile and/or DevOps development strategies. Questions provided to the participants asked about their firms’ attitudes toward software development automation, risk management, and testing practices, as well as the metrics they track and value in the software development life cycle.