INDUSTRY RESEARCH REPORT

Testing Expectations vs. Reality

How different development & delivery methods affect tester scope, status, job satisfaction and more

Tricentis & TechWell surveyed testers and testing leaders across the globe to gather insights about the evolving role of the tester against the backdrop of agile adoption, DevOps transformation and more.

DevOps Survey Illustration

The Role of Testing in a DevOps Transformation

We asked respondents how far along their organizations were with DevOps transformation and used this information to assess how DevOps maturity levels affect testers’ experiences. The standout finding: Among organizations that have completed DevOps transformation, 87% say testing is critical to the success of their DevOps initiatives. These respondents were also more than three times more likely to say their organizations prioritize testing than any other group — and tended to be significantly more advanced in terms of test automation.

Advanced Test Automation is Critical, but Tough to Achieve

​ ​ ​

Higher test automation levels correlate with DevOps maturity. But achieving a high rate of automation is proving to be more difficult than anticipated. In our survey of a similar audience in 2018, 44% of respondents said they expected to automate more than 50% of their test efforts in the next 12 months. When we asked this year, only 14% had achieved that goal. In fact, most respondents are currently automating less than 20% of their total testing effort.

2018 Expectation

Automate 50%+ of test suite in 2019

2019 Reality

Less than 20% of tests are automated

How Do Testers Spend Their Time?

We asked testers where they spend the most time, then segmented responses according to DevOps maturity level. Respondents reported that testing strategy was more important to the organization than any other task. But when testers ranked how they are actually spending their time, testing strategy only made the top five in organizations that are undergoing or have completed DevOps transformation. In organizations where DevOps isn’t a priority, testers spend the most time on implementation and documentation.
Hover over the categories below to explore.

  • DevOps Complete
  • DevOps in Progress
  • DevOps Not Started
Time Spent On Tasks
Implementing Tests Icon
Testing Strategy Icon
Documenting Issues Icon
Designing Tests Icon
Planning Tests Icon
Maintaining Automated Tests Icon

How Does Job Satisfaction Compare?

We wanted to understand how Agile, DevOps and test automation rates affect job satisfaction. Testers in Agile organizations are almost four times more likely to be satisfied in their jobs than testers in waterfall organizations. Across DevOps maturity levels, job satisfaction was lowest in organizations currently undergoing DevOps transformation, and higher at either end of the maturity curve. In mature DevOps organizations, managers are significantly more likely to be satisfied or very with their team’s work. DevOps maturity also correlates with a significant increase in the perceived importance of testing and the level of respect testers receive. Hover over the chart below for percentages.

Manager Approval

  • Devops Complete
    66 %
  • Devops In Progress
    52 %
  • Devops Not Started
    47 %
  • Devops Not A Priority
    23 %

Tester Satisfaction Levels Compared to Primary Lifecycle Model; More are Satisfied with Agile & DevOps than Waterfall

Regional Differences at a Glance

Asia-Pacific stands out as the region in which testing is prioritized most as part of DevOps transformation, and in which testers are most respected, followed by the Americas. Organizations in Europe and the Middle East had the greatest maturity divide, with both the highest percent of complete DevOps transformation and the highest percent reporting they haven’t started yet. The standout difference in the Americas was the rate at which respondents reported practicing Agile: 85%, compared with 76% in Asia-Pacific and just 30% in Europe and the Middle East. Hover over each region in the map below for more details, and download the report for the full analysis across regions.

map
map
devops transformation status
  • 10% Complete
  • 52% In Progress
  • 19% Planned, Not Started
  • 20% Not Planned
Most likely to be practicing Agile development & testing
devops transformation status
  • 11% Complete
  • 41% In Progress
  • 23% Planned, Not Started
  • 25% Not Planned
Greatest DevOps maturity divide
devops transformation status
  • 9% Complete
  • 59% In Progress
  • 20% Planned, Not Started
  • 12% Not Planned
Highest automation rates

Get the Full Report

Related Resources