INDUSTRY RESEARCH REPORT
Testing Expectations vs. Reality
How different development & delivery methods affect tester scope, status, job satisfaction and more
Tricentis & TechWell surveyed testers and testing leaders across the globe to gather insights about the evolving role of the tester against the backdrop of agile adoption, DevOps transformation and more.
The Role of Testing in a DevOps Transformation
We asked respondents how far along their organizations were with DevOps transformation and used this information to assess how DevOps maturity levels affect testers’ experiences. The standout finding: Among organizations that have completed DevOps transformation, 87% say testing is critical to the success of their DevOps initiatives. These respondents were also more than three times more likely to say their organizations prioritize testing than any other group — and tended to be significantly more advanced in terms of test automation.
Advanced Test Automation is Critical, but Tough to Achieve
Higher test automation levels correlate with DevOps maturity. But achieving a high rate of automation is proving to be more difficult than anticipated. In our survey of a similar audience in 2018, 44% of respondents said they expected to automate more than 50% of their test efforts in the next 12 months. When we asked this year, only 14% had achieved that goal. In fact, most respondents are currently automating less than 20% of their total testing effort.
2018 ExpectationAutomate 50%+ of test suite in 2019
2019 RealityLess than 20% of tests are automated
- DevOps Complete
- DevOps in Progress
- DevOps Not Started
How Does Job Satisfaction Compare?
We wanted to understand how Agile, DevOps and test automation rates affect job satisfaction. Testers in Agile organizations are almost four times more likely to be satisfied in their jobs than testers in waterfall organizations. Across DevOps maturity levels, job satisfaction was lowest in organizations currently undergoing DevOps transformation, and higher at either end of the maturity curve. In mature DevOps organizations, managers are significantly more likely to be satisfied or very with their team’s work. DevOps maturity also correlates with a significant increase in the perceived importance of testing and the level of respect testers receive. Hover over the chart below for percentages.
Devops In Progress
Devops Not Started
Devops Not A Priority
- 10% Complete
- 52% In Progress
- 19% Planned, Not Started
- 20% Not Planned
- 11% Complete
- 41% In Progress
- 23% Planned, Not Started
- 25% Not Planned
- 9% Complete
- 59% In Progress
- 20% Planned, Not Started
- 12% Not Planned