INDUSTRY RESEARCH REPORT

Testing Expectations vs. Reality

How different development & delivery methods affect tester scope, status, job satisfaction and more

Tricentis & TechWell surveyed testers and testing leaders across the globe to gather insights about the evolving role of the tester against the backdrop of agile adoption, DevOps transformation and more.

Get the Report

Watch the Webinar

The Role of Testing in a DevOps Transformation

We asked respondents how far along their organizations were with DevOps transformation and used this information to assess how DevOps maturity levels affect testers’ experiences. The standout finding: Among organizations that have completed DevOps transformation, 87% say testing is critical to the success of their DevOps initiatives. These respondents were also more than three times more likely to say their organizations prioritize testing than any other group — and tended to be significantly more advanced in terms of test automation.

Advanced Test Automation is Critical, but Tough to Achieve

Higher test automation levels correlate with DevOps maturity. But achieving a high rate of automation is proving to be more difficult than anticipated. In our survey of a similar audience in 2018, 44% of respondents said they expected to automate more than 50% of their test efforts in the next 12 months. When we asked this year, only 14% had achieved that goal. In fact, most respondents are currently automating less than 20% of their total testing effort.

2018 Expectation

Automate 50%+ of test suite in 2019

2019 Reality

Less than 20% of tests are automated

How Do Testers Spend Their Time?

We asked testers where they spend the most time, then segmented responses according to DevOps maturity level. Respondents reported that testing strategy was more important to the organization than any other task. But when testers ranked how they are actually spending their time, testing strategy only made the top five in organizations that are undergoing or have completed DevOps transformation. In organizations where DevOps isn’t a priority, testers spend the most time on implementation and documentation.
Hover over the categories below to explore.

DevOps Complete
DevOps in Progress
DevOps Not Started

Time Spent On Tasks

Implementing Tests

Testing Strategy

Documenting Issues

Designing Tests

Planning Tests

Maintaining Automated Tests

How Does Job Satisfaction Compare?

We wanted to understand how Agile, DevOps and test automation rates affect job satisfaction. Testers in Agile organizations are almost four times more likely to be satisfied in their jobs than testers in waterfall organizations. Across DevOps maturity levels, job satisfaction was lowest in organizations currently undergoing DevOps transformation, and higher at either end of the maturity curve. In mature DevOps organizations, managers are significantly more likely to be satisfied or very with their team’s work. DevOps maturity also correlates with a significant increase in the perceived importance of testing and the level of respect testers receive. Hover over the chart below for percentages.

Manager Approval

Devops Complete

66 %
Devops In Progress

52 %
Devops Not Started

47 %
Devops Not A Priority

23 %

Tester Satisfaction Levels Compared to Primary Lifecycle Model; More are Satisfied with Agile & DevOps than Waterfall

Download Report

Regional Differences at a Glance

Asia-Pacific stands out as the region in which testing is prioritized most as part of DevOps transformation, and in which testers are most respected, followed by the Americas. Organizations in Europe and the Middle East had the greatest maturity divide, with both the highest percent of complete DevOps transformation and the highest percent reporting they haven’t started yet. The standout difference in the Americas was the rate at which respondents reported practicing Agile: 85%, compared with 76% in Asia-Pacific and just 30% in Europe and the Middle East. Hover over each region in the map below for more details, and download the report for the full analysis across regions.