Contributed Articles

Defining Software Quality Metrics for Agile and DevOps

By Ian C. Schafer, SD Times reporter

In a a Tricentis-commissioned report from Forrester released in July, “The Definitive Software Quality Metrics For Agile+DevOps,” surveyors found that it’s a common trait of companies that have seen the most success from Agile and DevOps adoption that they have made another operational transition. These companies have moved on from considering “counting” metrics — for instance, whether you’ve run tests an adequate number of times — as key indicators of success, to “contextual” metrics — whether the software meets all of the requirements of the user experience, and that these are better indicators of success.

This conclusion is based on the report’s findings that of the 603 enterprise Agile and DevOps specialists surveyed, those referred to as “DevOps leaders” are significantly better at measuring success and are much further along in their pursuit of end-to-end automation, a key ingredient in meeting the rapid development and delivery expected of modern software cycles.

It seems that somewhere along the way, there is a bottleneck holding some businesses back while the DevOps leaders speed ahead; fifty-nine percent of respondents placed the blame squarely on manual testing.

This focus on automated end-to-end functional testing is among four other best-practices that  the report’s findings say are almost universally employed by DevOps leaders. They are:

  • Properly allocated testing budgets and a focus on upgrading testing skills
  • The implementation of continuous testing “to meet the demands of release frequency and support continuous delivery”
  • The inclusion of testers as part of integrated delivery teams
  • A shift left to implement testing earlier in development

But businesses shouldn’t be quick to rush the gates on implementing test automation across the board — there are some risks involved.

“Automating the software development life cycle is an imperative for accelerating the speed and frequency of releases,” Forrester wrote in its report. “However, without an accurate way to measure and track quality throughout the software development life cycle, automating the delivery pipeline could increase the risk of delivering more defects into production. And if organizations cannot accurately measure business risk, then automating development and testing practices can become a huge danger to the business.”

All businesses, even DevOps leaders, should pay heed. According to Forrester, 80 percent of respondents believe that they “always or often deliver customer-facing products that are within acceptable business risk.” This despite that only 29 percent of DevOps “laggards” think that delivering software within acceptable business risk is “very important,” and only 50 percent of leaders think so. Forrester speculates that this finding must be a mass overestimation.

“Given that most firms, even the ones following continuous testing best practices, admit that their software testing processes have risk gaps and do not always give accurate measures of business risk, it stands to reason that the 80% who say they always or often deliver within acceptable risk may be overestimating their capabilities,” Forrester wrote in the report.

Diego Lo Giudice, vice president and principal analyst at Forrester and the lead on the report, found this disparity unusual.

“I think it’s more intentional than truth — they get it that business risk is of paramount importance and have the illusion of addressing it,” Lo Giudice said. “It’s a misinterpretation of what real business risk is perhaps? I think it’s hard to keep testing and quality focus on business risk…”