Delivering high-quality software fast is no longer a luxury— it is a prerequisite for digital success and mandatory for customer loyalty and acquisition. We are in an era where customers are empowered to choose and switch to the company that can provide them the best experience quickly and at minimal cost. Organizations that cannot keep up with their customers’ expectations and adapt to new business paradigms will find themselves losing customers and business to competitors that can.
How can firms deliver with quality at speed? Implementing Agile+DevOps best practices and building in quality is the first step — and this is not trivial. Automation is the lever that development teams pull to go faster. However, if not properly managed, automation can jeopardize quality and increase risk. To ensure that quality issues and unacceptable risk do not negate the velocity benefits of automation, firms must ensure they are tracking the right metrics to measure quality throughout development-testing-deployment.
To provide the DevOps community an objective perspective on what quality metrics are most critical for DevOps success, Tricentis commissioned Forrester to research the topic. The results are published in a new 55-page ebook, Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics—Ranked by Industry Experts.
In this blog, let’s focus on the heat maps and analysis for each of the 4 major metric categories: build, functional validation, integration/API testing, and end-to-end testing.
Build
When measuring builds, unit testing done well matters. Counting unit tests is a waste of time but understanding change impact matters. Tracking “unit” tests prioritized by risk is the key. As the code base evolves, developers and testers need immediate feedback about change impact. This feedback is significantly more actionable if prioritized by level of risk.
Sixty-three percent of these firms consider the number of unit tests prioritized by risk as one of their top desired metrics. But far fewer can actually do so — while 34% of advanced DevOps firms track the number of unit tests run, only 27% prioritize by risk. And less advanced DevOps firms use it even less — just 15% can track the metric today. Other important metrics tracked in builds focus on ensuring code quality — like the number of successful code builds (61%), unit test pass/fail rate (60%), and total number of defects identified (59%).
Functional validation
When functional testing kicks in, user story/requirements coverage gets the focus. The top goal for testers in this stage is to minimize risk by ensuring that the functionality expressed in user stories works as expected. Extending this coverage concept to business risk coverage is an additional metric that leading DevOps firms execute to advance release automation.
Running functional tests and checking against covered requirements (69%), the density of bugs (62%) or number of functional defects found (66%), and the ratio of tests passed vs. failed (64%) are metrics that successful teams identify as important to manage risk and track quality during functional testing.
Integration/API testing
To deal with modern distributed architectures, measuring integration testing and API tests win all around. With application architectures becoming more decoupled, decomposed in services and microservices, API testing increases in relevance. So much is going on beyond the user interface that it’s impossible to keep quality high and diminish risk without addressing the API layer. Advanced DevOps firms prioritize many of the same metrics for integration testing as they do for the functional testing phase; this time, the focus is on APIs specifically.
These metrics include tests run against functional requirements (75%), total number of new API defects found (64%) and API bug density (63%), API test pass vs. fail rate (62%), and API code coverage (62%). Monitoring API risk coverage is an important metric as well, with 62% of advanced DevOps firms prioritizing it as a top metric for the category.
End-to-End regression testing
End-to-end regression testing also gets a first-class citizen role. In this category, leading organizations automate end-to-end tests at the process or transaction level. Automating these types of tests is not easy, and advanced test automation tools are required because speed matters. The choice of the testing technology matters because achieving and, more importantly, maintaining high levels of automation is crucial. Therefore, the more tests that are automated, the better, and so 70% of leading DevOps teams prioritize the percent of automated end-to-end test cases as a top metric.
Many of the other metrics that leading firms rank as important in this category are quantitative and measure coverage of functionality vs. requirements (70%), number of test cases executed (65%), and total number of defects identified during testing (66%).
***
For quadrant mappings, metrics details, and additional analysis, see the complete 55-page report: Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics—Ranked by Industry Experts. The report is available for free on the Tricentis web site.