It can be difficult for testers to effectively describe the business impact of testing because it requires them to take a step back from their day-to-day and think about which metrics most effectively illustrate the big picture. In this article, you will learn more about:
- The QA KPIs most important to software developers
- How to translate detailed QA and test automation metrics into business language
- How to design KPI reports that will resonate in the C-Suite
There’s no shortage of software testing metrics and KPIs that you could be monitoring and tracking. Some KPIs are very specific and tactical in nature. Tracking these in-the-weeds metrics is critical for helping testing teams manage their overall quality and efficiency. But they’re usually not the ones of interest to executives, who need a distilled version of the facts — just enough to help them make strategic decisions.
While it certainly can’t hurt to track a wide array of quality metrics, the fact is that it’s unlikely outside the testing team will have the bandwidth to review them all. So you need to read your audience and tailor your reporting based on who’s asking, and put them into a context that answers their big questions.
In this article, we’ll take a look at the metrics various stakeholders are most likely to be interested in, as well as how you can make it easier to report high-level progress to audiences outside of the testing organization — so you can spend less time building custom reports and more time testing.
To Report the Business Impact of Testing, Understand the Big Questions Leaders Are Asking
Everyone involved in the software development lifecycle has a distinct role to play. Software developers are busy writing the cleanest code they can. Meanwhile, testers and SDETs want to ensure that they’re testing that code as efficiently as possible to spot errors and defects early, so developers can course correct before the project is too far along. Outside of this feedback loop, other stakeholders will generally be looking for metrics that help them answer the following questions:
- How long will it take to test?
- How much money will it take to test?
- How bad are the bugs?
- How many bugs found were fixed? reopened? closed? deferred?
- How many bugs did the test team did not find?
- How much of the software was tested?
- Will testing be done on time? Can the software be shipped on time?
- How good were the tests? Are we using low-value test cases?
- What is the cost of testing?
- Was the test effort adequate? Could we have fit more testing in this release?
Knowing how to answer these questions with data is critical. Not only will that information give others a sense of how testing is progressing, but it will also serve as a useful checklist for ensuring your team is keeping the big picture, as well as their contributions to the organization’s goals, in mind. Executives are typically using this information to gain insight into three key areas:
1. Speed to market
It’s no surprise that executives want to know how quickly they can bring new features to market; speed often correlates with sales and customer satisfaction, which directly impact the bottom line. Factors that can affect speed to market include:
- How long your regression suite takes to run
- Whether quality assurance is a bottleneck for the dev team
- The percentage of your tests that are automated, the pass/fail rate, and how you’re scaling automation over time
2. The number of bugs in production
Although there will always be a risk of production errors, executives will always want to know how you are working to minimize them, in terms of both quantity and severity. They may also want to know how they were discovered and why there were missed in the first place. Production defects and the attending details are critical to track; for executives, this information can serve as a guidepost for assessing whether overall development efforts are headed in the right direction.
3. The amount of money being spent on testing
Last, but certainly not least, executives are concerned with how the cost of testing is impacting your company’s bottom line. Specifically, they’re going to want to understand the returns they’re gaining on their investment in testing, as well as whether there are opportunities for either improving ROI or cutting costs.
Keeping track of all of the metrics you’ll need to answer these questions, and being able to report on them quickly and efficiently, can be a painful process when done manually. At enterprise scale, you’ll either need an individual dedicated to reporting on your team or a reporting engine that can make the job easier for you.
What to Look for in a Reporting Engine
It’s important to define the key KPIs your team needs to track and that your executive team cares about, then find a tool that allows you to measure and track everything on your list. In almost all cases, it’s critical that reports are customizable since no two projects or organizations are the same. They should also be reusable and offer a variety of visual data representations to choose from, so you can quickly pull information in a digestible format.
One of the key sets of metrics you should define early are indicators of release readiness. That should include a variety of metrics, such as test coverage, pass/fail rate and the highest risk areas. You’ll also want to make sure that the tool you select automatically pulls in data from everywhere you’re running test, including open-source and proprietary test automation tools, and that you’re able to trace tests back to requirements in your agile planning tool.
Perhaps most important of all, you’re going to want to find a tool that’s intuitive and user-friendly so that everyone feels comfortable jumping into it to access the data and reports that they need — and so that release readiness can be assessed in real time and at a moment’s notice. That’s going to be critical for adoption and will help foster a culture of transparency within your organization, as well as a commitment to identifying and resolving weak spots in your process or in the software itself. Ultimately, the business impact of testing lies in its ability to provide a clear picture of release readiness and to equip business leaders with information that helps them improve both software quality and speed to market — and with reports that they’ll actually want to read.