Quality assurance can be a complex, layered process, often with many people playing different roles to help bring new software to market. And while everyone involved in that process wants the software to succeed, the reality is that they all care about different things. Developers, for example, just want to ensure that there aren’t any bugs in their code — particularly ones that they’re responsible for. Most members of the C-Suite, on the other hand, are almost exclusively focused on what testing means for the bottom line. So how do you determine which QA KPIs will resonate best with each of these audiences?
The truth is that there are many different metrics and key performance indicators (KPIs) that matter when it comes to ensuring the quality of your company’s software. Having said that, however, not all of them matter equally — or even at all — to everyone else in your organization.
For that reason, no matter where in your company’s quality assurance hierarchy you find yourself, it’s important to be aware that the metrics that you and your peers care about may differ from the way others measure success. That’s particularly true when it comes to managing up. To do so, you’ve got to know what metrics your boss (and their boss, and their boss’s boss) wants to see and how to find them quickly.
QA KPIs for different audiences
Different quality assurance KPIs matter to different people throughout the software development life cycle. Just consider the following examples:
Developers & automation engineers
Given their desire to create high-quality code with no defects attributed to them, developers are generally focused on two key metrics: whether or not they’ve written all of the necessary unit tests and the number of bugs they introduced. They use code coverage reports to track whether they have written enough unit tests and code quality tools to ensure that their code is secure and not deprecated. Any bugs that they find in the process, they fix along the way.
Automation engineers are concerned with the efficiency of their tests. They want to make sure that testing doesn’t take too long, but still gets results. As such, they want to know if their automated tests are passing or failing and if the results are accurate. They also pay attention to how brittle tests are, if they’re current and how often they need to be fixed.
Manual testers want to ensure that they have good test coverage and that they have tested all of the necessary requirements. They’re also the ones who want to know if the app that’s being developed actually works well and will result in a great user experience.
With overall responsibility for the product, product leads want to know if their team executed everything that was required prior to releasing any code. One of those requirements is testing and, as such, they have to satisfy the testing director. That means that they’re focused on issues like:
- Is there traceability between requirements and executed tests for auditing purposes?
- Did all of the required tests get run prior to the release?
- How many bugs were introduced in production or during an iteration of work?
- Are all of the new requirements and features covered?
- Have the appropriate risk factors been taken into consideration?
- How effective is the team?
Given their critical role in the process, team leads need to manage up more than anyone else in the software development life cycle. To help them do so, they rely on a variety of tools including quality dashboards and velocity and traceability reports.
Next come the testing directors, who want to ensure that their company is following all of the best practices and running all of the right tests across its products. Specifically, testing directors want to know:
- Is the team executing load tests, and if so, which loads and why?
- Are stress tests, performance tests and security tests being conducted? If so, how is the code being tested and measured in each case?
- Are the static code analysis tools being configured correctly for all products?
- Are the best testing tools being used for the job?
- Are there test plans and is the company set up for a successful verification and validation process?
- Are teams taking a risk-based approach?
- What percentage of tests are getting executed on each release of each product?
- Are there enough testers to cover all of the products?
CIOs or CTOs
As the top of the house, the CIO or CTO knows that testing is important, but worries about it at the highest level. In particular, people in this position want to know if higher-level items such as infrastructure are being tested and that testers are monitoring for high performance in production environments. In addition, they want to make sure that they’re keeping up with competitors, which means knowing that they have the best testing practices and tools.
Are QA KPIs relevant to other executives?
Since other members of the C-Suite don’t always fully appreciate the value of testing, they tend to look at it in much more black and white terms. For example, they’re going to want to know the total cost of testing versus the value the company is deriving from it. They’ll also want to know how many production bugs there are and whether that has resulted in lost customers. How quickly new releases are coming to market is also a top concern. That’s because speed to market is essential when it comes to adapting to meet ever-changing industry needs. Learn more about communicating QA’s value to the C-Suite.
The final word
Although it’s easy to adopt tunnel vision and only think in terms of the testing metrics and KPIs that matter most to you, to work effectively in an organization, you’ve got look beyond yourself. Specifically, that means taking note of what your boss cares about and making sure that you’re providing her with the data she needs — not just what’s important to you.
It’s important to make that process as easy as possible for everyone in your testing organization. Fortunately, using tools can help put the right data at your fingertips. Better yet, the right tool will automatically collect relevant data into dashboards for you, so that it’s there for everyone to see. With that level of transparency, you can continue to focus on what you care about, with the confidence of knowing that you can communicate your value across the organization with the QA KPIs that matter most to each stakeholder.