Software test coverage: what you need to know

Writing tests is a key part of building and maintaining quality software. Good tests exercise your code paths and document that code by showing how it behaves under specific circumstances. Tests also prevent regression when you modify code. Given that tests are so important, many software teams seek to measure how much of their code is covered by their tests. We call this metric test coverage, and you’re here to learn more about that today.

What is software test coverage?

Software test coverage is a metric that measures the amount of code in a repository “covered” by tests. Usually, this is measured directly by how many lines of code execute during a run of 100% of the tests in your test suite. For example, let’s say that you have a function that accepts a single parameter, and that function branches in an if/else block to handle two possible outcomes. If you write two tests, one that passes through the if section, and another that passes through the else section, you would measure 100% on the test coverage metric.

Now let’s say that you add a third if-else block to your function. Suddenly, your two tests don’t provide 100% coverage. Because you’ve added a third line of executable code, you’ve migrated from 100% test coverage to 67%.

What does software test coverage measure?

Test coverage metrics measure how many lines of executable code execute during a test run. This means that the metric won’t cover things like configuration files or comments. Test coverage usually doesn’t measure things like class or function definitions either. The easy way to think of this is that test coverage is using the bodies of the functions that you write as the denominator for the test coverage ratio. Then, when you execute a test suite, that suite internally marks each line of code the test suite executes. That forms the numerator of the ratio.

At the end of your test run, the test suite will report the ratio of lines executed to total lines of executable code, providing a “test coverage” ratio. This metric is particularly useful when combined with test automation, which will then report how much the coverage ratio has changed since the last time the automated test suite ran. Some companies will even reject change requests that decrease overall test coverage from merging into the main branch of a repository.

Test coverage is a useful metric for learning certain things about your tests. But it isn’t a perfect metric. You don’t want to rely too much on test coverage as a proxy for code quality.

What doesn’t test coverage measure?

Test coverage is a useful metric for learning certain things about your tests. But it isn’t a perfect metric. You don’t want to rely too much on test coverage as a proxy for code quality. Let’s dive into why.

Test quality

This is the biggest flaw with test coverage as a metric. Inexperienced technologists believe that the right target number for test coverage is 100%. They believe that because tests are important, your tests should cover every line of executable code. This view is short-sighted. If you think back to our example function earlier, you could write a test that executes all three branches of your function, and you’d have 100% test coverage. But if your tests don’t actually test any assertions, your 100% test coverage metric is useless.

Code quality

While it’s true that exercising a software testing lifecycle does generally improve your overall code quality, higher test coverage does not necessarily result in better code. Instead, the result may simply be that your team writes complicated, convoluted tests to cover their complicated, convoluted code.

Cycle time

Software cycle time is a measurement of how much time it takes to complete a new feature or entire software project. Reducing cycle time is a way to improve the value your software team delivers to your customers. Requiring high software test coverage metrics often negatively impacts cycle times. This is because changing any code usually means that you break existing tests. What’s more, any new code that you write will require additional tests. While these are negatives toward high test coverage, it doesn’t mean that you should avoid writing those tests. It’s just a consequence of the approach.

Developer happiness

This is a bit of a cheeky point, but it’s true. Pushing for very high test coverage metrics often leaves developers writing tedious tests that provide minimal value to the code base. Requiring developers to do this work for extended periods of time might lead to burnout and dissatisfaction with their jobs.

How should you think about test coverage?

The traditional ratio approach to test coverage as a metric has some obvious downsides, as we’ve noted. But that doesn’t mean that test coverage is a bust. Instead, you can rethink how you approach test coverage to improve the benefits and reduce some of the downsides.

Here are some ways that you can rethink test coverage to improve things.

Product coverage

Instead of approaching test coverage as a pure metric of lines executed versus total lines of code, you can instead measure how many key features of your product are covered by quality tests. A metric like this is more difficult to automatically measure but has long-term benefits compared to rudimentary test coverage metrics because you know that you’re testing your critical code paths.

Risk coverage

Another criticism of traditional test coverage metrics is that it treats every line of code as equivalent. Realistically, not all lines of code in your software are created equal. A code change that updates the language on an error message is not nearly as important as a change that updates customer billing. As such, one way to approach test coverage metrics is to instead measure how much of your most critical code is covered by tests.

What are the best practices for software test coverage?

If you’re still up in the air about measuring test coverage, you shouldn’t be. You should absolutely measure how much of your code is covered by tests and seek to find the sweet spot for your team and project. But how do you find that sweet spot? Let’s go through a couple approaches.

Identify your key testing metrics

As we’ve noted, pure ratio-based test coverage leaves a lot to be desired. Simply saying “our tests cover X% of our software” doesn’t tell anyone much of any value. Instead, the key to good test coverage is to first think about what you want your tests to do for your team. Do you have high levels of code churn? Test coverage might help you prevent regressions when you ship changes. Do you regularly ship code that doesn’t meet client requirements? Your test coverage might instead focus on enumerating all of the requirements in a feature, then ensuring that the software meets those requirements.

When you think about what you need your tests to do for you, you’ll find it much easier to know how to measure test coverage later on.

Adopt a high-quality automated testing platform

Test coverage by itself is of no value if you never run your tests. The best unit test suite doesn’t do anything if a broken test doesn’t prevent developers from shipping new builds. That’s why high-quality test management tools are a hard requirement for getting return on your test investment.

Invest in writing tests

This might feel like it goes without saying, but it doesn’t. The biggest mistake that development teams make with test coverage is not investing any time to write good tests. Software tests really are a very valuable part of your SDLC, but they take time to write and debug. If you’re thinking about adopting test coverage metrics for your team, that means you recognize the value that testing provides. But if you expect that investing in tests won’t come with any impact to development speed, you’ll quickly find tests falling right back by the wayside.

Software test coverage is beneficial, but not a silver bullet

If you only come away from this article with one impression, I hope it’s this: software testing is worth the time and effort you put into it. Test coverage is a useful metric for measuring how well you test your software, but it’s a fundamentally “lossy” metric. Like a lossy audio compression algorithm, test coverage loses information about the quality of your code when you evaluate it. So, you shouldn’t expect that simply improving code coverage by itself will make your software fundamentally better. But you can use code coverage as a useful indicator to help you find where you’re missing tests and improve that code.

This post was written by Eric Boersma. Eric is a software developer and development manager who’s done everything from IT security in pharmaceuticals to writing intelligence software for the US government to building international development teams for non-profits. He loves to talk about the things he’s learned along the way, and he enjoys listening to and learning from others as well.