What is mutation testing? A complete guide

Explore the concept of mutation testing: what it is, why it’s important, and what to know to get started with it.

In this article, we’ll cover the concept of mutation testing: what it is, why it’s important, and what to know in order to get started with it.

As you’ll learn, mutation testing isn’t necessarily a distinct category of testing. It’s not, for instance, a competitor to unit testing, but quite the opposite: it’s a technique or approach you can use to make your unit tests better.

What is mutation testing?

Definition

Mutation testing is an approach you can use to assess—and then, hopefully, improve—the quality of your unit tests.

We’ll explain it in more depth soon, but for now, understand that, in practical terms, mutation testing is a type of automated software testing that verifies your unit test suite instead of your production code.

What are the benefits of mutation testing?

One of the toughest challenges you can face when writing unit tests is assessing how good your tests are. Yes, code coverage is an important metric, but it can be illusory. It’s totally possible to have high coverage in a suite composed of low-quality tests.

Mutation testing is the answer to that problem, since it gives you a clear assessment of how capable your current tests are of catching real potential problems you could introduce to your codebase when changing it.

How does mutation testing improve the quality of my test suite?

To understand how mutation testing works and in what way it identifies weak spots in your unit testing suite, let’s imagine what it’d be like to do what mutation testing does, but manually.

Let’s say that all of the following is true:

• You have a function f
• There are unit tests that exercise f
• Currently all tests pass
• You’re confident that f’s code is correct

Based on the above, if you deliberately introduced a defect into your function, one or more unit tests should fail.

If, after introducing some damage to the code of f, all of your tests are still passing, that means that you should either fix some of your current tests or write more tests.

Mutation testing automates all of this. A mutation testing tool will add those defects to your code, run your tests, and then let you know whether the tests were able to detect the defects.

The defects that mutation testing introduces are called, unsurprisingly, mutations. Mutation testing tools add mutations to an application in a safe way.

For instance, in the case of Java, tools would add the mutations to bytecode instead of messing with source code. That way, there’s no risk you’ll ship code with those mutations in it.

Core concepts of mutation testing

Having briefly explained what mutation testing is, let’s now go a bit deeper and also cover some of the terminology associated with it.

In a simplified way, this is how the mutation testing process works:

• • A single defect is added to the application—these are called mutations
  • All tests are run

• If at least one test failed, we say the tests killed the mutation; otherwise, we say the mutation survived

• The defect is removed
• The mutation testing tool adds another defect, and the process repeats

After the process finishes, we’ll get a ratio of killed mutations within all generated mutations. The higher the rate, the better.

A mutation testing tool doesn’t just change your code at random.

Types of mutations

A mutation testing tool doesn’t just change your code at random. Usually, they apply those changes in a highly systematized way, picking from a list of pre-defined mutations.

The exact list depends on the specific tool and also the specific programming language it targets. What follows is a non-exhaustive list of common categories of mutations you can find:

1. Arithmetic mutations: such as replacing a plus sign with another operator.
2. Logical mutations: replacing an and operator with an or, or vice versa. Also, reversing an if statement.
3. Equality mutations: swapping equality/comparison operators.
4. Literal mutations: replacing hard-coded numbers or strings with random values; replacing boolean literals with their opposites.
5. Removal mutations: removing keywords, expressions, or even whole code blocks.

These are just some of the categories of mutations that many tools use. Tools will also provide more mutations that are specific to the language they target. For instance, a tool targeting C# might offer mutations targeting LINQ methods.

How does it differ from traditional software testing?

To explain how mutation testing differs from traditional software testing, we must first define what we mean by “traditional.” If by traditional we mean manual software testing, then it’s clear that mutation testing differs from that because it is a form of automated testing.

But if by “traditional software testing” we mean other forms of automated testing? In this case, mutation testing differs from these types of testing in the following ways:

1. Unlike unit testing, integration testing, and so on, you don’t write any test cases.
2. Mutation testing, unlike unit testing, is usually very slow.
3. Because of the above, you’d typically not integrate mutation testing into your pipelines.
4. Mutation testing, unlike unit testing/integration testing, doesn’t target your production code, but your test code.

Regarding the goal of mutation testing, the most striking way it differs from other forms of testing is probably #4. That is, mutation testing tests not your application, but your unit tests.

When it comes to the creation of tests, you just don’t; many tools will allow you some degree of control and configuration, but which mutations will be deployed to which portion of the code is something the tool itself decides for the most part.

Finally, don’t feel scared about #2 and #3: while mutation testing can be quite slow, it’s not in a way that prevents its use in a practical way. We’ll cover more of that next, when we mention the challenges of mutation testing.

While mutation testing can be quite slow, it’s not in a way that prevents its use in a practical way.

Mutation testing vs. code coverage

Let’s evaluate the relationship between mutation testing and code coverage.

As mentioned earlier, code coverage, despite being useful, can be an illusory metric. It is possible to have a high-coverage test suite that is full of low-quality tests.

Given that, there are people who talk about code coverage and mutation testing as if they’re at odds with each other, hence the “vs.” in the title of this section.

I disagree with this view. Mutation testing makes code coverage more meaningful. More specifically, it turns the high-coverage scenario into a high-signal one.

What do I mean by that? Well, imagine you have very low code coverage. That’s a meaningful, useful (albeit negative) piece of information. It’s a clear, unambiguous bad sign, and you know you can’t trust your test suite at this point.

Now imagine a second scenario: you have reasonably high code coverage (say, something like 85%). Is this great? Not necessarily.

Your tests could be terrific, but they also could be terrible, with useless assertions that don’t evaluate anything, despite exercising a good portion of the logical paths in the code.

Here’s a third scenario: you have the same coverage as before, but now you have a mutation testing score showing that your unit tests didn’t detect most of the mutations that the mutation testing tool introduced.

That’s actually great news, even though the situation is dire. You now are certain you need to improve your suite of unit tests, while in the previous scenario, you simply could never be sure.

A mutation score turns a high-coverage scenario from an ambiguous result into a clear, non-ambiguous one, and therein lies its value.

Mutation testing tools

Since mutation testing tools modify programs written in certain programming languages, your choice of tool will rely mostly on what your programming language is. And this space is not crowded; there is usually one dominant or “standard” tool per language.

Below you’ll see a non-exhaustive list of tools that support some of the major programming languages.

PiTest/ArcMutate: Java and Kotlin

PiTest is largely considered the state-of-the-art mutation tool for Java and the JVM. It was created by Henry Coles 17 years ago, and it boasts being fast and scalable.

Recently, the creators of PiTest released ArcMutate, which is the paid, enterprise version of PiTest. Besides enterprise features, it adds Kotlin integration, new mutators, and better integration with GitHub. PiTest continues to be actively developed as an open-source tool.

Stryker Mutator: JavaScript, C# and Scala

Stryker Mutator is another popular, free, and open-source mutation testing tool. Initially developed for JavaScript/TypeScript, it currently also supports C# and Scala. Stryker is currently the dominant tool for JavaScript and .NET.

Infection: PHP

Infection is a test mutation library for PHP. Like the other entries in this list, it’s a free, open-source project. It’s inspired by an earlier project, Humbug, which has since been discontinued in favor of Infection.

It focuses on ease of use and speed, having an interactive terminal-based UI and parallel execution as two of its most important features.

Mutmut: Python

Mutmut is a mutation testing library for Python. It focuses on ease of use and speed, having an interactive terminal-based UI and parallel execution as two of its most important features.

A mutation testing example

In order to help these concepts become concrete, here’s a brief but instructive example of mutation testing.

First, consider the following C# class:

namespace Tally.Core;

public class OrderCalculator
{
    private const decimal FreeShippingThreshold = 50m;
    private const decimal StandardShippingCost = 5.99m;

    public decimal CalculateSubtotal(IEnumerable<decimal> prices)
    {
        if (prices is null)
            throw new ArgumentNullException(nameof(prices));

        return prices.Sum();
    }

    public decimal ApplyDiscount(decimal subtotal, decimal discountPercent)
    {
        if (discountPercent < 0 || discountPercent > 100)
            throw new ArgumentOutOfRangeException(nameof(discountPercent));

        return subtotal * (1 - discountPercent / 100);
    }

    public decimal GetShippingCost(decimal subtotal)
        => subtotal >= FreeShippingThreshold ? 0m : StandardShippingCost;

    public decimal CalculateTotal(decimal subtotal, decimal discountPercent)
    {
        var discounted = ApplyDiscount(subtotal, discountPercent);
        return discounted + GetShippingCost(discounted);
    }
}

 

Now consider the following tests:

using Tally.Core;

namespace Tally.Tests;

public class OrderCalculatorTests
{
    private readonly OrderCalculator _sut = new();

    [Fact]
    public void CalculateSubtotal_SumsAllPrices() =>
        Assert.Equal(30m, _sut.CalculateSubtotal([10m, 15m, 5m]));

    [Fact]
    public void CalculateSubtotal_ThrowsOnNullInput() =>
        Assert.Throws<ArgumentNullException>(() => _sut.CalculateSubtotal(null!));

    [Fact]
    public void ApplyDiscount_ReducesAmountByPercent() =>
        Assert.Equal(90m, _sut.ApplyDiscount(100m, 10m));

    [Fact]
    public void ApplyDiscount_ThrowsWhenPercentExceeds100() =>
        Assert.Throws<ArgumentOutOfRangeException>(() => _sut.ApplyDiscount(100m, 101m));

    [Fact]
    public void ApplyDiscount_ThrowsWhenPercentIsNegative() =>
        Assert.Throws<ArgumentOutOfRangeException>(() => _sut.ApplyDiscount(100m, -1m));

    [Fact]
    public void GetShippingCost_FreeAtThreshold() =>
        Assert.Equal(0m, _sut.GetShippingCost(50m));

    [Fact]
    public void GetShippingCost_ChargedBelowThreshold() =>
        Assert.Equal(5.99m, _sut.GetShippingCost(49.99m));

    [Fact]
    public void CalculateTotal_IncludesShippingWhenBelowThreshold() =>
        Assert.Equal(45.99m, _sut.CalculateTotal(40m, 0m));
}

 

They all pass, and they look like they are all correct. Are our tests high quality though? Let’s examine the mutation report generated by Stryker Mutator.

As we can see, out of a total of sixteen mutants, two survived. The resulting mutation score was 88.89, which is a good result.

Let’s examine some of the results in more detail:

Here you can see that one mutation for the code above survived. Stryker mutator allows us to expand and see what the mutation was:

The mutation consists of removing the exception throw statement. This mutation survived, which means we don’t have a test that exercises the throwing of the exception.

public void CalculateSubtotal_ThrowsOnNullInput() =>
    Assert.Throws<ArgumentNullException>(() => _sut.CalculateSubtotal(null!));

 

Oh, what a surprise. We actually have a test, but why is the mutation not being killed then? Well, the line return prices.Sum(); already throws an ArgumentNullException either way, which makes the test pass even with the removal of our guard clause.

Look at this interesting result: mutation testing is showing that this guard clause, as it’s written, is useless.

Let’s fix the test by adding the parameter name to the assertions:

[Fact]
public void CalculateSubtotal_ThrowsOnNullInput()
{
    var ex = Assert.Throws<ArgumentNullException>(() => _sut.CalculateSubtotal(null!));

    Assert.Equal("prices", ex.ParamName);
}

 

With this change, the removal of the throw line will cause this test to fail, and that specific mutant will no longer survive.

Here we have another example of a surviving mutant:

As you can see, the greater-than sign was replaced by the greater-than-or-equals-to sign, and this fault wasn’t detected.

Challenges of mutation testing

Mutation testing is a great idea that’s hard to get right in practice. There are many challenges involved. For instance:

1. Mutation testing can be extremely slow, considering that mutations are applied to the code one by one, and then for each mutation the whole unit test suite is executed.
2. Not all possible mutations are useful, so it’s often hard to determine and apply the ones that correlate with realistic errors that could end up in a codebase.
3. Mutations need to be applied “cleanly” to a codebase. That is, code must still compile (if we’re talking about a compiled language), which can be tricky to achieve.

Thankfully, there’s a good deal of research happening on the topic of mutation testing, and many of the challenges above are being solved in practice.

For instance, some mutation testing tools, when integrated with pipelines, are smart enough to only generate mutations for the portion of the code impacted by the pull request, thus reducing the total run time for the tests.

As a general recommendation, don’t add mutation testing to the same pipeline that gets run after every PR is opened or merged.

Maybe create an entirely different pipeline that runs nightly or even weekly, runs only the mutation testing tool, and uploads the resulting analysis somewhere where all interested parties can see.

As an approach, mutation testing can be a really valuable tool in your toolbelt.

Conclusion

Mutation testing is a somewhat different form of testing: you don’t write test cases, and it targets your tests instead of your actual application.

As an approach, mutation testing can be a really valuable tool in your toolbelt. You can use it to make code coverage a more meaningful metric and reach the highest possible quality for your unit test suite.

What about AI? Would mutation testing be useful in this new era? Absolutely. As seen here, mutation testing is a powerful mechanism to identify gaps or opportunities for improvement in your test suites. You can then leverage AI-powered testing tools to close those gaps.

If you’re interested in learning more about AI-powered testing tools, take a look at Tricentis.

This post was written by Carlos Schults. Carlos is a skilled software engineer and an accomplished technical writer for various clients. His passion is to get to the bottom (the original source) of things and captivate readers with approachable and informative technical content.