Parallel testing: What it is and how it works

Learn what parallel testing is, how it works, and why QA teams use it to improve coverage, efficiency, and pipeline performance.

Running software tests takes time, every time that you run one. While running tests, there are two major options for how your testing library executes the tests: parallel, and sequential.

Parallel testing is simple: utilize multiple CPU cores to execute tests at the same time as one another. Sequential testing instead involves running each test one at a time, waiting until a test finishes before starting the next.

One of the biggest challenges with software testing is the time that it takes. As Stephan Schmidt notes, “time pressure during the development process is one of the largest reasons that developers skip testing.”

Many companies have gotten over that hump, only to reach a different time-based testing problem: the time it takes during the deployment phase. If you’re using a CI/CD deployment process, you’re looking to deploy as quickly as possible.

Suddenly, your application running thousands of tests balloons your deploy times from a few seconds to several minutes.

One way to tackle this problem is by adopting parallel testing. In this post, we’re going to break down what parallel testing is, why it’s valuable for your team, how you can adopt it, and what mistakes you need to avoid to get the most out of it.

What is parallel testing?

Let’s consider an application with two test files: one tests a validator function for email addresses and the other tests some calendar-based scheduling logic. The email tests take ten seconds to run, and the calendar tests take 45 seconds.

When run in sequence, those tests will take 55 seconds to run. But, with even a little bit of clever work, we can substantially cut down that time.

Instead of running one big block of tests in sequence on a single CPU core, we could instead fork two processes: one running the email tests and one running the calendar tests.

While the forking logic adds a little bit of overhead, adding a second to our runtime, our tests now complete in 46 seconds instead of 55.

Now, let’s imagine that we add 3 more test files, which add up to a total of 30 seconds of run time. Executed linearly, that would put us at 85 seconds of run time.

Executed perfectly, the parallel workflow would still be just 46 seconds! By running independent tests in parallel, we remove bottlenecks on the testing pipeline, which decreases runtime substantially.

In real-world scenarios, no team is optimizing for 10 seconds of quicker test execution. But most companies that invest in tests don’t have two test files, they have two thousand. And their execution timeline isn’t 55 seconds, it’s more like 55 minutes.

Parallel testing vs. sequential testing

Parallel versus sequential testing is not a trivial decision. Generally, running tests in parallel is an overall positive for your team, but those changes don’t come without cost. Let’s break down the pros and cons of each approach.

 

On the surface, parallel testing is a pretty simple concept: run your tests in parallel!

Why is parallel testing important?

As we already noted, the primary benefit of adopting parallel tests is the execution performance. If you run your tests on a 12-core CPU, but 1 thread operates the whole time and the other 11 cores sit idle, you’re wasting both developer time and CPU time.

By adopting parallel test execution, you save time for your developers and better utilize your computing resources.

But time speedups aren’t the only benefit to adopting parallel testing. Another benefit is that you’ll write better tests.

One of the core considerations when writing tests intended to be run in parallel is avoiding shared resources between tests. Now, some shared resources are inevitable. It doesn’t make sense to do something like set up a brand-new database for every single test.

But consider a test that writes a file, and then another test that reads the results of that file output. This might seem like a clever way to minimize the setup work for two different tests.

However, if you introduce an error in the logic to write a file, you now have two failing tests with only one breaking change. The logic for parsing the file still works and hasn’t changed, but the test fails. That’s not good!

When you know that your tests will all run in parallel, it forces you to isolate those tests and their side effects. This makes your tests easier to reason about, debug, and use for future regression testing. All of those are big wins for your testing patterns.

As Tim Ottinger and Jeff Langr usefully explain: “When tests are interdependent, a change to one test can cause several others to fail in puzzling ways. Ordering unit tests to optimize their execution is a sign that isolation is poor. Each unit test should instead stand on its own as a complete case that documents one discrete behavior.”

 

How does parallel testing work?

We’ve already covered the simplest form of parallel testing. You have a small setup script that forks a number of testing processes and then waits for them all to finish. If they all succeed, you report that back to the user. If not, you report that back to the user, too.

In reality, you probably don’t need to do this yourself. It’s likely that your testing library already comes with the capability to run tests in parallel. For smaller and more moderately-sized codebases, that’s probably sufficient.

If you’re running tests across multiple languages, it might be worth looking into a dedicated parallel testing tool. A library like btest provides the ability to configure your parallel tests across all your languages, meaning that your team doesn’t need to know how to set up parallel tests in all your languages.

Those tools are good fits for small-sized codebases. As your codebase expands, you will be faced with a problem. Even with parallelized tests, running all of the tests in your codebase will be too time-consuming to leave to individual developers.

Realistically, by the time your test suite takes about five minutes to run, you have too many tests to make your developers wait for it. That presents a problem.

The solution to this is to build a continuous testing system that relies on parallel, automatically-executed tests run in the cloud on every commit. Tricentis Testim test automation fits the bill for this perfectly.

It not only makes authoring new tests faster and easier, it also makes running them in parallel as easy as a few clicks.

The outcome of adopting cloud-based parallel automated testing is that your team derives the most value from your automated tests, while gaining the benefits of parallel testing, keeping your tests running quickly.

And because the tests run in the cloud, they don’t eat up precious work time for your developers. The five-minute testing window is often reachable in a few hundred to a thousand tests.

A traditional enterprise codebase will often have better than ten thousand tests. That means that cloud-based parallel testing is a requirement to scale testing for a serious software codebase.

What are some best practices for parallel testing?

If you’re considering adopting parallel tests, there are some things you should keep in mind:

1. Keep your tests isolated

We’ve talked about the importance of this before, but it bears repeating: tests that rely on other tests are a sign that your tests are poorly written.

2. Don’t expect miracles

You might think that you have 10 cores and 100 tests, so your tests will run 10 times faster! Unfortunately, that’s not how these things usually work. Amdahl’s Law still applies, even when thinking about tests.

3. Manage test suite growth

Even with all your work, test suites grow faster than your ability to contain them. Eventually, running all your tests on local developer machines won’t be a reasonable goal.

4. Improve individual test performance

A high number of tests for your application is important for improving the quality of your deployments, but it means that the amount of CPU time you spend testing grows, too.

While parallelizing your test runs will buy you some overall execution time, it’s also worth it to identify costly tests and improve their performance.

A flaky test is one which fails unpredictably for reasons unrelated to the code change in the current commit.

5. Avoid flaky tests

One of the biggest thorns in the side of continuous testing plans is the flaky test. A flaky test is one which fails unpredictably for reasons unrelated to the code change in the current commit. If you discover a flaky test in your code base, fix it immediately.

The purpose of parallel testing is to reduce the time you spend running tests. If you need to re-run tests to determine whether a test failed because of a change or because it is flaky, you’ll lose that same precious time.

6. Keep your test data clean

Test data is an underappreciated part of the testing cycle. You need data that matches normal use cases, abnormal use cases, erroneous data, and extremely wrong data.

That data needs to be accessible by your cloud-based parallel testing units. It shouldn’t ever be modified by your tests; that would make it impossible to run your tests in parallel.

7. Don’t parallelize code which mutates shared resources

You can’t avoid all mutations in your tests. You’re going to write code that changes something; you need to test that code to ensure it’s working before you deploy it. Imagine you have two tests: one that tests that a function correctly modifies a user’s phone number.

Another test attempts to delete the phone number entirely. Running these tests in parallel creates a race condition. That results in a flaky test. The way to avoid this is to simply avoid running those tests in parallel entirely and instead require that they run in sequence.

If you’ve committed to automated testing as part of your deployment process, the day is going to come when those tests outgrow your ability to manage them.

Common challenges of parallel testing

Parallel testing is efficient, but it isn’t a silver bullet. There are some common problems you should watch out for when you start running tests in parallel.

1. Debugging

Adopting parallel testing isn’t always sunshine and roses. Obviously, sooner or later, you’re going to have tests fail within your test suite. Usually, that’s good news. A failing test means that you’ve caught an issue before shipping it to customers.

But when it comes to parallel tests, that’s not always the case. Because parallel tests share resources like memory, storage and database connections among the test library, there’s a chance that a failing test is caused by issues like resource contention, not faulty logic.

This is a big problem when it comes time to debug the test. If the source of the problem is resource contention or issues with parallelization, you can’t discover that by running the test in isolation.

Debugging parallelized code is notoriously more difficult than debugging code running in serial, and that’s true for testing as much as it is production code.

2. Writing tests

This is related to our debugging issue from above. Parallel tests usually share resources like memory or database connections. If a particular set of tests involves a large amount of pre-test setup, like inserting many rows into a database, running tests in parallel is very challenging due to that shared resource.

While it’s not true that every test suite needs special care to run in parallel, certain types of tests definitely do.

3. Library and language support

Another key issue with parallel testing can be the support provided by your programming language or testing library.

While there aren’t any mainstream programming languages on the market today which have zero support for parallel tests, certain popular languages like Ruby and Python present challenges with code parallelization in general.

This is one of the most common issues that leads to teams avoiding test parallelization in the first place: if your team needs to spend time and effort to set up parallel tests, it’s easy to put it off until some point in the future.

Tricentis Tosca is here for your testing needs

By now, the benefits of parallel testing should be obvious. If you’ve committed to automated testing as part of your deployment process, the day is going to come when those tests outgrow your ability to manage them.

You might already be there! If you are, Tosca is right here to help. Our continuous test automation platform works to run tests in parallel right out of the box, meaning that you won’t need to worry about complicated configuration.

And if you’re not quite at that point yet, but you need help working out how you want to handle your tests, we’ve got that covered too. Reach out, and our expert team will be happy to help you find the exact right solution for your needs.

This post was written by Eric Boersma. Eric is a software developer and development manager who’s done everything from IT security in pharmaceuticals to writing intelligence software for the US government to building international development teams for non-profits. He loves to talk about the things he’s learned along the way, and he enjoys listening to and learning from others as well.