Parallel testing: what it is and how it works

One of the biggest challenges with software testing is the time that it takes. As Stephan Schmidt notes, “time pressure during the development process is one of the largest reasons that developers skip testing.”

Many companies have gotten over that hump, only to reach a different time-based testing problem: the time it takes during the deployment phase. If you’re using a CI/CD deployment process, you’re looking to deploy as quickly as possible.

Suddenly, your application running thousands of tests balloons your deploy times from a few seconds to several minutes. The same problem happens with developers trying to run the entire test suite during their development flow. Things simply take too long.

One way to tackle this problem is by adopting parallel testing. In this post, we’re going to break down what parallel testing is, why it’s valuable for your team, how you can adopt it, and what mistakes you need to avoid to get the most out of it.

What is parallel testing?

On the surface, parallel testing is a pretty simple concept: run your tests in parallel! Or, to expand a little, utilize multiple CPU cores to execute tests at the same time as one another.

Let’s consider an application with two test files: one tests a validator function for email addresses and the other tests some calendar-based scheduling logic. The email tests take ten seconds to run, and the calendar tests take 45 seconds.

When run in sequence, those tests will take 55 seconds to run. But, with even a little bit of clever work, we can substantially cut down that time.

Instead of running one big block of tests in sequence on a single CPU core, we could instead fork two processes: one running the email tests and one running the calendar tests.

While the forking logic adds a little bit of overhead, adding a second to our runtime, our tests now complete in 46 seconds instead of 55.

Now, let’s imagine that we add 3 more test files, which add up to a total of 30 seconds of run time. Executed linearly, that would put us at 85 seconds of run time.

Executed perfectly, the parallel workflow would still be just 46 seconds! By running independent tests in parallel, we remove bottlenecks on the testing pipeline, which decreases runtime substantially.

In real-world scenarios, no team is optimizing for 10 seconds of quicker test execution. But most companies that invest in tests don’t have two test files, they have two thousand. And their execution timeline isn’t 55 seconds, it’s more like 55 minutes.

On the surface, parallel testing is a pretty simple concept: run your tests in parallel!

Why is parallel testing important?

As we already noted, the primary benefit of adopting parallel tests is the execution performance. If you run your tests on a 12-core CPU, but 1 thread operates the whole time and the other 11 cores sit idle, you’re wasting both developer time and CPU time.

By adopting parallel test execution, you save time for your developers and better utilize your computing resources.

But time speedups aren’t the only benefit to adopting parallel testing. Another benefit is that you’ll write better tests.

One of the core considerations when writing tests intended to be run in parallel is avoiding shared resources between tests. Now, some shared resources are inevitable. It doesn’t make sense to do something like set up a brand-new database for every single test.

But consider a test that writes a file, and then another test that reads the results of that file output. This might seem like a clever way to minimize the setup work for two different tests.

However, if you introduce an error in the logic to write a file, you now have two failing tests with only one breaking change. The logic for parsing the file still works and hasn’t changed, but the test fails. That’s not good!

When you know that your tests will all run in parallel, it forces you to isolate those tests and their side effects. This makes your tests easier to reason about, debug, and use for future regression testing. All of those are big wins for your testing patterns.

As Tim Ottinger and Jeff Langr usefully explain: “When tests are interdependent, a change to one test can cause several others to fail in puzzling ways. Ordering unit tests to optimize their execution is a sign that isolation is poor. Each unit test should instead stand on its own as a complete case that documents one discrete behavior.”

How does parallel testing work?

We’ve already covered the simplest form of parallel testing. You have a small setup script that forks a number of testing processes and then waits for them all to finish. If they all succeed, you report that back to the user. If not, you report that back to the user, too.

In reality, you probably don’t need to do this yourself. It’s likely that your testing library already comes with the capability to run tests in parallel. For smaller and more moderately-sized codebases, that’s probably sufficient.

If you’re running tests across multiple languages, it might be worth looking into a dedicated parallel testing tool. A library like btest provides the ability to configure your parallel tests across all your languages, meaning that your team doesn’t need to know how to set up parallel tests in all your languages.

Those tools are good fits for small and medium-sized codebases. If you’re working on a bigger codebase, you might want to invest in dedicated tooling that will provide more benefits than just the simplest possible parallelization.

Tricentis Testim test automation not only makes authoring new tests faster and easier, it also makes running them in parallel as easy as a few clicks.

What are some key considerations with parallel testing?

If you’re considering adopting parallel tests, there are some things you should keep in mind:

• Keep your tests isolated. We’ve talked about the importance of this before, but it bears repeating: tests that rely on other tests are a sign that your tests are poorly written.
• Don’t expect miracles. You might think that you have 10 cores and 100 tests, so your tests will run 10 times faster! Unfortunately, that’s not how these things usually work. Amdahl’s Law still applies, even when thinking about tests.
• What works today isn’t always going to continue working. You may need to adjust your approach as your test suite grows.
• Even with all your work, test suites grow faster than your ability to contain them. Eventually, running all your tests on local developer machines won’t be a reasonable goal.

If you’ve committed to automated testing as part of your deployment process, the day is going to come when those tests outgrow your ability to manage them.

Tricentis Tosca is here for your testing needs

By now, the benefits of parallel testing should be obvious. If you’ve committed to automated testing as part of your deployment process, the day is going to come when those tests outgrow your ability to manage them.

You might already be there! If you are, Tosca is right here to help. Our continuous test automation platform works to run tests in parallel right out of the box, meaning that you won’t need to worry about complicated configuration.

And if you’re not quite at that point yet, but you need help working out how you want to handle your tests, we’ve got that covered too. Reach out, and our expert team will be happy to help you find the exact right solution for your needs.

This post was written by Eric Boersma. Eric is a software developer and development manager who’s done everything from IT security in pharmaceuticals to writing intelligence software for the US government to building international development teams for non-profits. He loves to talk about the things he’s learned along the way, and he enjoys listening to and learning from others as well.