What Is back-to-back testing in software testing?

There are many types of software tests, all serving many different purposes. One of the benefits of working with experienced developers is that their knowledge and experience can provide wisdom about which tests are best for which purposes.

One common type of test is the “back-to-back test,” also known as a “comparison test.” In this post, we’ll break down what the back-to-back test is, why you might want to use one, and when it’s the wrong choice for what you’re doing.

What is back-to-back testing? The short version

The short version of back-to-back testing is this: A back-to-back test is where you test the same piece(s) of code twice: once before a set of changes, and once after the changes are applied.

For some types of code changes, the outcome of the change is obvious. If you modify a web page layout function to change the title of the page, when you deploy the change to your customers, the webpage title will be different.

While you might want a unit test to validate that the title of the page is what you expect it to be, you don’t need to run a test before and after that change. You know the outcome when you change the page.

But not all software changes are so static or predictable. For those kinds of changes, you’ll want to reach for a back-to-back test. Let’s walk through how you perform back-to-back tests and how you can adopt them for your team.

Back-to-back testing is also used in more formal software environments for the opposite effect: to validate that two pieces of software are effectively identical.

For example, consider this point from the paper presented to the Lambda Mu conference in 2023 by Guignard et al.: “By running the two products ‘back to back’ on a set of test cases, it is verified that they produce the same output. If the scenarios are chosen carefully and the two runs produce the same results, it can be inferred that the two products are equivalent with some level of confidence.”

A back-to-back test is where you test the same piece(s) of code twice: once before a set of changes, and once after the changes are applied.

How back-to-back testing works

The basic structure of a back-to-back test is fairly straightforward. You can explain it in two easy steps:

• Perform a test with your legacy code in place, and record what happens as you execute the test
• Perform the exact same test with your new code in place, and record what happens as you execute the test again

However, things aren’t always that simple in practice. Effectively building and executing a back-to-back test takes a little bit more planning and work to be effective. If you’re planning a back-to-back test for your organization, here are some key considerations to keep in mind.

Understand the purpose of the test

Depending on the scope and scale of your test, it’s possible to collect a lot of data during a back-to-back test. In many instances, back-to-back tests are run in production or production-like environments (like a user acceptance environment).

These environments generate a lot of traffic, which leads to a lot of data collected during your test. This makes it critical to turn your data into actionable information. In order to do that, you need to filter down to only the useful data when collecting outcomes of your initial test.

I recognize that this may sound contradictory. However, it is wise to examine anomalous data that arises in the second half of a back-to-back test, even when it is not the core focus of the test.

One of the reasons that you’ll choose a back-to-back test is uncertainty. You’re not totally sure what’s going to happen when you change your code. When that’s the case, it’s wise to investigate anomalies that arise. This is true even if they’re not directly related to the purpose of the test.

Whenever possible, the best way to perform back-to-back testing is through replaying the exact same set of events for both sets of code.

Mirror the tests as closely as possible

One common issue with back-to-back tests is that they are often not run in identical conditions. Unlike more automated software tests, back-to-back tests are commonly used to measure “real-world” behaviors of your application.

This means that variations in usage often lead to statistical noise in the behavior of the application.

When you’re not truly sure how a code update will impact your codebase, that noise makes it difficult to tell whether a change you see in the outputs is related to your code, or simply a difference in behavior during data collection.

Whenever possible, the best way to perform back-to-back testing is through replaying the exact same set of events for both sets of code.

For instance, Apache Kafka makes it possible to replay events in an exact match. Normally, this is used to protect against data loss, but it can be a great feature for working with comparison testing, too!

Follow the scientific method

This might seem like an odd choice for an automated testing approach, but stop and think it through. The reason that you reach for a back-to-back test is because you’re not sure what outcome will arise after making a code change.

But it’s a bad idea to ship code without having any idea what will happen! Instead, when you make your change, propose a hypothesis and then measure the results in light of that hypothesis. If you receive an unexpected result, instead of simply shrugging your shoulders and considering it good enough, investigate the result and refine your approach.

Why back-to-back testing matters

Back-to-back testing is an important tool in your QA toolbox. Despite our best efforts, it’s not possible to fully understand the real-world impact of every software change before deploying those changes.

Some systems are simply too complex, or their usage patterns too noisy, to fully understand what will happen before deployment. Back-to-back testing provides a mechanism for you to test the outcome of those changes.

Common use cases for back-to-back testing

Perhaps the most common variant of back-to-back testing is one that we don’t necessarily think of as being a “software test” in the traditional model: A/B testing.

The idea behind A/B testing is that you don’t know how customers will react to specific application changes. As a result, you send some customers into one version of the application. At the same time, you send other customers into a different version.

Then you record how they behave based on which version routes them. This test does not strictly behave “back to back” because the testing runs simultaneously, but it still fits the spirit of our comparison test behavior.

More traditional and also common back-to-back tests are performance tests and load tests. Performance testing is a great use case for back-to-back testing.

As a developer, you can identify performance bottlenecks. But it’s often impossible to know the true impact of the changes you make until you test them at full load.

That’s why a back-to-back test is a perfect fit. Test your performance characteristics with your previous code, record them, and then compare them to performance with the new implementation.

Test your performance characteristics with your previous code, record them, and then compare them to performance with the new implementation.

Tricentis provides an expert testing experience

If you’re thinking about adopting a back-to-back test, but you’re not sure where to start, let Tricentis help! We’re testing experts. We offer automated testing tools. These tools integrate easily with your technology stack. They work regardless of your level of expertise.

For instance, Tricentis NeoLoad is ready to take your performance testing to the next level, and it’s easy to start today.

Whatever your testing needs, Tricentis is always ready to be your trusted partner.

This post was written by Eric Boersma. Eric is a software developer and development manager who’s done everything from IT security in pharmaceuticals to writing intelligence software for the US government to building international development teams for non-profits. He loves to talk about the things he’s learned along the way, and he enjoys listening to and learning from others as well.