Regression testing is the process of running a test suite designed to determine if new or modified code negatively impacts existing functionality. For example, regression testing should alert the team if the code added for a new payment feature unintentionally broke the existing checkout functionality. The more effectively your regression test suite checks your existing functionality, the better it will be at alerting your team when changes inadvertently break something.
Think of your regression test suite as a baseline test that captures the project code’s current functionality. To detect changes from this baseline, you run your evolving code base against this test suite on a regular basis. This creates a change-detection safety net, letting you modify code without fear of accidentally changing or breaking the code’s intended behavior.
Regression testing can be performed using manual testing or automated testing. It can (and should) include a mixture of UI testing, API testing, and unit testing.
It is important to recognize that not every regression test failure means that application changes “broke” the application under test. Often, test cases fail due to false positives: test cases that fail when the software is actually functioning properly. 46% of all false positives are caused by automation issues. 32% are caused by missing, incomplete, incorrect and/or outdated test data. 22% are caused by issues related to test environments.
Regression Testing and Continuous Testing
For Continuous Testing, regression testing is performed during a team’s Continuous Integration (CI) process. For testing to be completed during CI, regression testing must be automated.
To ensure that the regression test suite provides fast feedback, most teams decide to run only a subset of all available tests on a regular basis, then run the complete regression testing before major milestones. To select the subset of tests used for the shorter “smoke testing,” teams may use risk-based prioritization, test impact analysis, or other techniques.
Many regression test suites have a high level of redundancy (tests that do not add any value to the regression test suite). Reducing the level of regression test suite redundancy is a popular way to “slim down” the regression test suite for Continuous Testing.
Regression Testing Resources
- A Regression Test a Day Keeps the Bugs Away: A good overview of the idea behind regression testing.
- Get Smart about Your Regression Tests’ Value: Helps you measure and maximize the value of your regression tests.
- Cure HP UFT Regression Test Suite Bloat: Learn how to assess the severity of your test suite bloat, and the treatments and prevention—including risk-based testing and model-based testing via lightweight tooling.
- Dolby Laboratories Amplifies Regression Testing: Dolby, the top name in cinema audio, almost produced a cliffhanger of their own when inconsistent test coverage and a lack of testing standardization threatened to undermine the integrity of their most business-critical processes. Read this case study to learn how a risk-based test automation approach saved the day.
- The Top Software Testing Tools- The Ultimate List of Lists: A list of the top lists of regression testing tools.