Regression Testing

Software testing 101

Regression Testing

Regression testing is the process of running a test suite designed to determine if new or modified code negatively impacts existing functionality.  For example, regression testing should alert the team if the code added for a new payment feature unintentionally broke the existing checkout functionality. The more effectively your regression test suite checks your existing functionality, the better it will be at alerting your team when changes inadvertently break something. 

Think of your regression test suite as a baseline test that captures the project code’s current functionality. To detect changes from this baseline, you run your evolving code base against this test suite on a regular basis. This creates a change-detection safety net, letting you modify code without fear of accidentally changing or breaking the code’s intended behavior.

Regression testing can be performed using manual testing or automated testing. It can (and should) include a mixture of UI testing, API testing, and unit testing.

It is important to recognize that not every regression test failure means that application changes “broke” the application under test. Often, test cases fail due to false positives: test cases that fail when the software is actually functioning properly.  46% of all false positives are caused by automation issues. 32% are caused by missing, incomplete, incorrect and/or outdated test data. 22% are caused by issues related to test environments. 

Regression Testing and Continuous Testing

For Continuous Testing, regression testing is performed during a team’s Continuous Integration (CI) process. For testing to be completed during CI, regression testing must be automated. 

To ensure that the regression test suite provides fast feedback, most teams decide to run only a subset of all available tests on a regular basis, then run the complete regression testing before major milestones. To select the subset of tests used for the shorter “smoke testing,” teams may use risk-based prioritization, test impact analysis, or other techniques. 

Many regression test suites have a high level of redundancy (tests that do not add any value to the regression test suite). Reducing the level of regression test suite redundancy is a popular way to “slim down” the regression test suite for Continuous Testing. 

Regression Testing Resources