Test Failure Prioritization and Healing
Goal: Automatically identify and resolve test failures that don’t indicate a problem in the application under test
Every morning, developers and testers grab their coffee and start chipping away at all the test failures reported for the nightly regression tests. Even with stable test cases, the number of test failures increases exponentially as test automation rates, test run frequency, and system complexity are all rising in parallel.
A test failure could be a sign that the latest application changes broke business-critical functionality…or it could stem from a number of other issues. For example, a dependent system (e.g., a third-party application or API) may be temporarily unavailable, broken, or so slow that it’s causing the test to time out. A simulated test environment might not be available or functioning correctly. Or, the test data provisioning process might be feeding the test expired or inappropriate data.
Diagnosing the root cause of a single test failure can take anywhere from minutes to hours. Given that extensive regression test suites can easily produce upwards of 500 test failures, streamlining the diagnosis process could yield significant productivity gains.
