kelvin ross logo

Ross Report 2012

Software Testing Industry Benchmark

Ross Report 2012 – Software Testing Industry Benchmark – lists Tosca Testsuite as second most widely used software test automation tool in Australia and New Zealand.

The “Ross Report 2012 – Software Testing Industry Benchmark” is Australia’s most comprehensive study of trends in the software testing and assurance industry. The respondents included stakeholders in software quality, assurance and testing, such as test managers and test directors. The study identified specific performance benchmarks across six major industries throughout Australia. Tosca Testsuite was listed behind HP Quick Test Pro as the second most widely used test automation tool.

Key Survey Findings

According to the Ross Report, “a significant benefit of test automation is the reduced cost to re-execute an automated test compared to a manual test.” Furthermore, “Investment in automation will increase, supporting plans to increase automation coverage. Further investment is planned by over 60% of organizations (surveyed), and 11% are planning significant investment.”

The report highlights typical test failure rates.  On average, respondents anticipate that 20% of test cases will fail. This figure, however, drops during retesting by more than half to 8% . Regression testing has a similar failure rate at 7.59%.  Thus automation may be a good solution for normalizing the cost per defect, both for new functions and regression.

The key findings of the 2012 Ross Report include the following:

  • Organizations intend to significantly increase their level of test automation coverage.
  • Future adjustments in budgets will show a correlation between the increased use of tools and a reduction in staffing costs
  • Testing effort is focusing more on analysis instead of test execution. In the future more effort will be put into test management.
  • Most testing consists of functional testing (80%), while the budget for non-functional testing remains limited (20%).
  • The proportion of the average IT budget spent on testing is stabilizing at 20-25%.
  • It is very difficult to convince business stakeholders to include software testing in the budget.
  • Organizations are poor at finding and fixing defects in early lifecycle stages, with most defects being found in the testing phase or later, limiting opportunities for early defect correction.
  • At current levels of detection, the introduction of more appropriate early detection techniques could reduce the costs resulting from finding and fixing defects by 40% and project costs by 25%.