Gartner just released a new report comparing 13 software test automation tools: “Critical Capabilities for Software Test Automation 2018.” While the “Magic Quadrant for Software Test Automation 2018” evaluates the major vendors used to accelerate and scale functional test automation, “Critical Capabilities for Software Test Automation” focuses on how the test automation tools themselves compare on the 10 critical capabilities that Gartner selected as the most critical points of comparison.

[Download the Critical Capabilities for Software Test Automation report now]

The report evaluates the tools’ “effectiveness in addressing users’ needs in 4 use cases:

  • Continuous Testing: The ability to achieve continuous quality and mitigate risks before progressing to subsequent software development life cycle stages by offering support for automating the activities of each stage.

  • Intelligent Testing: The ability to provide advanced, self-optimizing and adaptive automated testing through the usage of advanced analytics, AI, machine learning, self-healing heuristics automatic or similar technologies.

  • Progressive Web Apps/Responsive Web Design/Native Mobile Testing: The ability to simplify the automated testing of native, progressive and responsive web applications.

  • Enterprise End-to-End Testing: The ability to simplify testing that involves a set of systems or applications in various technology stacks that must interact with each other.”

Tricentis received the highest product score of all vendors for the Enterprise End-to-End Testing (3.84/5) use case, as well as for the Progressive Web/Responsive Web/Native Apps (3.83/5) use case. Tricentis also received the second and fourth highest Product Score for the other two use cases named in the report.

The report says “Application leaders who are developing strategies for digital business must address a wide range of testing practices covering increasingly complex technology stacks.”

We believe that Gartner’s emphasis on the expanding the breadth of automated testing will only grow in importance as enterprises increasingly make SAP and related platforms the centerpiece of their digital transformation strategy. As organizations modernize their SAP architecture (e.g., move to S/4HANA), they connect it to all the applications, devices, and users involved in transactions that promote digital transformation. This has distinct impacts in terms of testing speed and scope.  

The New Imperative End-to-End Test Automation

When you look at regression testing for SAP and the apps that are connected to SAP, it’s logistically impossible to complete the required scope and frequency of end-to-end testing with a predominantly manual testing process. Teams need to test much more frequently (and thus much faster) as organizations move from annual or biannual SAP upgrades to a monthly or biweekly update cadence. This is a challenge in itself.

However, in addition to testing faster, teams also have to test more. As we believe the Gartner report underscores, end-to-end testing involves testing a transaction that might cross multiple interfaces, including mobile devices, APIs, ESBs, other packaged applications, the data layer, and more. This requires the tester to know how to configure and validate testing for each associated technology, as well as pass data between the layers to ensure that the test replicates a realistic user experience.

Transforming  Enterprise End-to-End Testing: 5 Real-World Examples

Taking testing to this new level is not trivial. Yet, with the right approach to enterprise end-to-end test automation, it is certainly feasible.

Here’s a look at how several industry-leading organizations have modernized their enterprise end-to-end testing processes for Agile, S/4HANA migrations, DevOps, and other digital transformation initiatives ….

Gartner Disclaimer

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Tricentis.

Gartner Inc, “Critical Capabilities for Software Test Automation,” Joachim Herschmann, Thomas E. Murphy, Jim Scheibmeir, 18 December 2018.