Criteria for Evaluating Software Test Automation Tools

Trying to decide which of the hundreds of available test automation tools is right for your organization isn’t typically a fun process—but it’s a critical one. If you make a bad decision, testing will be blamed for negating the speed gains achieved by Development and Operations. But, if you choose right, you can take credit for accelerating delivery speed, minimizing business risks, and even lowering testing costs.

How do you move past the flashy product demos and decide which tool will really work for your culture, your existing team members, and your application architectures?

One respected rubric for evaluating tools is Gartner’s Critical Capabilities for Software Test Automation.  Each year, Gartner analysts assess how the software testing industry is evolving, then develop a set of criteria for evaluating and comparing software testing tools in light of the latest trends. The report analyzes how the most popular commercial software tools compare versus these criteria—but you can use their framework as a vendor-neutral guide for evaluating and comparing any software testing tools you might be considering.

Top considerations for evaluating test automation tools

Here are the top 10 test automation capabilities outlined in the latest version of the report:

• Match to nontechnical role skills: Suitability for business analysts, subject matter experts, and other non-programmers.

• Match to technical role skills: Suitability for developers and test automation experts.

• Templates and accelerators: Template libraries and wizards that facilitate testing for specific verticals and applications (e.g., SAP S/4HANA, Salesforce….).

• Change impact analysis: Identification of tests to prioritize based on code changes and risk.

• Cross-platform-/browser support: Testing across operating systems, browser types/versions, and mobile devices.

• Intelligent automation: Intelligent automated testing with AI, machine learning, self-healing technologies, bots, and automated test generation.

• Breadth of technology support: Support for various UI and API formats/protocols across desktop, web, mobile, AR/VR, IoT, and full-stack cloud application testing.

• DevOps tool integrations: Support for DevOps, agile planning, and ALM tools.

• Open source support: Support for, and extension of, open source testing tools and frameworks.

• Dashboards and analytics: Tracking and analysis of overall test progress, sprint-level test activities, and individual tests.

Aligning test automation criteria with your expected use cases

The report encourages evaluators to customize the criteria weights based on their unique environment and requirements, but offers the following four “use cases” as a starting point:

• Enterprise End-to-End Testing: Testing end-to-end business processes that might cross SAP, other packaged apps (Salesforce, Oracle EBS…), APIs, ESBs, web/mobile UIs, etc. The primary consideration for this use case is Breadth of Technology Support (18%). Change Impact Analysis and Templates/Accelerators are also important.

• Progressive web apps/response web/native apps: Automated testing on a broad array of end-user browsers and devices. With a weighting of 34%, cross-browser testing dominates this use case. DevOps Tool Integrations and Match to Technical Role Skills are the runners up.

• Continuous Testing: The ability to mitigate risks at every stage of the application delivery pipeline. For Continuous Testing, Gartner places the heaviest emphasis on match to technical role skills (22%) vs non-technical role skills (7%). Open Source Support, DevOps Tool Integrations, and Intelligent Automation are also heavily weighted.

• Intelligent testing: Testing capabilities that take advantage of AI, advanced analytics, machine learning, and similar technologies. Not surprisingly, Intelligent Automation is the primary factor here (25%). Match to non-technical role skills and change impact analysis are also key.

Download your testing tool comparison matrix

To help you compare commercial and open source testing tools, we’ve prepared a fully-customizable testing tool comparison matrix. You can enter your own scoring for each tool (or use Gartner’s), specify how you want to weight various criteria, and even factor in additional criteria that are important to your team/organization.

[Download testing tool comparison matrix]