Analyst research

The Top Automated Testing Tools: 2 Analyst Opinions

Joe Colantonio (@jcolontonio) recently offered some great advice on how to select an automated testing tool:

“There is no “correct” test tool for automation testing. Ultimately, it all depends on your team’s unique needs and skill set. Also, you sometimes need a combination of different tools to get the coverage your application requires. Due to this there are a few things to look for when trying to select a test tool:

  • Look at the product road map and make sure the tools you select will handle future features and technologies.
  • Evaluate the cost, including maintenance.
  • Use a tool that a leverages the same tools and languages your developers use.
  • Don’t just assume a tool will work for you. Create small POC for each tool and get team feedback before committing to anything.
  • Is the tool extensible?
  • How easy is it to use and get started?
  • Does it provide reporting and debugging capabilities?
  • Does it recognize all the objects in your application?
  • Can it integrate with other tools like version control, test management tools, continuous integration tools?
  • Find out if the tool has an active user base.
  • Select tools that other companies are using.
  • How much training will it take to get your teams up to speed with the tool?
  • Finally, determine how easy is it to hire folks that have the skills needed to create your automated tests”

[Read Joe’s complete Automation Testing Resources and Best Practices article for more tips, as well as very complete list of automated testing tools.]

If your organization is starting or scaling your test automation efforts and not yet sure which automated testing tools to evaluate, we encourage you to review two new pieces of research by Forrester and Gartner. Both automated testing tool reports are the culmination of extensive analysis, and provide insight into test automation trends as well as how the different tools compare.

Forrester – Functional Test Automation Wave

Forrester identified the top 11 automated testing tools that provide API testing, cross-browser testing, mobile testing, and UI testing. After evaluating these tools based on vendor interviews, product evaluations, and customer interviews, they scored the tools on 33 criteria and ranked them against one another.

Download the complete Forrester Functional Test Automation Wave report to review the results of Forrester’s automated testing tool comparison, their insights on why API testing and Continuous Testing are critical for meeting business expectations for “quality @ speed,” and their detailed recommendations for test automation.

Automated testing tools covered in this report: Conformiq, Hewlett Packard Enterprise (HPE), IBM, LogiGear, Micro Focus, Microsoft, Original Software, Parasoft SmartBear, TestPlant, and Tricentis.

Here’s a quick excerpt of their key points for the automated testing tools they ranked in the leader category:

  • “Tricentis provides top test automation and optimization design capabilities, test asset reuse, and combined automation. Clients praised Tricentis’ great service continuity and the productivity benefits of the tool in helping them reach high levels of automation.
  • IBM’s leadership position relies on a comprehensive set of features it offers in the UI-based approach and an even stronger one in API testing. IBM recently closed an important strategic product partnership with HCL that transfers the responsibility for developing many its products, including software functional testing tools, to HCL.
  • Parasoft’s focus on developers, API testing, and evolved analytics pays off. Some Parasoft clients highlighted their satisfaction with not only the technical tool’s capabilities and ease of use, but also the vendor’s outstanding support.
  • HPE’s UFT and LeanFT address all kinds of test automation needs. UFT, LFT, and Octane are part of the broader business that HPE Software is spinning off and merging with Micro Focus.”

According to the report:

“We evaluated each tool’s operating environment (including mobile, non-mobile, cloud, and browsers) and its design and execution automation; we placed the heaviest emphasis on API testing, some on UI testing, and to a lesser degree on packaged-app testing. We also evaluated automation execution and orchestration in continuous delivery pipelines, tool qualities, and third-party integrations with nonfunctional testing tools and Agile project management (PM) tools.”

Gartner – Magic Quadrant for Software Test Automation

Gartner evaluated 9 tools that met their criteria for enterprise-grade automated testing tools. The evaluation involved inquiries with Gartner clients, surveys of tool users, vendor responses to Gartner questions, and product evaluations. Gartner required tools to support native Windows desktop application testing and Android or iOS testing support as well as support 3 of the following: responsive web applications, mobile applications, packaged applications, API/web services.

Download the complete “Magic Quadrant for Software Test Automation” report to review the results of Gartner’s automated testing tool comparison, their insights on Selenium and other source of open source test automation tools, and their detailed recommendations for automated testing tools.

Automated testing tools covered in this report: HPE, IBM, Micro Focus, Microsoft, Ranorex, SmartBear, TestPlant, Tricentis, Worksoft, Selenium, Appium, SoapUI.

In their top-level assessment of the automated testing tools they ranked in the Leaders category (Tricentis, IBM, HPE), the report notes that leaders are marked by:

  • Broad adoption, global deployments and integration of other products
  • Loyal customer bases
  • Strong revenue growth and/or high market share
  • Superior vision and execution for emerging and anticipated market requirements, as well as a consistent track record of innovation
  • A strong vision of the needs of the enterprise, support for open source and standards, a solid understanding of IT requirements, and scalable channels and partnerships
  • Breadth across a significant number of supported technologies and platforms, and deliver market-leading functionality in one or more functional areas
  • Solutions that are easy to purchase, use, manage and upgrade, and which can connect to a range of complementary systems — from both the same vendor and third parties

According to the report:

“We have rated innovation as “high” and looked specifically at new and differentiating capabilities in these areas:

  • Technology —for example, visual testing
  • Continuous testing —for example, enabling more-frequent releases through support of modern test practices
  • Support for different roles and skill sets —for example, developer tester, test automation engineer, business analyst

Leaders have a complete vision or an ability to support everything that a quality team will need. They support leading technologies and architectures, with a clear understanding of the shift this introduces to applications and the complexity involved in testing those applications.”