A new vision for test automation: Vision AI

Software testers have made great strides with test automation over the past decade, but some nagging pains remain:

• Testing needs to shift left, but UI test automation requires a completed (and stable) UI
• Extremely new/old/specialized technologies are unsupported or require extensive customization
• Virtual and remote applications are beyond reach
• Even highly-resilient tests require occasional attention and updating
• App modernization = test destabilization

Enter Vision AI, the latest innovation by Tricentis.

Vision AI is the next generation of automation: deep-learning-driven automation that can see and use the UI like a human would. Human behavior is simulated using various AI and machine learning strategies — for example, deep convolutional neural networks combined with advanced heuristics — to deliver stable, self-healing, platform-agnostic UI automation.

From the tester perspective, you provide a natural language description of what actions to perform, and the engine translates that to the appropriate UI interactions. UI elements are identified based on their appearance rather than their technical properties. If some UI element is redesigned or the entire application is re-implemented using a new technology, it doesn’t matter at all. Like a human, the automation will simply figure it out and adapt.

A look under the hood

Tricentis Vision AI taps intelligent object detection technologies to recognize UI elements. This approach is new to software testing, but it’s used by companies like Tesla to detect objects (other cars, pedestrians, signs, stoplights, trees, etc.) for self-driving cars.

At this point, you’re probably wondering: why the heck would we take technology for self-driving cars and apply it to test automation? Speed and accuracy. A self-driving car must accurately recognize objects in real time. Any delay will cause accidents — and could even result in loss of life. The stakes might not be that high for test automation purposes, but real-time precision is still critical. Slow automation can’t survive in the world of DevOps and CI/CD. Up until now, UI automation tools have been able to process only 1.8 frames per second (on average). For comparison, humans can process 24. Using intelligent object detection, Vision AI processes 40 frames per second. Make sense now?

So, we took intelligent object detection technology and adapted it to detect controls and understand user interfaces. Instead of looking for pedestrians, signs, and stoplights, it can find dropdowns, or tables, or lists, or menus — any control a human can recognize, really.

But detecting the controls is only one part of the challenge. We also need to read the screen in real time. This is where OCR (optical character recognition) comes in. OCR has been around for 20+ years, but it’s still not very fast. Even the OCR developed by industry titans takes seconds to read a screen, but navigating UIs like a human requires real-time character recognition that can react in milliseconds. That’s why we invented a completely new category of AI-driven optical character recognition.

With these technologies, we can see and understand the user interface…fast. Here’s a quick look at how that plays out on a ServiceNow application:

 

But, with test automation as with self-driving cars, it’s not enough to just see and understand things. You also need to drive. Vision AI was never intended to be a “test automation tool,” but rather a “test automation engine.” As such, it needs to be integrated into something that can provide core capabilities like test data management and test case design. You can think of Vision AI as supercharging your existing tool suite rather than requiring a whole new set of tools. With this added layer of intelligence, your automation is smart enough to work through the vast majority of UI changes — things that inevitably trip up traditional automation, but wouldn’t make a human think twice.

Vision AI is quite smart because it’s been “trained” on 9M controls and over 12M examples. But, like humans, it does sometimes make mistakes — especially when it’s faced with situations that it has never encountered before. In this case, it does what you would hope a reasonable human would do: learn from its mistakes. There are two ways that Vision AI learns: from Tricentis (in supervised training sessions) and from you. Vision AI provides a capability for you to teach it on the fly. The data you provide for training immediately updates our logic, teaching your AI to be smarter. Optionally, you can send Tricentis this data to make the entire solution smarter. We believe in an opt-in model to data privacy, and so your data is never shared with Tricentis unless you give explicit permission.

Interesting…but what’s the real value to testers and testing?

This probably isn’t the first time that you’ve heard someone talking about “AI in software testing.” But here are some firsts that add real value to your software testing practice:

You can build UI automation before a UI even exists

You can build automation before a UI exists. Vision AI can take a simple definition — like a textual requirement definition, a mockup, or even a whiteboard drawing — and construct a running automation case based off of it. With this “shift left,” in-sprint UI test automation is finally within reach.

Your tests can withstand app modernization

When you upgrade your application, Vision AI’s automation doesn’t break. And I’m not talking about like a minor upgrade, like you’ve changed some technical IDs or something like that. I’m talking about pretty major upgrades like moving from one technology to another or upgrading your JavaScript library from an old jQuery UI to a new Angular Material Design.

Citrix? Customizations? Old tech? new tech? No problem.

If you can see it, Vision AI can automate it. Vision AI works on any visual interface. Since it’s using the visual interface of the technology connection, it can work through things like Citrix and it can work on Remote Desktop. It can work when you’re not even connected to the machine that you’re looking at (for example, you’re viewing it through an RDP interface). It works on interfaces using technology that’s not available anymore — things like outdated/deprecated versions of Gupta, Silverlight, and Flash (we know they’re not fashionable, but they’re still out in the world today). It also works with very new technologies that aren’t yet supported by (most) automation tools: for example, Flutter, Blazor, and Electron.

It’s so easy, your grandmother could do it

With Vision AI, test automation is really, REALLY easy. In fact, one of our guiding principles for the project was “So easy, your grandmother could do it.” Since Vision AI sees interfaces the same way that humans do, you can define the automation just like you would describe it to another human. For example:

Click the ‘Apply Now’ button.

Enter “David” in the ‘First Name’ field.

Enter “Colwell” in the ‘Last Name’ field.

Select “Australia” from the ‘Country’ dropdown.

…

But, as testers know, the true pain of test automation is in the maintenance: changing XPaths, dealing with dynamic IDs…probably not things your grandmother would want to tackle. With our self-healing execution, tedious technical tune-ups aren’t required.

Vision AI: See it to believe it

I know, talk is cheap. Here are some concrete examples that demonstrate how we really achieve the four benefits I mentioned above…

 

And here’s a fun little montage of even more things that Vision AI can do…

 

Ready to give it a spin? Vision AI works along with existing test execution platforms, such as:

• Tricentis Tosca (public preview now + general availability November 30)
• Tricentis qTest (beta)
• Visual Studio Code (alpha)

In other words, it’s readily available to Tricentis Tosca users, but you don’t need Tosca to benefit from Vision AI.

My team and I would be happy to guide you through the experience; just contact us and we’ll get you on the road to the next era of test automation. And as you start exploring Vision AI, please keep in mind that your honest feedback is incredibly valuable to this major new technology’s advancement. If you find areas where you think our AI could be smarter, or see opportunities for new adaptations, please let us know!

To learn more about Vision AI, watch this conference presentation.