Software testers have made great strides with test automation over the past decade, but some nagging pains remain:
Testing needs to shift left, but UI test automation requires a completed (and stable) UI
Extremely new/old/specialized technologies are unsupported or require extensive customization
Virtual and remote applications are beyond reach
Even highly-resilient tests require occasional attention and updating
App modernization = test destabilization
Enter Vision AI, the latest innovation by Tricentis.
Vision AI is the next generation of automation: deep-learning-driven automation that can see and use the UI like a human would. Human behavior is simulated using various AI and machine learning strategies — for example, deep convolutional neural networks combined with advanced heuristics — to deliver stable, self-healing, platform-agnostic UI automation.
From the tester perspective, you provide a natural language description of what actions to perform, and the engine translates that to the appropriate UI interactions. UI elements are identified based on their appearance rather than their technical properties. If some UI element is redesigned or the entire application is re-implemented using a new technology, it doesn’t matter at all. Like a human, the automation will simply figure it out and adapt.
A look under the hood
Tricentis Vision AI taps intelligent object detection technologies to recognize UI elements. This approach is new to software testing, but it’s used by companies like Tesla to detect objects (other cars, pedestrians, signs, stoplights, trees, etc.) for self-driving cars.
At this point, you’re probably wondering: why the heck would we take technology for self-driving cars and apply it to test automation? Speed and accuracy. A self-driving car must accurately recognize objects in real time. Any delay will cause accidents — and could even result in loss of life. The stakes might not be that high for test automation purposes, but real-time precision is still critical. Slow automation can’t survive in the world of DevOps and CI/CD. Up until now, UI automation tools have been able to process only 1.8 frames per second (on average). For comparison, humans can process 24. Using intelligent object detection, Vision AI processes 40 frames per second. Make sense now?
So, we took intelligent object detection technology and adapted it to detect controls and understand user interfaces. Instead of looking for pedestrians, signs, and stoplights, it can find dropdowns, or tables, or lists, or menus — any control a human can recognize, really.
But detecting the controls is only one part of the challenge. We also need to read the screen in real time. This is where OCR (optical character recognition) comes in. OCR has been around for 20+ years, but it’s still not very fast. Even the OCR developed by industry titans takes seconds to read a screen, but navigating UIs like a human requires real-time character recognition that can react in milliseconds. That’s why we invented a completely new category of AI-driven optical character recognition.
With these technologies, we can see and understand the user interface…fast. Here’s a quick look at how that plays out on a ServiceNow application:
But, with test automation as with self-driving cars, it’s not enough to just see and understand things. You also need to drive. Vision AI was never intended to be a “test automation tool,” but rather a “test automation engine.” As such, it needs to be integrated into something that can provide core capabilities like test data management and test case design. You can think of Vision AI as supercharging your existing tool suite rather than requiring a whole new set of tools. With this added layer of intelligence, your automation is smart enough to work through the vast majority of UI changes — things that inevitably trip up traditional automation, but wouldn’t make a human think twice.
Vision AI is quite smart because it’s been “trained” on 9M controls and over 12M examples. But, like humans, it does sometimes make mistakes — especially when it’s faced with situations that it has never encountered before. In this case, it does what you would hope a reasonable human would do: learn from its mistakes. There are two ways that Vision AI learns: from Tricentis (in supervised training sessions) and from you. Vision AI provides a capability for you to teach it on the fly. The data you provide for training immediately updates our logic, teaching your AI to be smarter. Optionally, you can send Tricentis this data to make the entire solution smarter. We believe in an opt-in model to data privacy, and so your data is never shared with Tricentis unless you give explicit permission.