Start your AI agent testing with deterministic tooling

Key Takeaways

• Treat AI agents like outside consultants, not team members. Their work must always be verified by someone you trust
• Start with deterministic testing tools you already have in order to establish a quality baseline before layering in AI-specific capabilities
• AI testing should be continuous, not a one-time checkpoint. Agents can fail unpredictably and need ongoing oversight to stay on track.

By now everyone is aware of the limitations inherent in generative AI and the AI agents that use it to complete their tasks, and the challenges involved in getting them enterprise quality. If you are planning to incorporate AI agents into your enterprise IT architecture, how are you planning to validate their quality and accuracy?

Perhaps you are thinking about using testing agents to ensure your enterprise agents produce correct results? A good way to think about it is as an automation system in which testing validates agentic operations but humans maintain oversight, as they would with any set of external consultants. In his recent blog post on leveling up quality engineering for agentic development, my colleague Jason English identifies three “bosses” to satisfy: Quality, Business Outcomes, and Organizational Risk. In this follow up, I’m going to discuss how to test AI agents to validate enterprise quality and satisfy the “bosses.”

By starting with the test management, test automation, and test data tools that your team already uses, you can establish the right foundation for validating your AI agents. Use your testing tools to establish a baseline of QA metrics, including test coverage, defects, and bugs fixed. Generative AI coding systems and agents rely on non-deterministic LLMs and the code will therefore contain errors, hallucinations, and incorrect results, so it’s an important first step to collect this data.

Are agents people?

It’s a natural tendency to personify an AI agent that’s executing a task for you, or automating a manual task that a person would otherwise be doing. Many organizations think about AI agents as members of an existing team, sometimes even assigning them employee IDs and corporate email addresses. Such agentic team members can assume roles, such as testing, design, documentation, taking and summarizing meeting notes, or serving as a reference for questions about source code and application requirements.

(It’s tempting to imagine the AI agents can think for themselves, since they are capable of accomplishing so much just by processing natural language prompts and instructions, but they cannot be trusted to do anything other than what you task them to do and verify that they have done.)

It’s also common practice to have one set of agents doing the work and another set of agents testing those agents, debugging the agentic code and validating their work output. And like humans, agents make mistakes and someone has to catch and fix them. Just like pair programming or automated CI/CD tests, you can have one set of agents executing a task and another set of agents validating that the task executes correctly.

If an agent’s code passes unit tests, regression tests, integration tests, system tests (if applicable) and meets requirements, then submit it to agent validation tests (typically by testing agents) to confirm its results are correct. Agents that configure pre-prod environments, deploy code, and observe production telemetry are examples of agents you may want to allow to act autonomously, once you validate them sufficiently.

AI agents are like outsourcing roles

Although the tendency to think about an AI agent as part of the team is understandable, it’s also not really accurate. An AI agent can never be a part of a team in the same way a human can. An AI agent is more like an outsourced role assigned to someone outside the organizational trust boundary. You can’t expect them to behave with the same sense of loyalty to the company or dedication to its goals.

The idea of roles that are not really part of your team – untrusted entities, in this case AI agents – doing work for you outside the organization and returning it to you is basically the same thing. You have to give them a task with detailed instructions and specifications. You can’t expect them to have the context of an employee who has worked for the organization for a number of years or even for one year.

When you get work back from a consultant at a systems integrator, for example, you have to be sure that someone you trust is checking the work. And if it turned out that you didn’t provide sufficient details or requirements, the results are not going to be what you asked for. The underlying need is to inject certainty, validation, and confidence into a pipeline in which work is done by untrusted entities without creating significant additional review and governance processes.

The Tricentis agentic testing approach

The Tricentis platform starts with test management basics: qTest for test management, Tosca for end-to-end testing, Data Integrity for data assurance, and Testim for custom web and mobile testing. These basic testing capabilities lay the foundation for AI agent testing by achieving the basic level of quality required to meet business outcomes and mitigate organizational risk. The deterministic tools establish the ground truth for seeding the testing agents and become the skills and tools agents use at the next level of testing.

The next step is to add capabilities specific to AI agent testing, such as the Tricentis AI Workspace, which includes:

• Agentic Test Creation – generates tests from requirements and risk signals, using AI to assess coverage and prioritize tests
• Agentic Test Automation – generates automation scripts from requirements and natural language prompts, using AI to maintain automation and improve coverage
• Agentic Performance Testing – uses AI to design, generate, execute, and analyze performance tests and identify performance bottlenecks

The AI Workspace’s centralized control plane allows you to define the AI agent testing workflow, enforce policies, implement approval gates, and monitor AI testing across the QA team. The AI Workspace governs agentic workflow capabilities. For example, when a Jira ticket is created or code is committed, an agent checks it for failure types, measures how well it performs, and tunes it to improve performance.

The Intellyx take: Continuous oversight is non-negotiable

Traditional testing validates after a change has already occurred. In an AI-driven world, testing has to become a continuous, adaptive system. AI agents are going to fail, they are going to return unexpected results, and they may have to execute their tasks more than once to achieve expected results.

Just like you have to monitor the work of outside consultants, you have to monitor the work of AI agents. If they don’t do the job correctly, you have to have a good way to determine that and tell them to do it again. A comprehensive platform of AI testing tools such as the Tricentis AI Workspace gives you the capabilities you need for AI testing in addition to traditional deterministic testing tools.

Copyright © Intellyx BV. Tricentis is an Intellyx customer. Intellyx retains final editorial control of this article. No AI was used to write this article.