Leveling up quality engineering for agentic development

One day in an agentic developer’s life:

Whether due to competitive pressure or management FOMO, organizations are adopting AI-driven software to better serve customers and employees, automate business processes, and pull together knowledge and analytics for improving business operations.

Unfortunately, the software development function is often stuck on the first level. While more than 80% of companies are already using coding agents for some part of their work, less than half of them have agentic development projects that make it past the pilot phase.

Some vendors are claiming whopping 100-percent-or-higher gains in developer productivity – after all, agents can crank out way more code, faster, so they can release features more frequently. Another MIT study says only 5% of agentic AI projects achieve meaningful ROI – with a handful of high-performing development shops offset by many others who accrue technical debt as they struggle to review more code and attempt to scale in production.

Engineers and vibecoders alike can now choose from an array of AI development tools capable of coding sophisticated applications, and their underlying foundational models keep improving. So why can’t organizations trust the applications they are getting from AI?

Agentic AI is not an incremental change. It represents a new paradigm for software development and testing – a living ecosystem of action-oriented agents that cross organizational boundaries. If you want to advance in the agentic development game, your organization should level up its quality engineering practices.

Responsible development teams need to reassert their domain knowledge and authority over quality to defeat three major forces, or the three “final bosses” that can destroy trust in agentic AI development. Consider this story a walkthrough for the game.

Boss 1: Quality – ensuring accuracy and scalability

As our software delivery practices evolved through Agile and DevOps movements, companies shifted testing activities left in the lifecycle, inserting automated testing within design, development, and deployment phases.

Architectural change further decentralized traditional QA organizations because high-velocity businesses could no longer afford to throw finished releases “over the wall.” Quality engineers, performance engineers, and SDETs started working alongside developers to validate the interactions of distributed services for accuracy and scalability with each iteration.

Now that AI coding assistants and agents are entering the fray, we are seeing increased release frequency and even faster architectural change.

Early on, many companies tried to get ahead of this problem by putting human engineers in a state of constant review of AI code. Human attention to detail is simply unsustainable at the volume agents can generate code – and because AI can write code faster than humans, it can also create bugs faster. Quality becomes a bottleneck, and team morale will suffer.

The strategy for winning the Quality level is visibility.

No matter how good Codex or Claude Code get at generating code, they are still non-deterministic systems that can make mistakes, usually due to the lack of relevance or completeness of the prompts and training data they have been fed with.

When multiple teams and agents are working in parallel, the feedback loop of each delivery cycle can reinforce shortcuts or hallucinations that won’t work in production, such as refactoring tests in order to pass a build, like our hapless agentic developer experienced at the outset of this story.

The only solution for such AI model pollution is visibility: continuous validation and observability that is completely independent of the general training information and assumptions about testing that the models hold.

Visibility requires orchestration of a suite of AI testing tools that conduct specialized test automation at every architecture level (code, supply chain, integration, UI, and now, prompt engineering) as proof.

Tricentis offers capabilities within its agentic quality platform, including quality intelligence so teams can spot gaps in coverage at every release and synthetically generate tests. SeaLights can find non-obvious agent ‘hallucinations’ and block untested changes from reaching production.

Agentic quality test results should be fed back into the agentic AI coding model’s corpus of data as regressions. When human testers can focus attention solely on critical changes, testing will no longer be the bottleneck of every release cycle.

Boss 2: Business Outcomes – maintaining contact between requirements and reality

Quality is key for preventing agentic AI development mistakes, but now we have a far bigger monster to deal with: misunderstandings.

If agentic development requirements are not bound to business outcomes, AI misunderstandings can quietly shift the whole project off course, leading to software with baked-in faults that will never align to its intended business purpose.

We can use prompt engineering or “vibe coding” to give our requirements to a coding agent, and it will likely do a pretty good job of generating a pilot app that seems plausible as a starting point, but behind the surface it still “leaves the wires hanging” like a starship control panel in a cheap 1980’s TV show set.

A lot of the fault here is on us. We’re just not talking to our agents clearly enough.

Remember behavior-driven development, or BDD? This practice gave developers and business people a common framework for describing what “the system shall do” in plain English. Now it’s time to share BDD with our agents, since we already talk to them through natural language prompts.

The strategy for winning the Business Outcomes level is context.

Before we can align agentic development with business requirements, we don’t just need to speak the same language as agents, we need to agree on what the words mean – a shared semantic understanding.

An agent may be very tightly scoped to do a specific task, such as travel reservations. We want to validate the model’s AI abilities for recognizing that a hotel, motel, campsite, or rental condo are all potential answers for lodging inquiries, while understanding that a jail or Waffle House is not valid lodging.

This semantic understanding, or semantic layer, provides context between the training data itself, the user’s prompts, and the intended business outcomes. In quality engineering practices, it allows testers to steadily refine their test suites and data to meet the evolving real-world requirements of the live application.

Since this doesn’t come naturally to everyone, wouldn’t it be helpful if we had a “requirements pre-processor” agent that could take in our prompts, and help us fill in other necessary context the development or test agent would need to validate our own specific business outcome?

Tricentis has agentic test creation and automation solutions that look at the current architecture and codebase of an application in the context of its business requirements. When a human tester asks to enhance a feature or add more tests to validate accuracy and performance, this agent understands the business context, then it navigates the codebase for changes, recommends detailed test criteria, and generates multi-step application and business process tests to send for automated execution.

Results are fed back into the system to further improve semantic understanding and business relevance over time, in the context of the application and its real-world purpose, in order to prevent future misalignment.

Final Boss: Organizational Risk

The final boss of agentic development trust is organizational risk. Players who don’t overcome it will lose their mandate for agentic change, and risky initiatives to adopt AI-driven software will be set aside in favor of maintaining the status quo.

Every organization has different risk tolerances for unique applications and workloads – for instance, a selfie-morphing AI app for social sharing can afford to fail, while an insurance claims photo processing agent in production had better be accurate, compliant, performant at scale, and securely handle private customer data, or lost customers and lawsuits will follow.

Agentic application failures are really only a symptom of a bigger organizational risk: wasting company resources and budget on uncoordinated human effort and out-of-control infrastructure costs that fail to produce measurable ROI.

Who controls the flow of work when humans, systems, and agents are all acting at once, and how can they deliver positive results?

The strategy for winning the Organizational Risk level is governance.

Just keeping humans “in the loop” while agents go off and do their thing will never overcome this final risk challenge. Humans need to be at the center of agentic AI governance.

Agents are only as good as the data they are trained on, so a steady infusion of human knowledge and insights is essential to sustain agentic development and testing.

Since there is not usually a centralized QE group anymore, governance should be federated to stakeholders across the org. No one group or individual at the top of a hierarchy should be a bottleneck for key decisions. This means building cross-functional teams who can control people, processes, and technology through platform engineering practices or a center of excellence for promoting agentic AI development.

“Both humans and AI agents need clear visibility into the ideal customer journey, product flow, employee experience, and execution cadence. They need to know who owns each step and where friction is building up. Without these maps, the system can’t self-optimize — it can only thrash.”— Les Sisney, founder, Organizational Physics

Tricentis recently released an Agentic Quality Engineering Platform that provides an intelligence layer for controlling and monitoring agentic test creation and executing functional and performance test suites. The platform’s shared AI Workspace provides a management layer for orchestrating and observing agent work in process, promoting critical decision points where human oversight is needed.

The Intellyx take

Teams have agents, but humans have agency.

Having worked in the software testing and development space for years, I’ve always said “testing is doing.” So don’t believe any vendor or technology pundit who tells you that we don’t need the work of quality engineers anymore because of agentic testing.

Testing drives future requirements. The good old QA metrics of test coverage, defects discovered, and bugs fixed become more important than ever in this fast-moving space, even if the tests must seemingly cover an infinite number of edge cases that are influenced by agentic development tools and agents using software.

To achieve trust in agentic development, we need to give our teams visibility, context, and governance across a growing virtual workforce of autonomous agents as well as their associated test assets and data.

Copyright ©2026 Intellyx B.V. Intellyx is an industry analysis and advisory firm focused on enterprise digital transformation. Covering every angle of enterprise IT from mainframes to artificial intelligence, our broad focus across technologies allows business executives and IT professionals to connect the dots among disruptive trends. At the time of writing, Tricentis is an Intellyx customer. No AI was used to write this article.