AI agent reliability: why internal evals miss real failures

On July 14, 2026, at the VB Transform conference in Menlo Park, Bryan Silverthorn, Director of AGI Autonomy at Amazon, argued that what blocks enterprise agent deployment is reliability rather than model capability. Cisco data cited on stage puts numbers on it: 85% of enterprises are piloting agents, and only 5% have shipped them to production. A VentureBeat Pulse survey from June 2026 supplies the other half of the picture, with half of the organizations running evaluations reporting that they deployed an agent feature that passed internal tests and still caused a customer-facing failure. For any company planning a deployment, that leaves one question to answer before launch: how do we know our test measures what actually breaks in front of a customer.

What happened

VB Transform 2026, a conference focused on running agents inside enterprises, took place on July 14 and 15, 2026 in Menlo Park. Speaking there, Bryan Silverthorn of Amazon's AGI lab described why companies stall at the pilot stage. Silverthorn joined Amazon through its acquisition of Adept AI and leads multimodal agent training. His argument was that better benchmarks will not close the gap, because they measure something different from what fails in practice.

Instead of a single verdict that an agent works well, he proposed splitting reliability into four dimensions, a framework he credited to research from Princeton:

• consistency, whether the same input produces the same result,
• robustness, whether the agent copes with input that departs from the typical,
• predictability, whether you know in advance where the agent will fail,
• safety, whether the agent stays inside the scope it was granted.

VentureBeat supplied the numbers from its own Pulse survey, fielded in June 2026 among 157 respondents at organizations with more than 100 employees. Half of the companies that run evaluations at all had deployed an agent or LLM-backed feature that passed internal testing and then caused a customer-facing failure. Two thirds already allow production changes to ship on automated evaluation alone, with no human in the loop, or are deliberately engineering toward that. At the same time only one company in twenty fully trusts that automated evaluation, and the most cited reason for distrust, named by 29% of respondents, is the gap between test results and real outcomes.

A separate study from the same publisher points to where the errors originate. Within the past six months, 57% of enterprises traced an agent answer that was delivered with full confidence and was wrong, back to missing or inconsistent business context on their own side.

Why it matters for business

In most organizations the spending order is the reverse of the risk order. Budget goes into building the agent, not into establishing whether it can be released safely. The result is a pilot that impresses in a demo and never leaves one department, because nobody can take responsibility for its behavior across 300 cases a day.

The second reason is financial. Gartner estimates that by 2030 up to 234 billion USD of enterprise application spending will be exposed to what it calls agentic arbitrage, where agents complete work across several systems and reduce the need to touch traditional interfaces. George Brocklehurst of Gartner summarized the shift by saying companies are no longer buying software primarily for people and are increasingly buying it for agents. Once an agent operates a process, its quality stops being a matter of user convenience and becomes a matter of operating cost and accountability toward customers.

The third reason concerns where responsibility sits. When an agent's output reaches a customer without review, risk moves from the technical team to customer service, sales, and finance. Without a quality threshold defined in advance and a named process owner, that transfer happens quietly, and the company learns about it from complaints.

Business use cases

First-line customer service

Business problem. Repetitive questions about order status, return terms, and delivery dates consume most of the team's time, and response times stretch during sales peaks.

AI solution. An agent answering from the company knowledge base and order system data, with automatic escalation to a person for anything outside its scope.

Data and processes required. A current knowledge base with a single content owner, read-only access to order status, an archive of three to six months of real tickets as the test set, and an explicit list of topics the agent does not handle.

Potential effect. Shorter first response times and a team redirected toward cases that need a decision.

Risk. The agent will answer confidently and incorrectly whenever the knowledge base holds two conflicting versions of a procedure. That is exactly the mechanism 57% of enterprises reported to VentureBeat. Clean up the source before you launch the agent.

Quotes and pricing preparation

Business problem. Preparing a quote means pulling together the price list, customer history, and commercial terms, which stretches the response to an inquiry by several days.

AI solution. An agent assembling a draft quote from the inquiry, the price list, and the account history, handed to a salesperson for approval.

Data and processes required. One current price list as the source of truth, discount rules written down explicitly, CRM history for the account, and mandatory human approval of the final price.

Potential effect. Faster response to inquiries and fewer quotes written from scratch.

Risk. Pricing is irreversible in the customer's eyes. A discount error costs either margin or credibility, so this process should not run without human confirmation even when the agent passes its tests.

Invoice and document processing

Business problem. Manually retyping invoice data and matching it to purchase orders consumes finance team hours and produces errors during month-end close.

AI solution. An agent that reads documents, fills fields, and proposes coding, with a rule that routes anything above a threshold amount or mismatched against an order to a person.

Data and processes required. A set of historical documents with their correct coding as the test base, a supplier list, an amount threshold for manual approval, and a log of every agent decision.

Potential effect. Less manual data entry and a shorter month-end close.

Risk. A high average accuracy hides errors in rare document formats. Measure quality separately for typical and atypical cases, which is the robustness dimension described at VB Transform.

Sales meeting prep and CRM hygiene

Business problem. Salespeople walk into meetings without a full account picture, and notes reach the CRM late or never.

AI solution. An agent that builds a pre-meeting briefing from contact history and prior commitments, then proposes a record update after the meeting for one-click approval.

Data and processes required. A tidy CRM with one record per account, access to correspondence history, and a defined set of fields the agent may change against fields that stay read-only.

Potential effect. Better prepared meetings and fuller CRM data without extra admin work.

Risk. An agent writing directly into the CRM can pollute the database faster than the team notices. Restrict writes to selected fields and keep confirmation with the salesperson.

What companies can do now

Build the test set from real cases. Take 50 to 100 tickets, inquiries, or documents from recent months, together with how a person resolved them. That is the only test that measures reality. A set invented while building the agent measures the project team's idea of reality.

Define the quality threshold before launch, not after. Write down the number at which the agent goes to production, for example the share of cases closed without correction plus zero errors in the irreversible category. Without that number, the deployment decision rests on how the demo felt.

Separate reversible from irreversible actions. A draft reply, a summary, and a proposal sit on one side. Sending a priced quote, issuing a document, and changing customer records sit on the other. Keep a human in the loop where an error cannot be undone, rather than everywhere.

Clean the data before the agent runs on it. Two versions of a procedure, an outdated price list, and three records for the same customer turn into answers delivered with confidence and wrong. Naming one source of truth per domain costs less than repairing the consequences.

Log and review. Store the input, the response, and the agent's decision, and schedule a recurring review, for example weekly through the first quarter. Reviewed errors become the next batch of test cases.

Name a process owner. Not a tool owner, but the person accountable for the outcome of the process the agent works in. That person sets the quality threshold, approves scope expansion, and can switch the agent off.

Measure business outcomes, not correctness alone. Handling time, share of cases without correction, escalation counts, and the cost of an error tell a board more than a one-to-five rating of answer quality.

Risks

A test that measures the wrong thing. The largest exposure is not the absence of testing, it is testing that produces false confidence. Half of the companies in the VentureBeat Pulse survey shipped a feature after passing tests and saw a failure reach a customer.

Automating the release decision. Two thirds of surveyed organizations allow deployment without a human in the loop, against five percent who fully trust automated evaluation. That gap is where incidents are created.

Data security and shadow AI. When the official agent is slow or does not cover a frequent case, employees revert to personal tools and paste company data into them. Match the agent's scope to how people actually work, or the deployment opens a new leak channel.

Maintenance cost. The test set, the logs, and the recurring review are a standing cost that does not appear in a pilot quote. Budget for it upfront, otherwise it resurfaces as the reason to wind the project down two quarters later.

No process owner. An agent without someone accountable for the outcome runs until the first incident, after which nobody holds the mandate to fix it or shut it down.

Changes without a retest. Swapping the model, updating the prompt, or editing the knowledge base can change agent behavior in an area nobody touched. Each of those changes should trigger the same test set.

Key takeaways

The barrier to deploying AI agents has moved from what models can do to whether a company can prove the agent will behave correctly in front of a customer. 85% of enterprises are piloting agents, 5% run them in production, and half of the organizations running evaluations saw a failure after their internal tests passed.

A company that wants to move past the pilot needs three things before launch: a test set built from real historical cases, a written quality threshold, and a process owner with the authority to switch the agent off. Separating reversible from irreversible actions determines where the human stays in the loop.

Cleaning up data is a precondition, not a follow-up phase. Conflicting procedures and outdated price lists turn into answers delivered with confidence and wrong, which 57% of enterprises confirmed over the past six months.