The contact center vendor demo is a controlled environment. Clean intents. Predictable turns. No policy ambiguity, no mid-conversation sentiment drop, no customer who has already called twice and is out of patience. It is designed to impress, not to replicate.

That gap is where agentic AI deployments fail.

TL;DR: To test agentic AI before it talks to real customers, run seven tests: replay, repeatability, multi-turn, adversarial, stack access, escalation quality, and disclosure audit. Each one catches a failure mode a standard demo will not. Score repeatability across eight consecutive runs rather than one, and verify the agent actually writes to your systems rather than talking about them. Every test below has a pass condition you can set before you start.

These are the seven tests we use, and the conviction behind them is worth stating up front. Every agentic AI deployment we have watched fall over broke in the same region of the system: underneath the conversation. Integrations that could only read. Context that died at the handoff. Refund limits written into a prompt, where a patient customer could talk past them. The talking was rarely the hard part, and it is the only part most evaluations measure.

That is why UJET built AXO the way we did, and why tests 4 through 7 below go further than most checklists. Take the tests anyway. Run them on your current platform, on the vendor in your pipeline, and on us.

Why does agentic AI pass the demo and fail in production?

Because a demo measures capability once, and production measures reliability every time.

Salesforce AI Research's CRMArena-Pro benchmark (arXiv:2505.18878, May 2025) found leading LLM agents achieved around 58% success on single-turn service tasks, dropping to roughly 35% in multi-turn settings that require context retention and sequential action. The same paper found agents exhibit near zero inherent confidentiality awareness, and that prompting them into discretion often compromises task performance.

τ-bench (arXiv:2406.12045, June 2024), a separate framework simulating real retail and airline service interactions, found state of the art function-calling agents succeeded on under 50% of tasks on a single run. It then introduced pass^8: the probability the agent gets the same task right on all eight attempts. In retail, that number was under 25%.

Side-by-side comparison showing an AI agent succeeding on under 50% of tasks on a single run versus under 25% across all eight consecutive runs, labelled as what a demo shows you versus what production asks of it.

Read that as an ops director rather than a researcher. Fewer than one in four tasks your agent can do, it can do reliably. The other three are a coin flip you are handing to a customer.

The second number is the more useful one. Multi-turn performance does not collapse because the model degrades between turn one and turn four. It collapses because context, state and system access have to survive the gap between turns, and in most deployments they do not. The benchmark says model. It is measuring plumbing.

Your customers do not speak in single turns. Your demo did.

What does an untested agent actually cost?

Agentic AI failure is not always dramatic. Sometimes it is a refund that should not have been issued. Sometimes it is a policy confidently misquoted. Sometimes it is a customer who escalated and had to restart from zero.

But the record is already building, and the consequences are on the books.

It costs a legal precedent. In Moffatt v. Air Canada (2024 BCCRT 149), decided February 14, 2024, the British Columbia Civil Resolution Tribunal ordered Air Canada to pay CAD $812.02 after its chatbot invented a bereavement fare policy that contradicted the airline's own published terms. Air Canada argued it was not responsible for information provided by its chatbot. Tribunal member Christopher Rivers replied: "This is a remarkable submission. While a chatbot has an interactive component, it is still just a part of Air Canada's website."

Your bot is you. A tribunal said so, on the record, two years ago.

Two-panel contrast showing Air Canada's argument that its chatbot was a separate entity responsible for its own statements, beside the tribunal's ruling that a chatbot is still just a part of the airline's website.

It costs customers, quietly. On April 19, 2025, developer tool company Cursor had an AI support bot signing itself "Sam" tell users that forced logouts were "expected behavior" under a new single-device login policy. No such policy existed. The bot invented it. Users cancelled subscriptions on public forums before anyone at Cursor knew what the bot had said. Cofounder Michael Truell confirmed the incorrect response came from a front line AI support bot (Fortune, April 19, 2025). The bot was also not disclosed as AI, which is now its own exposure.

And it costs the program itself. Gartner predicted on June 25, 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. In the same release, Gartner estimated that of the thousands of vendors marketing agentic AI, only about 130 are real, naming the pattern "agent washing": rebranding assistants, RPA and chatbots as agentic without substantial agentic capability.

The pattern across all three: failure surfaces faster than internal review catches it. Customers experience it first.

The seven tests to run before go-live

Run these in order. Each one catches a failure mode the previous one misses.

Test 1: The replay test. Can it handle the calls you already lost?

Do not write synthetic test cases. Pull real ones.

Anthropic's engineering guidance on agent evaluation, published January 9, 2026, is blunt about the starting volume: "20-50 simple tasks drawn from real failures is a great start." Draw them from a wider pool of real conversations, weighted toward your escalation queue, your CSAT detractors and your repeat-contact tickets. Those are the interactions that already beat your current setup. Expand toward 100 as your coverage grows.

Then score the agent against what your best human agents actually did, not against an abstract ideal.

The quality bar for each test case: write the pass condition so precisely that two of your QA leads, working independently, reach the same verdict. If your own people disagree on whether the agent passed, the test is broken before the agent is.

Pass condition: independent graders agree on every pass/fail definition before a single run, and the agent matches or exceeds human resolution quality on at least 80% of straightforward intents while correctly escalating ambiguous and high-risk ones.

Test 2: The repeatability test. Can it do it again?

Run the same 20 conversations through the agent eight consecutive times. Score each run independently, then score the cases where it succeeded every single time.

This is τ-bench's pass^8 methodology, and it is the single highest value change most contact centers can make to their AI evaluation. It costs nothing but compute. A system that resolves correctly 70% of the time fails a customer roughly one interaction in three. That is not a pilot. That is a liability.

Single-run scoring is why agents look production ready in a pilot and fall apart in week two. Consistency is the product. Capability is just the ticket to the evaluation.

This is a number you generate, not one a vendor hands you. Nobody publishes a pass^8 figure, because it depends entirely on your intents, your policies, your data and your integrations. Any vendor who quotes you one is quoting a number from their environment, not yours. Running it yourself, before go-live, is the only way to actually have it.

Pass condition: consistent, policy-compliant outcomes across all eight runs with no meaningful variance. Set your floor before you start. If your agent clears 60% pass^8 on your top 20 intents you have something. If it clears 90%, you have something you can scale.

Test 3: The multi-turn test. Can it survive a conversation?

Design 15 to 20 scenarios that force the agent to retain context across three or more turns, change direction mid-conversation, and handle a customer who changes their mind.

Test with personas, not prompts. The customer who gives the wrong order number first and corrects it. The customer who asks about two issues at once. The customer who is polite but brief, and the customer who is neither.

CRMArena-Pro's drop from roughly 58% to roughly 35% is what happens when you skip this. That gap is where most contact center conversations live.

Pass condition: the agent maintains accurate context through turn three and beyond, including at least one user correction, without contradicting earlier statements, losing the original intent, or asking the customer to repeat information already given.

Test 4: The adversarial test. What happens when someone tries?

Attempt to break the agent deliberately: prompt injection, jailbreak attempts, policy boundary probing, and edge cases technically within scope but designed to produce harmful output.

In December 2023, a customer told a Chevrolet dealership's chatbot to agree with anything he said, then got it to offer a 2024 Tahoe for one dollar with the words "no takesie backsies." That was not a hallucination. It was a prompt injection, and it is the most reproducible class of AI failure there is.

Microsoft's AI Red Teaming Agent documentation (updated August 19, 2026) scores this with a metric worth borrowing: Attack Success Rate, the percentage of successful attacks over total attempts, not just attempts logged. It scans agent-specific categories including prohibited actions, sensitive data leakage, task adherence and indirect prompt injection, using tiered strategies from simple encoding through multi-turn and crescendo attacks.

You do not need Microsoft's tooling to start. You need someone on your team whose job for one afternoon is to break the agent on purpose, and a written record of what worked.

Where a guardrail lives determines whether it holds. Written into a system prompt, it is an instruction to a language model, and a patient user can eventually talk past it. Written into code the model cannot reach, it holds under any phrasing a customer can invent.

So skip the broad question about how a vendor handles prompt injection and ask a narrow one: where does the refund limit physically live? If the answer is the system prompt, you have learned what you needed to.

Pass condition: the agent refuses out-of-scope requests without hallucinating a workaround, and Attack Success Rate is zero for unauthorized financial action, policy invention and PII disclosure. Anything above zero on those three is a hard block.

Test 5: The stack test. Can it actually do the thing, or only say it?

This is the test most vendors skip in demos, and the one that matters most in production.

Give the agent a task that requires real action. Not explaining the return policy. Processing the return: read the order in the CRM, verify eligibility against the actual record, issue the credit, generate the label, write the disposition back, and tell the customer what it did. Then audit every system it touched.

Language quality masks orchestration weakness. Most "agentic" deployments fail here instantly, because the agent has read access to a knowledge base and nothing else. It talks about work. Your human agent still does the work, across every system the resolution touches, while the customer waits.

That is the architecture problem UJET built AXO (Agentic Experience Orchestration) to solve: agentic AI that orchestrates across the systems in your stack while the conversation is still happening, instead of AI that summarizes it afterward. That distinction is testable in about twenty minutes, and it is the fastest way to detect agent washing in a vendor demo. Gartner's estimate that only about 130 of thousands of agentic AI vendors are real is not a mystery once you start asking for write receipts. Most of that catalog cannot produce one.

Decision flow taking one real customer interaction through three checks on whether the CRM record changed, the credit was issued and every write was correctly attributed, sorting the result into either a very confident FAQ or agentic orchestration.

Pass condition: the agent completes system actions accurately, every system of record shows a correct and correctly attributed write, write failures are handled gracefully, and no orphaned or duplicate records are created. If the agent cannot write, it is not an agent. It is a very confident FAQ.

Test 6: The escalation test. Does it know when to stop?

Design five scenarios that should trigger a human handoff: low confidence, explicit customer request, detected frustration, regulated topic, and a case where the agent has already failed once in the same conversation.

OpenAI's practical guide to building agents names the two triggers that belong in every contact center deployment:

"Exceeding failure thresholds: Set limits on agent retries or actions. If the agent exceeds these limits (e.g., fails to understand customer intent after multiple attempts), escalate to human intervention."

"High-risk actions: Actions that are sensitive, irreversible, or have high stakes should trigger human oversight until confidence in the agent's reliability grows."

Write your thresholds down before launch. Two failed intent recognitions and it hands off. Any refund over your defined ceiling routes to a human. Any account change involving credentials or payment method routes to a human. Any detected distress routes to a human immediately, and warm.

The handoff itself is a test case. Run it. Does the receiving human get the transcript, verified identity, the systems already touched and what the agent already promised? Or does your customer start over? Zendesk's CX Trends 2026 research, fielded across 11,000+ respondents in June 2025 and published November 18, 2025, found 74% of consumers are frustrated when required to repeat information.

The escalation is not the failure. The failure is an escalation that arrives without context.

Confidence scores drift. Permissions hold. The durable version of this is a permission attached to each individual action: some the AI runs alone, some require a supervisor to approve before they execute, some stay human only. Built that way, chaining a harmless lookup to a risky refund cannot launder the refund into autonomy, because the chain inherits the strictest permission in it.

Pass condition: the agent transfers correctly in all five scenarios, and the receiving human has full conversation context, customer history and a stated reason for escalation without asking the customer to start over.

Test 7: The disclosure and audit test. Can you prove what it said?

This one is a legal obligation now. Article 50 of the EU AI Act requires that people are informed they are interacting with an AI system, from the start of the first interaction, in a clear and distinguishable manner, unless it is obvious. Per the European Commission, Article 50 applies as from 2 August 2026. That was one month ago. The "obvious" exception is to be interpreted narrowly, measured against a reasonably well informed and observant person. A bot signing its emails "Sam" does not clear that bar.

In the United States, California's B.O.T. Act (Bus. & Prof. Code §17941) has since 2019 made it unlawful to use a bot to mislead about its artificial identity in order to incentivize a commercial transaction, with a safe harbor for clear and conspicuous disclosure. Utah's amended AI Policy Act (effective May 7, 2025) requires disclosure on clear and unambiguous request, with proactive disclosure for high risk interactions involving sensitive personal information.

Beyond compliance, auditability is what lets you investigate a failure after the fact. If you cannot reconstruct what the agent did and why, you cannot fix it.

Pass condition: disclosure fires on turn one and on direct question. Every session is logged in a retrievable format including tool calls and the reasoning path, not just the output. Your team can reconstruct any conversation within 24 hours of a complaint.

Quick reference: the seven tests

Test	What it catches	Pass condition
Replay	Real-world input failure	Graders agree on definitions, 80%+ on straightforward intents
Repeatability	Inconsistency across runs	Consistent outcomes across all 8 runs
Multi-turn	Context collapse	Accurate context through turn 3+, one correction handled
Adversarial	Prompt injection and policy drift	Zero attack success on financial action, policy invention, PII
Stack	Orchestration failure and agent washing	Clean writes, graceful failures, no orphaned records
Escalation	Bad handoffs	Full context transferred, no customer restart
Disclosure and audit	Compliance and accountability	Turn-one disclosure, logged reasoning, reconstructable in 24h

What you are really testing

Two-layer diagram separating tests one to three as the model layer, covering language and reasoning, from tests four to seven as the stack layer, covering guards in code, write access, handoff and audit, where production breaks.

Most pre-launch failures are not language failures. They are system failures, write-access failures and handoff failures. The agent sounds right. The problem is what it does, or fails to do, underneath the conversation.

This is why the stack test and the escalation test carry the most weight. Language quality is relatively easy to optimize. Reliable orchestration across real enterprise systems, under real load, with real edge cases, is where the gap between a polished demo and a production-ready agent becomes visible.

It is also why this belongs in evaluation, not just in launch prep. A test suite that only measures what the agent says will confirm a purchase you should not make.

The right question to ask any vendor is not "can your agent handle this scenario?" It is: "what does your agent actually do to my systems when it handles this scenario, and what happens when it cannot?"

That is an architectural question, and the answer tells you whether you are buying an agent or a very sophisticated interface. UJET built AXO for the first answer: agentic AI that acts on the CRM, the order system and the billing record while the conversation is still happening. The agent was never the problem. It was the stack behind them.

A confident response that cannot act is not automation. It is theater.

Does testing stop at go-live?

No. Passing all seven tests earns you a go-live decision. It does not earn you permanent trust.

Re-run your replay and adversarial suites any time the model changes, your prompts are updated, a policy shifts or your product catalog expands. Each of those events introduces failure modes your original test set never covered. Microsoft's red teaming guidance treats this as a four stage lifecycle: design, development, pre-deployment, and post-deployment with scheduled continuous runs. Treat a drop in pass^8 the way you would treat a drop in first contact resolution, because it is the same thing arriving earlier.

Four numbered events that invalidate prior AI agent test results, a model update, a prompt change, a policy shift and a catalog expansion, above a timeline showing testing continuing as a cycle rather than ending at a gate.

The deeper problem is what you watch between test runs. Standard QA sampling was defensible when humans handled every conversation. If you are reviewing 5% of conversations, you are blind to 95% of them. That is not a monitoring strategy. It is a hope strategy.

Spiral by UJET was built around this reality: conversational analytics that reads 100% of interactions and turns them into decision-grade intelligence, so a failure pattern surfaces as a trend rather than as a complaint. The principle applies to any post-launch monitoring approach. You need to read everything, not sample it.

Test before launch. Watch everything after.

Frequently asked questions

How do I test agentic AI before it talks to real customers?

Run seven tests: replay 20 to 50 real recorded failures, score consistency with pass^8 across eight runs rather than single runs, test multi-turn conversations with personas, red team for prompt injection, verify the agent actually writes to every system of record, define escalation thresholds before launch, and confirm turn-one AI disclosure with retrievable transcripts.

What is pass^8 and why does it matter more than accuracy?

pass^8 is the probability an agent succeeds on all eight attempts at the same task, introduced in the τ-bench paper (arXiv:2406.12045, June 2024). Accuracy tells you the agent can do the job. pass^8 tells you it will. In τ-bench's retail domain, leading agents scored under 25% on pass^8 despite roughly 50% single-run success.

How long does pre-launch testing take?

For a focused pilot on one channel and one intent cluster, a structured test cycle runs two to four weeks. That covers building the replay dataset, running the adversarial suite, and completing at least two full rounds of the stack and escalation tests. Compressing below two weeks usually means skipping the repeatability test, which is the one most likely to catch production-level inconsistency.

What is the minimum viable test set?

Twenty to 50 real conversations for the replay test per Anthropic's January 2026 evaluation guidance, 20 for the repeatability test, and five designed scenarios each for the adversarial and escalation tests. Below those thresholds you are not testing. You are confirming what you already believe.

Should we test in a sandbox or against production systems?

Both. Sandbox testing catches language and reasoning failures. Testing against real systems, in a controlled staging environment, catches orchestration failures. An agent that performs well in sandbox and fails on live system writes is a common pattern, and it is exactly what the stack test exists to expose.

What if the agent passes all seven tests but still underperforms in production?

That is a scope problem, not a test failure. It usually means live traffic included intents or edge cases outside the original test set. Narrow the live scope, expand the replay dataset to include the new patterns, and retest. Do not expand automation coverage until the new intents pass the full suite.

Is my company liable for what our AI agent tells customers?

In Moffatt v. Air Canada (2024 BCCRT 149, February 14, 2024), the tribunal held Air Canada liable for negligent misrepresentation by its chatbot, stating that a chatbot "is still just a part of Air Canada's website." Treat your agent's statements as your company's statements. This is not legal advice. Consult counsel on your jurisdiction.

Do I legally have to tell customers they are talking to AI?

In the EU, yes. Article 50 of the EU AI Act has applied since August 2, 2026 and requires disclosure at the start of the first interaction unless it is obvious. In the US, California's B.O.T. Act requires disclosure for bots used to incentivize commercial transactions, and Utah requires it on clear and unambiguous request. Consult counsel for your specific footprint.

Is this framework only relevant for large enterprises?

No. The tests scale down. A mid-market team with one AI channel and a limited intent set can run a meaningful version of all seven in a few weeks. Pass conditions adjust to your volume and risk tolerance. The failure modes do not change with company size.

How often should I re-test an AI agent after launch?

Re-run your replay suite monthly and your adversarial suite after every prompt, policy or model change. Microsoft's AI red teaming guidance frames this as a four stage lifecycle including scheduled post-deployment runs. Between test runs, monitor conversations continuously rather than sampling.

What is AXO?

AXO stands for Agentic Experience Orchestration. It is UJET's approach to agentic AI in the Experience Center: AI that orchestrates action across the systems in your stack while the conversation is still happening, rather than summarizing the conversation after it ends.

How do I tell real agentic AI from agent washing?

Ask it to complete a transaction, not describe one, then audit every system of record for a correct write. Gartner estimated in June 2025 that only about 130 of the thousands of vendors marketing agentic AI are genuinely agentic.

Take these tests to your next vendor call

The seven tests are not ours. Run them on whoever you are evaluating, us included. The stack test alone takes about twenty minutes and will tell you more than an hour of slides.

When you get to UJET, ask the same questions. Tests 4 and 6 are the ones we would rather be judged on.

See how AXO works | Talk to our team

Sources

Huang et al., Salesforce AI Research, CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions, arXiv:2505.18878, May 24, 2025. https://arxiv.org/abs/2505.18878
Yao, Shinn, Razavi and Narasimhan, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv:2406.12045, June 17, 2024. https://arxiv.org/abs/2406.12045
Gartner, Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027, June 25, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, February 14, 2024. https://decisions.civilresolutionbc.ca/crt/crtd/en/item/535491/index.do
Fortune, Cursor AI support bot invents policy, April 19, 2025. https://fortune.com/article/customer-support-ai-cursor-went-rogue
Anthropic Engineering, Demystifying evals for AI agents, January 9, 2026. https://anthropic.com/engineering/demystifying-evals-for-ai-agents
OpenAI, A practical guide to building agents, 2025. https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf
Microsoft Learn, AI Red Teaming Agent, Microsoft Foundry, updated August 19, 2026. https://learn.microsoft.com/en-us/azure/foundry/concepts/ai-red-teaming-agent
European Commission, Transparency obligations under Article 50 of the AI Act. https://digital-strategy.ec.europa.eu/en/faqs/transparency-obligations-under-article-50-ai-act
California Bus. & Prof. Code §17941 (B.O.T. Act), operative July 1, 2019.
Utah AI Policy Act as amended by SB 226, effective May 7, 2025.
Zendesk, CX Trends 2026, published November 18, 2025 (11,000+ respondents, fielded June 2025).
Gizmodo, Chevrolet dealership chatbot prompt injection, December 20, 2023.

BLOG

Subscribe to Our Blog

AI for the grind.

Humans for the gold.