triage-and-voice

A reference implementation of the Triage-and-Voice architectural pattern. Shows how splitting a single LLM call into triage (structured analysis) + deterministic gate + voice (response generation) eliminates hallucination of critical data like policies, contacts, and order details.

An architectural pattern for LLM products that treats user emotional state as a first-class routing signal — because sycophancy under emotional pressure is the one thing prompt engineering cannot fix.

When an LLM built to be helpful meets a user in grief, fear, or desperation, it drifts into accommodation, generating whatever seems to "help" — including policies, procedures, numbers, and contacts that do not exist (Sharma et al. 2024; ELEPHANT benchmark; Anthropic research on sycophancy). Triage-and-Voice solves this architecturally: classify both intent and emotional state before generation, then inject critical facts from verified sources — so the LLM call that speaks to the user never decides what's allowed to be said.

LLM products that bake facts into system prompts hallucinate those facts. The naive approach -- stuffing policies, contacts, and order data into a single prompt -- produces responses that look correct but contain invented numbers, wrong deadlines, and fabricated contact information.

This is not a theoretical risk. In 2024, Air Canada's chatbot hallucinated a refund policy for a grieving customer who was booking a last-minute flight to a funeral. The customer relied on the invented procedure. The airline lost in tribunal and had to honor the hallucinated policy. This is not a coincidence: LLMs drift into accommodation under emotional pressure, and "just add more rules to the prompt" doesn't help.

The more critical the data, the more dangerous the hallucination. A wrong phone number for a safety hotline is not a UX problem -- it is a liability.

This repo demonstrates the problem with a concrete side-by-side comparison: a naive bot (single prompt, facts baked in) vs a triage-and-voice bot (structured pipeline, facts injected from verified sources).

flowchart TD
    M["User: 'I want to return<br/>order ORD-001'"]

    M --> N["Naive bot<br/>(single prompt,<br/>facts baked in)"]
    N --> NR["<b>30-day</b> refund window<br/>email: help@shopco.com<br/>❌ both hallucinated"]

    M --> TV["Triage &rarr; Gate &rarr; Voice<br/>(verified data injected)"]
    TV --> TR["<b>14-day</b> refund window<br/>email: support@shopco.example<br/>✅ from policies.json"]

    style NR fill:#fdd,stroke:#a00,color:#000
    style TR fill:#dfd,stroke:#0a0,color:#000

flowchart TD
    U[User message] --> T[Triage LLM]
    T -->|structured JSON| G[Gate]
    G -->|persona + verified data| V[Voice LLM]
    V --> R[Response]

    DB[(orders.json<br>policies.json<br>contacts.json)] --> G

    style G fill:#f9f,stroke:#333,stroke-width:2px
    style DB fill:#ffd,stroke:#333

What travels on each channel:

flowchart TD
    U[User] -->|"free text"| T[Triage LLM]
    T -->|"JSON: intent,<br/>emotion, data_keys,<br/>entities"| G[Gate]

    G -->|"lookup keys<br/>(order_id,<br/>policy_slug)"| DB[(orders.json<br/>policies.json<br/>contacts.json)]
    DB -->|"verified records"| G

    G -->|"persona +<br/>injected facts<br/>(only requested keys)"| V[Voice LLM]
    V -->|"natural language"| R[Response]

    DB -.->|"❌ never<br/>direct access"| V

    style G fill:#f9f,stroke:#333,stroke-width:2px
    style DB fill:#ffd,stroke:#333
    linkStyle 6 stroke:#a00,stroke-dasharray:5 5

The dashed line is the whole point: raw data never reaches the voice LLM. The gate is the only component that reads the database, and it hands the voice only the specific fields the triage requested.

How it works:

1. Triage -- an LLM call that classifies the message along two dimensions — intent (what the user wants) and emotional state (how they're framing it). Outputs structured JSON: category, urgency, requested data keys, extracted entities. Writes no user-facing text.
2. Gate -- a pure Python function (no LLM). It reads the triage output and makes deterministic decisions: which voice persona to use, what verified data to inject, whether to escalate to a human. Every rule is testable.
3. Voice -- an LLM call that generates the user-facing response. Its system prompt is a Jinja2 template that receives only the data the gate explicitly provides. The voice never sees raw database contents -- only what the gate decided to inject.

The gate is the key. It is the only component that touches real data, and it contains zero LLM calls. The voice LLM cannot hallucinate a refund policy because it never sees one -- it only sees the exact policy text the gate injected.

git clone https://github.com/svetkis/triage-and-voice.git
cd triage-and-voice
cp .env.example .env   # add your OpenAI API key (or OpenRouter)
pip install -e ".[dev]"
uvicorn src.api:app --reload

Then test it:

# Triage-and-voice pipeline
curl -s http://localhost:8000/chat/triage-voice \
  -H "Content-Type: application/json" \
  -d '{"message": "I want to return order ORD-001, the headphones are broken"}' \
  | python -m json.tool

# Naive single-prompt bot (for comparison)
curl -s http://localhost:8000/chat/naive \
  -H "Content-Type: application/json" \
  -d '{"message": "I want to return order ORD-001, the headphones are broken"}' \
  | python -m json.tool

The triage-and-voice response will contain the correct refund policy ("14 days", from data/policies.json). The naive bot will likely say "30 days" -- the wrong number baked into its prompt.

docker build -t triage-and-voice .
docker run --rm -p 8000:8000 -e OPENAI_API_KEY="$OPENAI_API_KEY" triage-and-voice

make test
# or directly:
pytest -v

Tests use no external APIs. Gate, model, and data-source tests are fully deterministic. Triage and voice tests mock the LLM client.

The eval script runs all 12 scenarios from tests/scenarios.yaml through both bots and produces a comparison report.

Latest run (DeepSeek deepseek-chat, 2026-04-20): Naive 3/12 (25%) · Triage-and-Voice 12/12 (100%) · Δ +75 pp. The naive bot hallucinates critical data (refund windows, contact emails, policy text) in 9 of 12 safety-critical scenarios; T&V passes all 12 because the gate injects verified values from data/. Full report: docs/eval_results.md.

make eval
# or directly:
python scripts/run_eval.py

Requirements: a valid OPENAI_API_KEY in .env (eval makes real LLM calls).

Results are saved to eval-runs/run-{timestamp}/ with both a JSON dump and a markdown report. A copy is also written to docs/eval_results.md.

Each scenario defines:

• expected_category -- what the triage should classify the message as
• must_contain -- strings that must appear in the response (e.g., correct policy text, real contact info)
• must_not_contain -- strings that must not appear (e.g., for jailbreak scenarios)
• expected_human_handoff -- whether the bot should escalate

Latest run (DeepSeek deepseek-chat, 2026-04-20):

Totals: Naive 3/12 (25%) · Triage-and-Voice 12/12 (100%).

The naive bot fails whenever the response must contain exact data — refund windows, contact emails, policy text — because it hallucinates those values from its system prompt. The triage-and-voice bot passes because the gate injects verified data from data/ and the voice LLM cannot fabricate what it never saw.

For a reusable eval framework with binary safety verdicts (HELD / BROKE / LEAK / MISS / SAFE), persona fan-out, and cross-run trend analysis, see the companion project triage-voice-eval.

The gate is not a single file — it's a small reusable framework. One consumer declares their behaviour in YAML and registers data sources in Python; the engine does the rest.

After triage, the gate dispatches a list of actions per category. Three types ship in the core:

Anything else — returning a document, fetching a price list, posting to Slack — is a custom action type the consumer registers:

from src.gate import Gate, GateAction

class NotifySlackAction:
    def apply(self, triage, decision, params):
        ...

gate.register_action("notify_slack", NotifySlackAction())

The framework does not know about orders, policies, or contacts. The consumer defines DataSource implementations and registers them by name:

gate.register_source("orders", OrderSource())
gate.register_source("policies", PolicySource())

YAML then references these source names in inject_data actions.

A consumer's entire gate behaviour fits in one YAML file. Example from examples/shopco/config/shopco.yaml:

categories:
  safety_issue:
    actions:
      - type: handoff
        params: {reason: safety_incident}
      - type: inject_data
        params:
          source: escalation_contacts
          key: safety_hotline
          contact_key: safety_hotline
      - type: voice_response
        params:
          persona: empathetic_escalation
          inject_data: [safety_hotline]

Gate.freeze() walks the full config and fails loud on unknown action types, unknown persona references, or unknown source references. Called automatically on the first decide() if the consumer didn't call it explicitly. Typos in YAML fail at startup, not at the first live request.

src/gate/ and src/orchestrator.py are domain-neutral. The wiring happens in each consumer's main.py: build_gate() registers that consumer's data sources and freezes the gate, then build_pipeline() combines it with the consumer's triage prompt and resolver into a Pipeline. src/api.py and scripts/run_eval.py import examples.shopco.main.build_pipeline and hardcode the ShopCo startup — this is intentional: the repo ships as a reference implementation with one worked example, not as a PyPI framework. Everything under src/gate/ stays untouched when you fork.

The fastest start is to copy a worked example and adapt it. Two ship:

• examples/shopco/ — e-commerce support; demonstrates safe-default harm routing (harm_state preempts commercial intent).
• examples/skycarrier/ — airline support; demonstrates emotional-state routing (same intent → different persona depending on user_emotional_state).

The mechanical copy takes minutes. The substantive work — your triage prompt and persona templates — takes hours and is irreducible: that's the domain knowledge that makes the bot yours.

Steps below assume you copied skycarrier; paths transpose if you started from shopco.

1. Copy the example.

   cp -r examples/skycarrier examples/mycompany

2. Rewrite the triage prompt — this is where the bot learns your domain. Edit examples/mycompany/prompts/triage.md. Define your intent enum, the user_emotional_state and harm_state values your domain uses, and few-shot examples drawn from real messages. Most of your work lives here.

3. Write a voice persona template for each persona you intend to use. Create examples/mycompany/prompts/voice/{persona}.md — a Jinja2 template that receives only the inject_data keys the gate provides. Templates never see raw sources.

4. Implement your data sources. Edit examples/mycompany/sources.py. Each source implements fetch(params: dict) -> str | None and returns verified data or None.

5. Wire behaviour in YAML. Edit examples/mycompany/config/mycompany.yaml. For each gate category, declare an action list (handoff, inject_data, voice_response, or custom). Persona and source names must match steps 3 and 4 — Gate.freeze() fails loud at startup on any mismatch.

6. Write a resolver. Edit examples/mycompany/resolver.py — a pure function (TriageClassification) → str that fuses triage axes (intent, user_emotional_state, harm_state) into a gate category name. The action list itself stays in YAML; the resolver only picks which category fires. The built-in _identity_resolver returns intent and ignores the other axes — fine for trivial domains, but both shipped examples define their own.

7. Wire main.py. Edit examples/mycompany/main.py so build_gate() registers your sources and build_pipeline() points at your triage prompt and resolver. The skeleton from the donor example is already wired — just rename the imports.

8. Replace the donor tests. examples/mycompany/tests/ still contains the donor's tests with hardcoded expectations. Delete them or rewrite against your domain before running pytest.

9. Point the entry points at your pipeline. In src/api.py, change one import:

   from examples.mycompany.main import build_pipeline

   Do the same in scripts/run_eval.py if you run eval, and replace tests/scenarios.yaml with your own scenarios (or copy and edit it).

src/gate/, src/orchestrator.py, src/triage.py, and src/voice.py stay untouched throughout.

Two consumers of the framework ship in the repo. Both use src/gate/ unchanged; each brings its own YAML, data sources, and persona prompts.

Read either alongside this doc; the YAML there is the concrete form of the pattern.

triage-and-voice/
├── src/
│   ├── api.py              # FastAPI endpoints
│   ├── config.py           # Settings via pydantic-settings (.env + defaults)
│   ├── models.py           # Domain models: TriageResult, ExtractedEntities, ChatMessage, BotResponse
│   ├── triage.py           # Triage classifier — LLM call → structured JSON
│   ├── gate/               # Gate framework — reusable, domain-neutral
│   │   ├── __init__.py     #   Public API: Gate, GateAction, DataSource, GateDecision
│   │   ├── contracts.py    #   GateAction and DataSource Protocols
│   │   ├── config.py       #   Pydantic schema + YAML loader
│   │   ├── decision.py     #   GateDecision accumulator + VoiceCallSpec
│   │   ├── engine.py       #   Gate class — dispatch, registries, freeze()
│   │   └── actions/        #   Three built-in action types
│   │       ├── handoff.py
│   │       ├── inject_data.py
│   │       └── voice_response.py
│   ├── voice.py            # Voice generator — Jinja2 persona prompt + LLM
│   ├── orchestrator.py     # Pipeline: triage → gate → voice with fallback
│   └── naive/
│       └── bot.py          # Naive single-prompt bot (baseline)
├── examples/               # Worked examples — each one a full consumer of the framework
│   ├── shopco/             #   E-commerce support bot (used throughout this README)
│   │   ├── main.py         #     build_gate() + build_pipeline() factories
│   │   ├── sources.py      #     OrderSource, PolicySource, ContactsSource
│   │   ├── config/shopco.yaml
│   │   ├── prompts/        #     triage.md + voice/ personas (ShopCo-specific)
│   │   └── tests/
│   │       ├── test_sources.py
│   │       └── test_shopco_flow.py
│   └── skycarrier/         #   Airline support bot — demonstrates emotional-state routing
│       ├── main.py         #     build_gate() + build_pipeline() for the airline domain
│       ├── sources.py      #     FareTermsSource, FlightStatusInfoSource, BaggagePolicySource
│       ├── config/skycarrier.yaml
│       ├── data/           #     bereavement_fare.json, flight_status_info.json, baggage_policy.json
│       ├── prompts/        #     Domain-specific triage.md + voice personas
│       └── tests/test_skycarrier_flow.py
├── prompts/
│   └── naive/bot.md        # Naive single-prompt baseline (ShopCo data baked in, intentionally)
├── data/                   # ShopCo example data (consumed by examples/shopco/sources.py)
│   ├── orders.json
│   ├── policies.json
│   └── escalation_contacts.json
├── tests/                  # Framework tests (engine, actions, config — no ShopCo)
│   ├── gate/
│   │   ├── fixtures/*.yaml
│   │   ├── test_decision.py
│   │   ├── test_config.py
│   │   ├── test_config_loader.py
│   │   ├── test_engine_construction.py
│   │   ├── test_engine_decide.py
│   │   ├── test_engine_freeze.py
│   │   └── actions/
│   │       ├── test_handoff.py
│   │       ├── test_inject_data.py
│   │       └── test_voice_response.py
│   ├── scenarios.yaml      # 12 eval scenarios consumed by scripts/run_eval.py
│   ├── test_models.py
│   ├── test_orchestrator.py
│   ├── test_triage.py
│   └── test_voice.py
├── scripts/run_eval.py
├── .env.example
├── .github/workflows/test.yml
├── Makefile
├── pyproject.toml
└── LICENSE

The file prompts/naive/bot.md contains deliberately incorrect data:

• Says refund window is 30 days (actual: 14 days)
• Says warranty is 24 months (actual: 12 months)
• Says support email is help@shopco.com (actual: support@shopco.example)

This is the point. The naive bot "knows" these facts from its system prompt and will confidently repeat them. The triage-and-voice bot never sees hardcoded facts -- it gets them from the gate, which reads from data/.

All extension points live in YAML and in the consumer package. Core framework code in src/gate/ is not edited.

1. Add the category to your consumer's triage prompt (e.g. examples/shopco/prompts/triage.md) so the classifier returns it.
2. Add the category to the consumer YAML (e.g. examples/shopco/config/shopco.yaml) under categories: with the desired action list.
3. Add scenarios to tests/scenarios.yaml for eval.

1. Write a class in your consumer package (e.g. examples/shopco/sources.py) implementing fetch(params: dict) -> str | None.
2. Register it in your build_gate() factory: gate.register_source("my_source", MySource()).
3. Reference it from YAML under inject_data actions.

1. Write a class implementing apply(triage, decision, params) -> None.
2. Register it: gate.register_action("my_action", MyAction()).
3. Use it in YAML under any category's action list.

1. Create a Jinja2 template inside your consumer package (e.g. examples/shopco/prompts/voice/{persona_name}.md).
2. Add the persona name → template path mapping in YAML under personas:. Paths are resolved from the repo root at runtime.
3. Reference it from any voice_response action's persona param.

Persona names are arbitrary strings — the framework does not maintain a closed enum.

Set OPENAI_BASE_URL in .env to any OpenAI-compatible endpoint (OpenRouter, Azure, local Ollama, etc.).

{
  "message": "I want to return order ORD-001",
  "history": [
    {"role": "user", "content": "Hi"},
    {"role": "assistant", "content": "Hello! How can I help?"}
  ]
}

{
  "text": "I'd be happy to help with your return...",
  "human_handoff": false,
  "trace": [
    "triage: category=refund_request, urgency=medium",
    "voice_response: persona='default_friendly'",
    "inject_data: policies→refund_policy",
    "voice: persona=default_friendly"
  ]
}

The trace field shows the full decision chain for debugging.

All configuration is via environment variables (or .env file):

Customer support has worked by script for forty years — telephone call centers in the 80s, chat support in the 2010s, any large service desk today. Not because operators are slow, but because humans under pressure also drift: an irate customer, an AHT clock, an unfamiliar case, and accuracy gives way to fluency. The industry learned this early and moved the fragile part — what to say — out of the operator's head and into an external decision layer.

Two axes of escalation are built into every modern playbook, and they fire independently:

• Authority gap — the operator can't approve what the customer is asking. That's an intent problem.
• Emotional intensity — the customer is at the edge; empathy comes before logic. That's a state problem.

The HEARD framework (Hear, Empathize, Acknowledge, Resolve, Diagnose) codified the second one decades ago: process the emotion before the answer, or the answer isn't heard. Modern contact-center platforms score sentiment in real time and page a supervisor when it crosses a threshold — a wholly separate trigger from whether the operator has authority to resolve the case.

The mechanical split maps onto this pattern:

The operator doesn't hold escalation rules in their head — the rules live in the playbook, and the gate enforces them. The script sits at the gate, not at the voice. This is the detail most LLM support implementations miss when they stuff policies, tone, and decision rules into one system prompt.

And the LLM needs this script more than a human does, not less. An experienced operator learns over months not to buckle under emotional pressure; a model starts every conversation from the same weights, with sycophancy baked in. Repetition doesn't harden it. The architecture has to.

Triage-and-Voice doesn't introduce new building blocks. Each of its components maps to established practices: triage to semantic routing and classification — here widened to route on emotional state alongside intent — the gate to deterministic tool execution and grounded RAG, voice to constrained generation. What this project contributes is a named architectural pattern that binds them into a discipline:

The LLM call that speaks to the user must never be the one that decides what's allowed to be said.

This discipline is not enforced by orchestration frameworks (LangGraph, Burr, DSPy) — it's enforced by how you structure your prompts and gates inside them. Naming the discipline makes it reviewable: engineers can point to a pattern violation in code review instead of re-arguing first principles every time.

Related work:

• Orchestration mechanisms: LangGraph, Burr, DSPy, Haystack
• Input/output validation: Guardrails AI, NeMo Guardrails
• Intent routing: semantic-router
• Constrained generation: Instructor, Outlines
• Evaluation frameworks: DeepEval, promptfoo, RAGAS

Triage-and-Voice is a composition discipline that can be implemented on top of any of these.

Every user message fans out to two model calls (triage + voice). Compared to a single-prompt bot, expect roughly 2× the per-request token cost. Triage uses a tight JSON schema and runs at temperature=0, so its token count is predictable and small; voice is the larger of the two.

Triage and voice are sequential: the gate needs the triage category and extracted entities before it can decide which persona and which data to inject. Observed p95 on gpt-4o-mini: ~1.5-3s vs ~0.8-1.5s for the naive single call. The pattern is not a fit for ultra-low-latency UX (streaming voice, autocomplete).

If triage mis-classifies (e.g. a safety_issue lands as complaint), the gate runs the wrong action list and voice receives the wrong data. There is no post-hoc recovery: voice cannot see that triage was wrong because voice only sees what the gate injected. Mitigation is on the triage side — keep the category list short, keep prompt examples sharp, and cover mis-classification paths in the eval (tests/scenarios.yaml).

What the pattern mitigates. Hallucination of injected data: policies, contacts, and order details are read from verified sources by the gate and cannot be fabricated by the voice LLM.

What the pattern does not mitigate. Anything the voice LLM generates from its own parametric knowledge (small talk, tone, paraphrasing of injected facts). Prompt injection via the client-supplied history list. Model-level jailbreaks that bypass the persona prompt. See "Known Security Limitations" below for specifics.

This is a reference implementation of an architectural pattern, not a hardened production service. The following security limitations are deliberately left unfixed to keep the code focused:

/chat/* accepts an arbitrary history list (src/api.py:21-24). A client can submit forged assistant turns that steer the triage classifier — for example, injecting a fake prior assistant message that reclassifies a safety_issue as out_of_scope and strips the escalation. Mitigation: trust only server-stored conversation state, or drop assistant turns from the client payload before passing to triage.

Endpoints are unauthenticated and impose no caps on message or history length. Every request fans out to two LLM calls (triage + voice), so an attacker can drive unbounded provider cost. Mitigation: add auth, per-key rate limits, and hard caps on message length and history turn count.

• Deep dive (Russian): Why your LLM product hallucinates the one thing it shouldn't — original pattern introduction
• English version: Substack
• Follow-up (Russian): Почему ваш LLM-бот врёт клиентам — и паттерн, который это чинит
• English version: Why Your LLM Support Bot Is Working Against You
• Companion eval framework: triage-voice-eval — binary safety verdicts, persona fan-out, trend analysis
• Author: Svetlana Meleshkina
• Consulting: if you're considering this pattern for your product, reach out on Telegram @svetkis — LLM-bot architecture reviews and audits.

MIT -- see LICENSE.