OpsPilot

Agentic internal IT/ops support assistant — a grounded, hybrid-retrieval RAG over a real corpus of resolved support tickets. Answers cite real ticket IDs; out-of-scope questions are refused, not hallucinated. Built to show end-to-end AI-engineering: real data → hybrid retrieval + reranking → typed, grounded generation → evaluation → a deployable demo.

▶️ Live demo: https://jimmy-opspilot.streamlit.app — ask a real IT question, get a cited, grounded answer (or a refusal when it's out of scope).

Two modes:

• Ask (grounded RAG): ask an IT question — in any language — and OpsPilot retrieves the most relevant resolved tickets, reranks them, and has Claude synthesise a grounded answer citing the ticket IDs it used — or refuse if the KB doesn't cover it. A 中文 question gets a 中文 answer grounded in the (English) tickets.
• Agent (takes action): a LangGraph agent that diagnoses with tools (look up a user's devices, account, access; search resolved tickets) and can act — create a ticket, grant access — but every write action pauses for human approval first.

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env                 # add your ANTHROPIC_API_KEY
python ask.py --build                # build the index once (downloads corpus, ~10 min)
python ask.py "my VPN keeps disconnecting and software crashes"
# or the web UI:
streamlit run app.py

flowchart TD
    Q["User question (any language)"] --> R{Mode}

    R -->|Ask · RAG| TR["translate to English<br/>if non-English"]
    TR --> HS["hybrid search<br/>vector + BM25 → RRF"]
    HS --> RR["FlashRank rerank → top-k"]
    RR --> GATE{"relevant enough?"}
    GATE -->|yes| GEN["Claude → Pydantic-typed<br/>grounded answer + ticket citations"]
    GATE -->|no| REF["refuse / labelled general fallback"]

    R -->|Agent| AG["LangGraph ReAct agent"]
    AG --> TOOLS["read tools: find_user · assets ·<br/>permissions · tickets · search_knowledge"]
    AG --> WRITE{"write action?<br/>create_ticket / grant_permission"}
    WRITE -->|interrupt| HITL["human approval gate"]
    HITL -->|approve| BK[("mock IT backend<br/>directory · assets · IAM · tickets")]
    HITL -->|reject| AG

    subgraph Index["built once, baked into the image"]
        HF["3k real IT tickets (HF)"] --> EMB["bge embeddings + BM25"] --> LDB[("LanceDB")]
    end
    LDB -.-> HS
    LDB -.-> TOOLS

• Corpus — 3,000 English IT/Technical tickets sampled from Tobi-Bueck/customer-support-tickets (real subject/body + resolution); cached to data/corpus.parquet.
• Retrieval — LanceDB vector (bge-small-en-v1.5) + Tantivy BM25, fused with Reciprocal Rank Fusion.
• Multilingual — the corpus is English; a non-English question is translated to English for retrieval (so the strong English retriever + reranker do the work), then the answer is generated in the user's original language. This beat swapping in a multilingual embedding, which halved English recall (0.87 → 0.47) — see eval/crosslingual_eval.py.
• Rerank — FlashRank cross-encoder (ms-marco-MiniLM-L-12-v2), local ONNX, no torch / no API key.
• Generation — Claude (claude-haiku-4-5) via LiteLLM, returning a Pydantic-validated {answered, answer, cited_tickets} object.
• Guardrails — a rerank-score gate + a grounding prompt: a grounded answer is only produced when the retrieved tickets actually support it, and it cites the ticket IDs used.
• Answer modes (provenance-aware) — when nothing in the KB is relevant, the strict default refuses (this is what the eval below measures — zero fabricated answers). The app also exposes an opt-in general-knowledge fallback: instead of refusing it answers from the model's own knowledge, returned clearly labelled and uncited (source="general") so it's never confused with a grounded, verifiable answer. Grounded vs. general is an explicit, labelled tradeoff — not a silent hallucination.

Retrieval — 150 held-out tickets, subject as the query, ablating each component (full results in eval/RESULTS.md):

Reranking gives the clearest precision lift (recall@1 +4 pts, MRR +2 pts). On near-exact subject queries vector retrieval is already strong; on messier natural-language queries the hybrid + rerank gap widens.

Cross-lingual — same gold ticket, asked in English vs. Chinese (translated to English for retrieval): EN recall@5 0.87 → ZH 0.80 (a multilingual embedding instead halved EN to 0.47). See eval/crosslingual_eval.py.

Generation — a custom LLM-as-judge (Claude), plus real RAGAS for the canonical metrics:

RAGAS faithfulness is stricter (per-claim verification) than the holistic custom judge — an honest gap worth reporting rather than cherry-picking. Out-of-scope refusal correctness: 4/4 (no hallucination). RAGAS pins an older LangChain that clashes with the agent's LangChain 1.x, so it runs in an isolated venv (eval/requirements-ragas.txt) via a two-phase generate-then-score flow — see eval/ragas_eval.py.

A LangGraph ReAct agent (langchain.agents.create_agent, LangGraph 1.x) over a mock IT backend — the kind of thing an internal IT bot actually needs to do, not just answer.

• Tools — read: lookup_assets, get_account, check_permission, get_ticket, search_knowledge (bridges the grounded RAG above); write: create_ticket, grant_permission.
• Human-in-the-loop — every write tool calls LangGraph's interrupt(); the graph pauses and surfaces the proposed action for approval, and only mutates state when the reviewer resumes with approve (via Command(resume=...) + a checkpointer). Reads run freely.
• Circuit breaker — a recursion_limit step budget caps a runaway tool loop.
• Tools are written as plain typed functions so the same backend could be exposed over MCP with no agent changes. The agent layer lives in src/opspilot/agent/; the HITL gate is tested offline with a fake tool-calling model in tests/test_agent.py.
• Agent eval — scored with DeepEval (eval/agent_eval.py): tool correctness 1.00, task completion 0.96 over 4 scenarios. (DeepEval for the agent; RAGAS for RAG — two frameworks, each used where it's strongest.)
• The simulated company is generated deterministically — 38 employees across 7 departments, ~55 assets, a permission matrix — see company.py.

Beyond the Streamlit demo, OpsPilot ships a production-style web app: a FastAPI backend (server.py) exposing the same RAG/agent/persistence/telemetry stack as a JSON API — grounded answers stream over SSE (stage events → live tokens → citations) — and a React + TypeScript frontend (web/) with chat history, the agent approval flow, live company-system tables and an ops widget. FastAPI serves the built SPA, so it deploys as one container.

pip install -r requirements.txt && cd web && npm install && npm run build && cd ..
uvicorn server:app --port 8000        # → http://localhost:8000

• Per-call telemetry — every model call (RAG and agent) records latency, tokens and cost (telemetry.py); the sidebar ops panel shows live spend, $/call, p50/p95 latency and calls by purpose. Generation runs on Haiku, so a typical grounded answer costs well under $0.01.
• Langfuse tracing — set LANGFUSE_PUBLIC_KEY/LANGFUSE_SECRET_KEY and LiteLLM's native callback ships full traces automatically (v2 SDK — what LiteLLM currently supports).
• Model tiering — utility calls (query translation, chat titles) always use the cheap tier (llm_model_fast); upgrade answer quality to Sonnet via OPSPILOT_LLM_MODEL and the cost profile barely moves.
• Input guardrails (guardrails.py) — PII (emails, phones, card numbers, IPs) is redacted before any model call or persisted log, and prompt-injection attempts ("ignore previous instructions…", system-prompt extraction, role hijacks) are refused outright. Deterministic regex: zero latency, zero extra spend, unit-tested.

Internal IT/ops teams drown in repetitive questions and access requests. OpsPilot is a grounded assistant that answers from real resolved tickets and acts through tools with a human approval gate — built end to end (data → retrieval → eval → agent → deploy) to show an AI feature shipped to production: grounded, evaluated, observable-ready, and cost-controlled. Knowledge is a public sanitised ticket dataset; the company the agent acts on is synthetic — no real employer data.

Real engineering judgement, not a happy-path demo:

• Stale cache crashed the live app. Streamlit's @st.cache_resource kept the object built on first boot across hot-reloads; when a method signature later changed, the stale object threw on the new code — green CI, red prod (CI mocked the LLM and never booted the app). Fix: a source-hash cache key that busts on any code change, plus a Streamlit AppTest boot smoke test so app-level breakage fails CI.
• A "better" multilingual embedding halved English recall (0.87 → 0.47). I measured it, rejected it, and got cross-lingual support a better way — translate the query to English before retrieval (EN 0.87 → ZH 0.80, English unchanged).
• RAGAS pinned an old LangChain that clashed with the agent's LangChain 1.x. Rather than destabilise the app, I isolated the eval in its own venv and decoupled it via a JSON hand-off; chose DeepEval (which coexists) for the agent metrics. Eval tooling shouldn't dictate runtime deps.

Every model/library choice is verified against live 2026 data, not memory — see CURRENT_TECH_2026.md.

pytest (offline: RRF fusion, JSON parsing, answer/refusal orchestration, the agent HITL gate via a fake model, and a Streamlit AppTest boot smoke test) + ruff, run on every push via GitHub Actions.

Container + AWS App Runner / Bedrock (apac. inference profile in Sydney) — see DEPLOY.md. Production swaps: LanceDB→pgvector/OpenSearch, FlashRank→Cohere Rerank 3.5, add Langfuse v4 tracing.

Knowledge data is a public, sanitised ticket dataset — no real employer data is used.