Alibi

A small murder mystery where every suspect is an LLM agent that remembers, keeps secrets and lies. You are the detective.

Alibi is a learning project: one clean, working example of each core LLM-engineering concept (structured outputs, RAG, MCP, agent orchestration, evals, fine-tuning, observability) in a codebase small enough to read. The roadmap and milestone checklists live in docs/PLAN.md.

make install                 # uv sync
make up                      # postgres + phoenix (docker)
cp .env.example .env         # add your OPENROUTER_API_KEY
make hello                   # one real LLM call: tokens, cost, latency (traced)

make case                    # generate + validate a case into cases/
uv run alibi interrogate --case cases/<id>.json --suspect <id> --debug
uv run alibi play --case cases/<id>.json --new
make test && make lint

Screenshots use the test case with scripted answers. A recorded GIF of a real game and a live link are still to do.

            Browser (HTMX)      Terminal (Rich)     Any MCP client
                  │                   │                   │
                  └─────── FastAPI / CLI ──────┐          │
                                               ▼          ▼
                                     MCP server (game tools)
                                               │
                              Game loop (LangGraph): actions, turns, score
                                               │
                       ┌───────────────────────┼─────────────────────┐
                       ▼                       ▼                     ▼
               Suspect agent (LangGraph)   Truth labeller      Lie detector
               retrieve → think → speak    claims → TRUE/FALSE (ONNX, CPU, M5)
               → remember                        │
                       │                         ▼
                 pgvector memory          stored labels → evals + training data
                       │
                       ▼
     llm.py (LiteLLM) ── hosted models, cost/latency/tokens, daily spend cap
                       │
     OpenTelemetry ── Phoenix

     Offline (laptop / CI): case generator + validator, solver agent, evals,
                            lie-detector training (Colab) → model.onnx

Every choice is here to teach one thing. The "why" is the point.

Each milestone adds one concept; see docs/results/ for the measured outcomes.

1. M1 — structured output + validation. One LLM call fills a Pydantic Case from config/setting.yaml; a deterministic validator checks eight rules and a bounded repair loop feeds violations back to the model. Creativity is the model's job; correctness is the code's. (case.py, generator.py, validator.py)
2. M2 — RAG memory + a suspect agent. Each suspect stores memories in pgvector and answers through a LangGraph. Information hiding is a code guarantee: the prompt only ever sees the persona, the suspect's own secrets and what knowledge.py says they know. (knowledge.py, memory.py, suspect.py)
3. M3 — the world and MCP. A Game owns the rules (20 actions, scoring, persistence); the terminal and an MCP server are thin adapters over it. State lives in Postgres, so a game survives a restart. (game.py, mcp_server.py)
4. M4 — evals. An automatic detective plays through the tools; a judge labels every claim against the ground truth; metrics gate changes in CI. (solver.py, truth.py, evals.py, config/gates.yaml)
5. M5 — detector, web UI, homelab. A tiny DistilBERT fine-tuned on our own TRUE/FALSE claims, exported to ONNX, plus a browser UI and the deployment below.

Measured numbers (full write-ups in docs/results/).

Game quality — 20-game eval (uv run alibi eval --games 20 --gate):

Honest caveat: the raw solve_rate counts timed-out games as losses; of the games it finished, the auto-solver won 17/17. A capable model finds a four-suspect mystery too easy — the number that matters for a game is the human one, which the browser UI provides. See docs/results/m4.md.

Lie detector vs baselines (one statement, no case context; 30-statement sample):

Baselines: uv run alibi baselines --data ml/data/test.jsonl. Details and the training run live in ml/MODEL_CARD.md.

The app is one small image (python:3.12-slim, no torch), run twice: as MCP over streamable HTTP (port 8080) and as the browser game (port 8000), alongside Postgres and Phoenix — sized to stay under ~2 GB on an Intel N150 with no GPU.

cp .env.example .env                      # set OPENROUTER_API_KEY (+ low budget)
docker compose -f compose.homelab.yml run --rm app alibi case --seed 1
make deploy                               # docker compose -f compose.homelab.yml up -d --build

Or build/run the image directly:

docker build -t alibi .
docker run --rm -p 8080:8080 --env-file .env -v alibi_cases:/app/cases alibi
# the browser game (same image, different command):
docker run --rm -p 8000:8000 --env-file .env -v alibi_cases:/app/cases alibi \
  alibi web --host 0.0.0.0 --port 8000

Full walkthrough — Cloudflare Tunnel (public access, no open ports), cost caps, memory limits and Postgres backup — is in docs/deploy-homelab.md.

The whole game is exposed as MCP tools. Point any client at the stdio server:

{
  "mcpServers": {
    "alibi": {
      "command": "uv",
      "args": ["run", "alibi", "mcp", "--case", "cases/<id>.json"],
      "cwd": "/absolute/path/to/alibi"
    }
  }
}

Or at the deployed HTTP endpoint: { "type": "http", "url": "http://<vm>:8080/mcp" }.

Tools: case_summary, status, list_rooms, search_room, inspect, question, lie_detector, accuse.

uv run alibi web        # http://127.0.0.1:8000

FastAPI + Jinja + HTMX, no JS build step (htmx and the fonts are served from src/alibi/static, so no CDN is needed). Generate a case (or pick one), search rooms, question suspects — presenting evidence when you have it — test an answer with the lie detector, then accuse and read "what really happened".

Every "Start a game" is a fresh game with its own URL, so a case can be replayed and visitors of a public deploy never share one. The story clock turns your 20 actions into a night: you arrive an hour after the death, each action takes a quarter of an hour, and the murderer leaves when the roads clear at dawn. The reveal page ranks you (Constable to Great Detective) and lists the real clues you missed. The same Game object backs the web, CLI and MCP server.

make eval                          # uv run alibi eval --games 20 --gate
uv run alibi eval --games 5 --seed 1

Metrics (config/gates.yaml): case_valid_rate, solve_rate, contradiction_rate, early_confession_rate, cost_per_game. A GitHub Action runs a 5-game gate on PRs touching src/alibi/prompts/** or suspect.py and comments the summary on the PR. The gate is exercised by deliberately breaking a suspect prompt: the closed "broken prompt" PR.

alibi export-dataset turns the stored labels into fine-tuning JSONL (split by case, no leakage); alibi baselines scores always-TRUE, majority and the LLM judge for comparison with the fine-tuned detector.

src/alibi/        case · generator · validator · knowledge · llm · memory
                  suspect · game · mcp_server · solver · truth · evals
                  detector · export · baselines · web · templates · static · cli
config/           models.yaml (role → model) · setting.yaml · gates.yaml
tests/            deterministic tests only (no network; FakeLLM)
ml/               dataset export + training (M5; never installed on the server)
assets/           banner + demo screenshots
docs/PLAN.md      roadmap · docs/adr/ decisions · docs/results/ measurements
Dockerfile        small runtime image        compose.homelab.yml  the deploy

• Every LLM/embedding call goes through src/alibi/llm.py, with a role.
• Models come from config/models.yaml; prompts from src/alibi/prompts/*.jinja.
• The validator, not the model, decides whether a case is sound.

MIT — see LICENSE.