Chaosline

Test whether your AI agent tells the truth when its tools break.

Chaosline is a pre-deployment safety gate for AI agents with real tool access. It breaks your agent's tools on purpose, in realistic ways, and answers two critical questions:

1. Does your agent cause actual harm when tools fail (not just fail gracefully)
2. Does it tell you the truth, or does it lie and pretend everything worked

Most testing stops at "did the agent recover?" Chaosline goes deeper: did it cause damage, and did it own up to it?

npx chaosline demo

This runs the flagship finding against a demo agent: a payment refund succeeds, the response is lost, and the agent retries without idempotency protection. The customer is charged twice. The agent is honest about being unsure, but honesty doesn't undo the duplicate charge already in the ledger.

See the output, then:

npx chaosline list                              # See all shipped scenarios
npx chaosline run --tag smoke -- <agent cmd>    # Run smoke tests against your own agent

Here's what real failure looks like: your payment tool successfully charges the card, but then the response gets lost in transit.

From the agent's perspective, this looks identical to a charge that never went through. So it retries automatically, with zero idempotency-key protection anywhere in the stack. Boom. Customer charged twice.

Even if your agent is being honest about thinking the charge failed, it's already caused the harm. That honesty won't undo the duplicate charge sitting in your ledger. This is where good intentions and bad architecture collide.

There's a dishonest version of the same problem too. When a tool returns the wrong dollar amount, some agents confidently report the original amount back to the user as fact. It's a polished message with a number that doesn't match either the ledger or what the tool actually returned. Try it with chaosline run --scenario payments/wrong-amount.

You probably have observability already. You might even have eval frameworks. But none of them ask what matters most when tools fail.

Agent failure has two independent dimensions, and mixing them up is why most testing misses the problem cases:

The key insight: a failed task is fine. A lie is never fine. An unintended duplicate charge is an incident that honesty alone can't fix.

You don't need to modify your agent code at all. We intercept at two points:

Tool boundary: An MCP-compatible proxy sits between your agent and its tools. It injects realistic faults (timeouts, wrong data, network errors) and records every single call.

Model boundary: An LLM API proxy (configured via a base-URL override) watches what the model is reasoning and what it finally tells the user.

Then we spin up a mock world with stateful fake tools. After your agent runs, we check the state. Did it double-charge? We count ledger entries. Did it return the wrong balance? We compare against actual state. No opinions from an LLM needed.

npx chaosline run --scenario payments/timeout-after-commit -- python my_agent.py

We ship fault scenarios across 6 different tool worlds: payments, database, email, filesystem, HTTP, and search. Each scenario is tagged for how serious it is: smoke for quick checks, full for thorough testing, and critical for the stuff that could really hurt.

chaosline list --tag smoke
chaosline run --tag smoke -- <agent command>

Got a tool we don't ship with? Run chaosline init and it'll scaffold a starter scenario for you. No need to modify Chaosline itself or add any new world packages. Check out guide/writing-a-scenario.md for the full walkthrough.

It's a monorepo. The thing you install from npm is packages/cli, and everything else is internal infrastructure that powers it.

Want to contribute or hack on Chaosline locally?

npm install
npm run build --workspaces --if-present

Every push runs tests for core, faults, grader, and reporter. Check .github/workflows/ci.yml for the full CI setup.

AGPL v3.0. See packages/cli/LICENSE.