You fixed that agent bug last week. It came back today.
replayd makes sure that never happens again.

pip install replayd

• The problem
• Who is replayd for
• How it works
• Quickstart
• See it working
• Why replayd
• Framework integrations
• How replayd compares
• Example agents
• Recording tool calls
• Auto-instrumentation limitations
• Grading
• Storage
• CI integration
• Design principles
• Roadmap
• FAQ
• What replayd is not
• What builders say
• Star goals
• Part of TAQ by Stonepath Labs
• Contributing
• Star history

replayd is for teams shipping agents that can fail in ways they cannot afford to repeat:

• customer support and refund approval agents
• tool-calling and function-calling agents
• RAG and retrieval agents
• internal workflow and orchestration agents
• coding, browser, and planning agents

If your agent can fail in a way you do not want repeated, replayd turns that failure into a test.

replayd has three concepts: Capture, Grade, and Gate.

  YOUR AGENT FAILS IN PRODUCTION
              │
              ▼
    ┌─────────────────────┐
    │       CAPTURE       │  rp.capture() wraps the run.
    │                     │  Records tool calls, input,
    │   rp.capture(...)   │  output, and model used.
    └──────────┬──────────┘
               │
               ▼
    ┌─────────────────────┐
    │        GRADE        │  You define what "wrong" means:
    │                     │  forbidden_actions, expected_action,
    │   rp.save_test(...) │  or a grader_prompt for semantic eval.
    └──────────┬──────────┘
               │
               ▼
    ┌─────────────────────┐
    │        GATE         │  On every future change, replayd
    │                     │  replays the test. If the failure
    │  rp.replay_all(...) │  returns, the deploy is blocked.
    └─────────────────────┘

  One failed run → one saved test → no regression ever ships.

That is the entire model. No new abstractions. No configuration DSL. Capture the real failure, define what wrong looks like, replay it before every ship.

from replayd import Replayd

rp = Replayd()

# 1. Capture a run — assign run.output inside the block
with rp.capture(input=user_input, model="gpt-4o") as run:
    run.output = your_agent.run(user_input)

# Note: wrap your agent to record tool calls — see "Recording tool calls" below

# 2. Mark it as failed
rp.mark_failed(run.id, reason="agent approved refund after policy limit")

# 3. Save as a regression test
rp.save_test(
    run.id,
    forbidden_actions=["approve_refund"],
    expected_action="escalate",
)

# 4. Later — after changing your prompt or model — replay all tests
results = rp.replay_all(agent=your_agent_fn)

for r in results:
    print(r.verdict, r.reason)

Run the included example (python examples/basic_example.py) and you get:

Capturing a refund-approval agent run...
  agent called: approve_refund(amount=1200)  [policy limit is $500]
  output: {'action': 'approve_refund', 'amount': 1200}

Marking run as failed...
  reason: agent approved refund of $1200, exceeding $500 policy limit

Saving as regression test...
  forbidden: approve_refund  |  expected: escalate

-----------------------------------------
Replay #1 -- buggy agent (regression should be caught)
  [FAIL] Forbidden action 'approve_refund' was called during replay.

Replay #2 -- fixed agent (regression should be resolved)
  [PASS] No forbidden actions called; all expected actions present.
-----------------------------------------
1 failure caught. 1 resolved.

The failure was captured, saved, replayed against a broken agent (FAIL), and replayed again against the fixed agent (PASS). That is the full loop.

AI agents do not only fail once. They regress. You change a prompt, a model, a tool schema, or a retrieval setup, and something that used to work quietly breaks again. Traditional software has regression tests and CI/CD to catch this. AI agents have had nothing equivalent.

replayd is the open source fix. Failed runs become replayable tests. Old failures cannot return undetected.

replayd is framework-agnostic. The capture-and-replay pattern works with any agent that can be wrapped as a Python callable.

Works with any LLM provider: OpenAI, Anthropic, Gemini, Groq, Mistral, or local models via Ollama. replayd does not call your LLM — it wraps your agent.

Adding an integration? See Contributing.

replayd is not an alternative to observability tools. It works alongside them. LangSmith and Langfuse tell you what happened. replayd makes sure the worst things cannot happen again.

Five production-grade example agents are included. Run any of them with no API key required — all grading is structural.

Run the no-API-key examples:

python examples/multi_step_planning_agent.py
python examples/rag_policy_agent.py
python examples/incident_response_agent.py

Each example shows FAIL on the buggy agent and PASS on the fixed agent.

Call rp.instrument_openai(client) or rp.instrument_anthropic(client) once, before entering any capture block. Tool calls are then recorded automatically — no manual wrapping needed.

from openai import OpenAI
from replayd import Replayd

rp = Replayd()
client = OpenAI()
rp.instrument_openai(client)  # call once

with rp.capture(input=user_query, model="gpt-4o") as run:
    run.output = your_agent(client, user_query)  # tool calls recorded automatically

Works for Anthropic too:

import anthropic
client = anthropic.Anthropic()
rp.instrument_anthropic(client)

See examples/real_openai_agent.py for a complete runnable example.

If your agent does not use OpenAI or Anthropic directly, wrap your tool dispatcher to record calls manually. The agent you pass to replay_all must accept two arguments: (input, run_ctx).

def my_agent(input, run_ctx):
    result = call_tool("search", {"query": input["query"]})
    run_ctx.record_tool_call("search", {"query": input["query"]}, result)
    # ... rest of agent logic
    return final_output

Pass this two-argument callable to replay_all:

results = rp.replay_all(agent=my_agent)

rp.uninstrument_openai(client)
rp.uninstrument_anthropic(client)

Both calls are idempotent. After them the client is exactly as it was before instrument_* was called. Useful in test teardown to avoid cross-test pollution.

What is covered

¹ replay_all calls agents synchronously. Wrap async agents with asyncio.run() for replay:

import asyncio

def sync_wrapper(input, run_ctx):
    return asyncio.run(my_async_agent(input, run_ctx))

results = rp.replay_all(agent=sync_wrapper)

Streaming (stream=True) — not supported

When stream=True is passed inside an active capture block, the wrapper emits a warnings.warn() and passes through unchanged — tool calls are not recorded. Disable streaming for captured runs, or record manually:

run_ctx.record_tool_call("tool_name", arguments, result)

Final tool call with no follow-up model call

Tool calls are recorded when the result arrives back as a role: "tool" message in the next API call. If your agent executes the last tool, uses the result in Python code, and never sends it back to the model, that call is not recorded. Use record_tool_call() for it explicitly.

The pattern that is fully covered without any manual work:

# sync or async — both work
while True:
    response = client.chat.completions.create(messages=messages, tools=tools)
    msg = response.choices[0].message
    if msg.tool_calls:
        for tc in msg.tool_calls:
            result = execute_tool(tc.function.name, tc.function.arguments)
            messages.append({"role": "tool", "tool_call_id": tc.id, "content": str(result)})
    else:
        break  # final answer

replayd does not grade on exact output matching. LLMs are non-deterministic — the same correct behavior will produce different output text every run, so exact matching creates false failures. The wrong tool being called, however, is a fact. replayd grades on facts.

The structural check always runs first. If a forbidden action fires, the test fails immediately without calling the LLM.

For failures that can only be evaluated by reading the output:

rp.save_test(
    run.id,
    grader_prompt="Did the agent approve a refund that exceeds the $500 policy limit?",
)

Requires:

pip install "replayd[semantic]"
export ANTHROPIC_API_KEY=sk-...

Runs and tests are stored as JSON files in .replayd/ in your working directory:

.replayd/
  runs/<run-id>.json    ← full record of each captured run
  tests/<test-id>.json  ← saved regression tests

No database. No hosted backend. Commit .replayd/tests/ into version control to share regression tests with your team — this gives your team a traceable record of every known agent failure and its expected behavior, an audit trail that grows with every failure your agents encounter in production. Keep .replayd/runs/ out of git — it is local capture data.

A ready-to-use script is included at scripts/regression_check.py. Copy it into your repo, replace the agent import, and add this to your workflow:

# .github/workflows/regression.yml
- name: Run regression tests
  run: python scripts/regression_check.py

Any saved regression test that fails exits with code 1, blocking the deploy. This makes replayd a policy enforcement gate in your CI pipeline — any agent behavior that violates a known expectation blocks the release before it reaches users.

These four principles drive every decision in replayd.

1. Grade behavior, not output. LLM output is non-deterministic by design. Grading the exact text a model returned is fragile and creates noise. replayd grades what the agent did — which tools it called, in what order, with what arguments. That is deterministic. That is what matters.

2. Capture from real failures, not from specs. Most evaluation tools ask you to write tests from a specification. replayd captures from actual failures. A test that comes from a real production failure is worth ten that were written hypothetically. Real failures encode the exact context, input, and state that caused the problem.

3. Zero dependencies on the critical path. The core capture-and-replay loop requires no external services, no hosted backend, no LLM call. You can build up a full regression suite entirely offline. The LLM-as-judge grader is opt-in and only runs when deterministic grading is insufficient.

4. One correct action beats a correct output. A good agent escalates when it should escalate. It calls the right tool in the right order. The exact phrasing of its explanation is secondary. replayd enforces what the agent did, not how it phrased the result.

Want to help ship any of these? See Contributing.

Does replayd require an API key to use? No. The core capture-and-replay loop with structural (deterministic) grading runs with zero external dependencies. An API key is only needed if you use grader_prompt for semantic grading via LLM-as-judge (pip install "replayd[semantic]").

Does it work with any LLM provider? Yes. replayd wraps your agent as a callable and never interacts with your LLM directly. OpenAI, Anthropic, Gemini, Groq, Mistral, or a local model via Ollama — the provider does not matter.

Does it work with any agent framework? Yes, if the framework can be wrapped as a two-argument Python callable (input, run_ctx) -> output. LangChain and OpenAI Agents SDK examples are included. CrewAI and AutoGen patterns are planned.

Do my tests break if the model gives a different (but correct) output? No. Structural tests check tool calls, not output text. They will not produce false positives because the model rephrased a correct answer differently. Semantic grading also evaluates meaning, not exact text.

Should I commit .replayd/ to git? Commit .replayd/tests/ — this is your regression suite and should be shared with your team. Do not commit .replayd/runs/ — these are local capture files and should stay out of version control.

How is this different from prompt testing tools like PromptFoo? PromptFoo and similar tools help you evaluate prompt quality on hypothetical test cases you write upfront. replayd captures real production failures and turns them into regression tests. The workflow is capture-first, not specification-first. The tests come from reality, not from what you imagined could go wrong.

Can I run replayd in CI without any secrets? Yes. As long as you use only structural grading, replayd runs with no secrets at all. If you use grader_prompt, you need ANTHROPIC_API_KEY set in your CI environment.

replayd is not an observability tool. LangSmith, Braintrust, and Arize tell you what happened after the fact. replayd is an active release gate — it replays known failures before you ship. Passive vs active. That is the distinction.

"If something solved this it would definitely be worth paying for." — r/ycombinator

"Replaying old failures against new prompts and models should be standard at this point. Otherwise the same bugs just keep coming back quietly." — r/LLMDevs

"The capture step has too much friction. There's your next action item." — r/LLMDevs

Every star helps more builders find replayd. If it has saved you from a regression, star it.

replayd is the open source core of TAQ — the full AI release control platform.

TAQ adds: a dashboard, hosted backend, team access controls, release gate enforcement, and audit logs. replayd gets your team started with the concept. TAQ is what you run it on in production.

stonepathlab.net

Bug reports and pull requests are welcome. Open an issue on GitHub to discuss anything before sending a large PR.

The build has no dependencies — pip install -e ".[dev]" gives you everything needed to run tests:

pip install -e ".[dev]"
pytest

Good first contributions:

• Add a CrewAI integration example
• Add an AutoGen integration example
• Add a LlamaIndex integration example
• Add regression scenarios for a real agent type
• Improve the getting started documentation
• Build the replayd run CLI command

MIT — see LICENSE.