Make quality a release check.
Build datasets, score the behavior that matters, and compare runs when prompts, models, or tools change.
Evaluate your agentsGive agents a workspace, a plan, memory, and subagents. Pydantic AI Harness is built for work that runs for hours, with tool and output validation to catch invalid data and durable execution to recover from interruptions.
Open source. MIT licensed. Your models, your infrastructure. Enterprise support for when it really matters.
Research the launch. Draft a brief.
Get approval before sending.
Harness keeps a plan, workspace, and working context as the agent breaks the research into smaller tasks.
Tools and subagents do the research. Invalid tool arguments or structured outputs produce feedback the model can use to retry.
A tool needs approval. The workflow waits, and a durable runtime preserves progress while the person decides.
After approval or a worker restart, the durable backend restores the workflow so the agent can continue from saved progress.
The agent returns a validated research brief. You can test its shape, evaluate its quality, and inspect the work in a trace.
Already doing real work
Datalayer moved from LangChain to a readable, typed foundation for its Jupyter agents.
Read their story OverjoyOverjoy runs its agent stack on Pydantic AI, from customer conversations to long-running work.
Read their story MixamMixam tests different models while keeping its customer-facing print assistant robust.
Read their storyBuild coding and research agents with a workspace, planning, subagents, and cross-session memory. Pydantic AI Harness gives you composable capabilities for hours of work, on the same Agent API. Start with a complete harness or control each capability yourself.
pydantic-ai-harnessRead and edit files, run allowed commands, and work inside a local or sandboxed environment.
Keep track of the job, delegate to subagents, and bring their findings back into the parent run.
Manage growing conversations, bound tool output, and add memory across sessions.
Validate tool inputs and structured results. Add guardrails for your rules, with feedback to retry when a check fails.
agent = Agent(
'openai:gpt-5.6-sol',
capabilities=[
Coder(),
Memory(FileStore('.memory')),
],
) Coder() brings the workspace, planning, and subagents. Memory() adds knowledge across sessions.
Run in a workspace the agent may edit. Install pydantic-ai-harness[cli], pydantic-ai-slim[openai], and logfire, and set OPENAI_API_KEY.
from pydantic_ai import Agent
from pydantic_ai_harness import (
Coder, Memory,
)
from pydantic_ai_harness.memory import (
FileStore,
)
import logfire
logfire.configure(
send_to_logfire='if-token-present',
)
logfire.instrument_pydantic_ai()
agent = Agent(
'openai:gpt-5.6-sol',
capabilities=[
Coder(),
Memory(FileStore('.memory')),
],
)
# Chat with the agent in your terminal.
agent.to_cli_sync() Use Coder or Researcher, then add, replace, or remove capabilities.
A research job takes hours. An approval takes until tomorrow. A worker disappears halfway through. Run your Pydantic AI agent on a durable backend to preserve progress and continue the workflow.
Choose a durable runtimeSave progress in the durable backend.
The workflow can wait for a human decision.
The runtime recovers the persisted workflow.
Resume the agent with the approved result.
Also integrates with Kitaru and Apache Airflow through third-party backends.
Long runs turn one result into the next step’s input. Pydantic AI validates tool arguments and structured outputs, then sends validation errors back to the model so it can retry. Define the checks your workflow has to pass before it continues.
OpenAI · Anthropic · Google · Bedrock · Mistral · Groq · Ollama
Explore providers and fallbacks@agent.output_validator
def require_sources(
brief: ResearchBrief,
) -> ResearchBrief:
if not brief.sources:
raise ModelRetry(
'Cite your sources.'
)
return brief The model gets Cite your sources.
and another attempt within your retry budget.
Install pydantic-ai and logfire, plus pytest for the tests. Set OPENAI_API_KEY to run
the agent. The tests use a local model double; Logfire export is optional.
from pydantic import BaseModel
from pydantic_ai import (
Agent, ModelRetry, RunContext,
)
import logfire
logfire.configure(
send_to_logfire='if-token-present',
)
logfire.instrument_pydantic_ai()
class ResearchBrief(BaseModel):
summary: str
sources: list[str]
follow_up: list[str]
agent = Agent(
'openai:gpt-5.6-sol',
deps_type=list[str],
output_type=ResearchBrief,
defer_model_check=True,
)
@agent.tool
def read_notes(
ctx: RunContext[list[str]],
) -> list[str]:
"""Read this customer's notes."""
return ctx.deps
@agent.output_validator
def require_sources(
brief: ResearchBrief,
) -> ResearchBrief:
if not brief.sources:
raise ModelRetry(
'Cite your sources.'
)
return brief
if __name__ == '__main__':
result = agent.run_sync(
'Read notes. Write a brief.',
deps=['We need a Python SDK.'],
)
print(result.output.summary)import pytest
from pydantic_ai import (
UnexpectedModelBehavior,
)
from pydantic_ai.messages import (
ToolReturnPart as ToolResult,
)
from pydantic_ai.models.test import (
TestModel,
)
from research_agent import agent
def test_research_brief():
notes = ['We need a Python SDK.']
output = {
'summary': 'Ship a Python SDK.',
'sources': ['Customer notes'],
'follow_up': ['Prototype SDK'],
}
model = TestModel(
call_tools=['read_notes'],
custom_output_args=output,
)
with agent.override(model=model):
result = agent.run_sync(
'Write a brief from notes.',
deps=notes,
)
tool_name = 'read_notes'
tool_results = [
part.content
for msg in result.all_messages()
for part in msg.parts
if isinstance(part, ToolResult)
and part.tool_name == tool_name
]
assert tool_results == [notes]
assert result.output.follow_up == [
'Prototype SDK'
]
def test_rejects_missing_sources():
model = TestModel(
call_tools=[],
custom_output_args={
'summary': 'Ship a SDK.',
'sources': [],
'follow_up': [],
},
)
with agent.override(model=model):
with pytest.raises(
UnexpectedModelBehavior
):
agent.run_sync(
'Write a brief.',
deps=[],
) Build a conversation that can do something. Stream speech in and out, handle interruptions, and call your backend tools during the conversation.
The session shares your agent’s dependencies, instructions, and message history. Hand the conversation to a text agent afterward to produce a structured follow-up.
Build a voice agentOpenAI · Azure · Google Gemini · xAI
Connect your frontend or audio transport“Can you move my booking to Friday?”
check_availability(day="Friday") 7 pm available “There’s a table at seven. Shall I move it?”
Build datasets, score the behavior that matters, and compare runs when prompts, models, or tools change.
Evaluate your agentsFollow model calls, tool calls, and application work in one trace. Investigate latency, costs, and failures with the context around them.
See agent observabilityRoute model requests through a shared gateway with spend controls. Keep the agent code and the operational policy separate.
Explore AI GatewayUse Logfire for the closest integration, or send traces to your existing OpenTelemetry backend.
Instrumentation docsWith Pydantic AI you have everything there readable. You have Python, it runs on Pydantic Graph. So this is also something I like. The graph is open and understandable.
Pydantic AI and Pydantic AI Harness are open source under the MIT license. You pay your model providers and hosting services. Logfire and AI Gateway are optional Pydantic products with their own pricing.
Use Harness for coding, research, and other multi-step jobs that need a workspace, planning, delegation, memory, and context management. Start with Coder or Researcher, or compose individual capabilities on any Pydantic AI agent.
Yes, when you run it with a durable execution backend. Pydantic AI integrates with Temporal, DBOS, Prefect, Restate, and AWS Lambda durable functions, with additional third-party integrations. The backend owns persistence and recovery; a standard in-memory Agent run does not persist itself.
Realtime sessions use Pydantic AI tools, dependencies, instructions, and message history. Your application connects the microphone, speaker, or browser transport. The conversation can then continue as a text-agent run for a structured follow-up.
You can call model providers directly and export OpenTelemetry traces to a compatible backend. Logfire adds agent views, traces, evals, and production diagnostics. AI Gateway adds shared provider access and operational controls.
Give it the whole job.
Your workspace. Your model. Your rules.