Evals quickstart

Overview

This quickstart shows how to check an assistant or squad’s next decision using a mock conversation. You’ll define the context, choose a judge, and inspect the result.

Evals run at the text and model layer. They don’t test speech recognition, audio quality, or turn-taking. Use Simulations for complete conversations, then controlled real calls for the phone path and live integrations.

For operator-focused guidance on choosing checkpoints and writing durable checks, see test decisions with Evals.

What are Evals?

Evals is Vapi’s AI agent testing framework that enables you to systematically test assistants and squads using mock conversations with automated validation. Test your agents by:

1. Creating mock conversations - Define user messages and expected assistant responses
2. Validating behavior - Use exact match, regex patterns, or AI-powered judging
3. Testing tool calls - Verify function calls with specific arguments
4. Running automated tests - Execute tests and receive detailed pass/fail results
5. Debugging failures - Review full conversation transcripts with evaluation details

When are Evals useful?

Evals help you maintain quality and catch issues early:

• Pre-deployment testing - Validate new assistant configurations before going live
• Regression testing - Ensure prompt or tool changes don’t break existing behaviors
• Conversation flow validation - Test multi-turn interactions and complex scenarios
• Tool calling verification - Validate function calls with correct arguments
• Squad handoff testing - Ensure smooth transitions between squad members
• CI/CD integration - Automate quality gates in your deployment pipeline

What you’ll build

Focused Evals for an appointment-booking assistant that test:

• Asking for missing information
• Tool call requests with specific arguments
• Response pattern matching with regex
• Semantic validation using AI judges
• Decisions at selected conversation checkpoints

These are separate saved Evals, not a native Eval suite. Group and run them yourself or use custom automation.

Prerequisites

Step 1: Create your first evaluation

Start with a booking assistant whose policy requires a timezone before booking. This example checks whether it asks for the missing timezone. Include prior messages establishing any other prerequisites, such as verified identity, so asking for the timezone is the appropriate next decision. Choose an equivalent decision from your own assistant’s requirements if it doesn’t handle bookings.

Step 2: Run your evaluation

Execute the evaluation against your assistant or squad.

Step 3: Understand test results

Learn to interpret evaluation results and identify issues.

Successful evaluation

When all checks pass, you’ll see:

{
  "id": "eval-run-123",
  "evalId": "550e8400-e29b-41d4-a716-446655440000",
  "status": "ended",
  "endedReason": "mockConversation.done",
  "results": [
    {
      "status": "pass",
      "messages": [
        {
          "role": "user",
          "content": "I want to book for January 20, 2027 at 2pm."
        },
        {
          "role": "assistant",
          "content": "What timezone should I use for your appointment?",
          "judge": {
            "status": "pass"
          }
        }
      ]
    }
  ]
}

Pass criteria:

• status is “ended”
• endedReason is “mockConversation.done”
• results[0].status is “pass”
• All judge.status values are “pass”

Failed evaluation

When validation fails, you’ll see details:

{
  "status": "ended",
  "endedReason": "mockConversation.done",
  "results": [
    {
      "status": "fail",
      "messages": [
        {
          "role": "user",
          "content": "I want to book for January 20, 2027 at 2pm."
        },
        {
          "role": "assistant",
          "content": "I will use Pacific time and book that now.",
          "judge": {
            "status": "fail",
            "failureReason": "The assistant assumed a timezone instead of asking."
          }
        }
      ]
    }
  ]
}

Failure indicators:

• results[0].status is “fail”
• judge.status is “fail”
• judge.failureReason explains why validation failed

Step 4: Test tool/function calls

Validate that your assistant calls functions with correct arguments.

Basic tool call validation

Test appointment booking with exact argument matching. This fixture uses an explicit date and an assistant configured to use America/Los_Angeles for bookings. Its bookAppointment tool accepts date and time. Supply any other required identity, availability, or policy context for your assistant before the checkpoint. Use a date your test environment accepts and keep the request and expected arguments in sync.

Tool call validation modes

The mock response doesn’t create an appointment. Verify external calendar state separately using a sandbox integration. Add a paired Eval where the tool returns an error and the assistant must not claim success, plus a case where missing information should prevent the tool call.

Exact match - Full validation:

{
  "judgePlan": {
    "type": "exact",
    "toolCalls": [
      {
        "name": "bookAppointment",
        "arguments": {
          "date": "2027-01-20",
          "time": "14:00"
        }
      }
    ]
  }
}

Validates both function name AND all argument values exactly.

Partial match - Name only:

{
  "judgePlan": {
    "type": "regex",
    "toolCalls": [
      {
        "name": "bookAppointment"
      }
    ]
  }
}

Validates only that the function was called (arguments can vary). Use this only when the name alone is the requirement. Keep argument checks when a wrong date, account ID, or other value would make the action incorrect.

Multiple tool calls:

{
  "judgePlan": {
    "type": "exact",
    "toolCalls": [
      {
        "name": "checkAvailability",
        "arguments": { "date": "2027-01-20" }
      },
      {
        "name": "bookAppointment",
        "arguments": { "date": "2027-01-20", "time": "14:00" }
      }
    ]
  }
}

Validates multiple function calls in sequence.

Step 5: Use regex for flexible validation

When responses vary slightly (like names, dates, or IDs), use regex patterns for flexible matching.

Common regex patterns

Greeting variations:

{
  "judgePlan": {
    "type": "regex",
    "content": "^(Hello|Hi|Hey),? (I can|I'll|let me) help.*"
  }
}

Matches: “Hello, I can help…”, “Hi I’ll help…”, “Hey let me help…”

Responses with variables:

{
  "judgePlan": {
    "type": "regex",
    "content": ".*appointment.*confirmed.*[A-Z]{3}-[0-9]{5}.*"
  }
}

This checks a format, not booking success. It also matches “Your appointment is not confirmed. Failed request reference APT-12345.” Use an AI judge to check meaning and tool checks to validate the requested action.

Date patterns:

{
  "judgePlan": {
    "type": "regex",
    "content": ".*scheduled for (Monday|Tuesday|Wednesday|Thursday|Friday).*"
  }
}

Matches responses mentioning weekdays.

Case-insensitive matching:

{
  "judgePlan": {
    "type": "regex",
    "content": "(?i)booking confirmed"
  }
}

The (?i) flag makes matching case-insensitive.

Example: Flexible booking confirmation

Step 6: Use AI judge for semantic validation

For complex validation criteria beyond pattern matching, use AI-powered judges to evaluate responses semantically.

AI judge structure

{
  "role": "assistant",
  "judgePlan": {
    "type": "ai",
    "model": {
      "provider": "openai",
      "model": "gpt-4o",
      "messages": [
        {
          "role": "system",
          "content": "Your evaluation prompt here"
        }
      ]
    }
  }
}

Writing effective judge prompts

Template structure:

You are an LLM-Judge. Evaluate ONLY the last assistant message in the mock conversation: {{messages[-1]}}.
Include the full conversation history for context: {{messages}}
Decision rule:
- PASS if ALL "pass criteria" are satisfied AND NONE of the "fail criteria" are triggered.
- Otherwise FAIL.
Pass criteria:
- [Specific requirement 1]
- [Specific requirement 2]
Fail criteria (any one triggers FAIL):
- [Specific failure condition 1]
- [Specific failure condition 2]
Output format: respond with exactly one word: pass or fail
- No explanations
- No punctuation
- No additional text

Example: Check for a clarifying question

Supported AI judge providers

Custom LLM:

{
  "model": {
    "provider": "custom-llm",
    "model": "your-model-name",
    "url": "https://your-api-endpoint.com/chat/completions",
    "messages": [...]
  }
}

AI judge best practices

Step 7: Control flow with Continue Plan

Define what happens after an evaluation passes or fails using continuePlan.

Exit on failure

Stop the test immediately if a critical check fails:

{
  "role": "assistant",
  "judgePlan": {
    "type": "exact",
    "content": "I can help you with that."
  },
  "continuePlan": {
    "exitOnFailureEnabled": true
  }
}

Use case: Skip expensive subsequent tests when initial validation fails.

Override responses on failure

Provide fallback responses to continue testing even when validation fails:

{
  "role": "assistant",
  "judgePlan": {
    "type": "exact",
    "content": "I've processed your request."
  },
  "continuePlan": {
    "exitOnFailureEnabled": false,
    "contentOverride": "Let me rephrase that...",
    "toolCallsOverride": [
      {
        "name": "retryProcessing",
        "arguments": { "retry": "true" }
      }
    ]
  }
}

Use case: Test error recovery paths or force specific tool calls for subsequent validation.

Example: Multi-step with exit control

This example assumes the assistant is required to say “This call may be recorded.” exactly. Don’t exact-match ordinary conversational wording. The later override supplies context for debugging; it doesn’t turn a failed checkpoint into a pass or prove that the booking happened.

Step 8: Test complete conversation flows

Validate multi-turn interactions that simulate real user conversations.

Fixed-path booking checkpoints

This example checks selected decisions along a supplied path. It isn’t an end-to-end booking test. Use a Simulation to explore different paths to the outcome. Use the tool and timezone setup from Step 4, and supply the caller’s test email in the mock context before expecting sendEmail.

System message injection

Inject system prompts mid-conversation to test dynamic behavior changes:

{
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "regex",
        "content": ".*help.*"
      }
    },
    {
      "role": "system",
      "content": "You are now in urgent mode. Prioritize speed."
    },
    {
      "role": "user",
      "content": "I need immediate help"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "ai",
        "model": {
          "provider": "openai",
          "model": "gpt-4o",
          "messages": [
            {
              "role": "system",
              "content": "PASS if response shows urgency. FAIL if response is casual. Output: pass or fail"
            }
          ]
        }
      }
    }
  ]
}

Step 9: Manage evaluations

List and update individual Evals. Organize groups in your own tracker or automation.

List all evaluations

Update an evaluation

Delete an evaluation

View run history

Expected output

Successful run

{
  "id": "eval-run-123",
  "evalId": "550e8400-e29b-41d4-a716-446655440000",
  "orgId": "org-123",
  "status": "ended",
  "endedReason": "mockConversation.done",
  "createdAt": "2024-01-15T09:35:00Z",
  "updatedAt": "2024-01-15T09:35:45Z",
  "results": [
    {
      "status": "pass",
      "messages": [
        {
          "role": "user",
          "content": "Hello"
        },
        {
          "role": "assistant",
          "content": "Hello! How can I help you today?",
          "judge": {
            "status": "pass"
          }
        }
      ]
    }
  ],
  "target": {
    "type": "assistant",
    "assistantId": "your-assistant-id"
  }
}

Indicators of success:

• ✅ status is “ended”
• ✅ endedReason is “mockConversation.done”
• ✅ results[0].status is “pass”
• ✅ All judge.status values are “pass”

Failed run

{
  "id": "eval-run-124",
  "status": "ended",
  "endedReason": "mockConversation.done",
  "results": [
    {
      "status": "fail",
      "messages": [
        {
          "role": "user",
          "content": "Book an appointment for January 20, 2027 at 2pm Pacific time"
        },
        {
          "role": "assistant",
          "content": "Sure, let me help you with that.",
          "toolCalls": [
            {
              "name": "bookAppointment",
              "arguments": {
                "date": "2027-01-20",
                "time": "2:00 PM"
              }
            }
          ],
          "judge": {
            "status": "fail",
            "failureReason": "Tool call arguments mismatch. Expected time: '14:00' but got: '2:00 PM'"
          }
        }
      ]
    }
  ]
}

Indicators of failure:

• ❌ results[0].status is “fail”
• ❌ judge.status is “fail”
• ❌ judge.failureReason provides specific details

Common patterns

Multiple validation types in one eval

Choose a judge for each checkpoint’s requirement, not to include every judge type in one test. Use exact matching for required tool arguments or mandated wording, regex for a stable format, and an AI judge for a decision’s meaning. Split independent questions into separate checks so a failure is easy to explain. Use a Simulation when you want to test a complete journey through different valid paths.

Test squad handoffs

Validate smooth transitions between squad members:

{
  "name": "Squad Handoff Test",
  "messages": [
    {
      "role": "user",
      "content": "I need technical support"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "exact",
        "toolCalls": [
          {
            "name": "transferToSquadMember",
            "arguments": {
              "destination": "technical-support-agent"
            }
          }
        ]
      }
    }
  ],
  "target": {
    "type": "squad",
    "squadId": "your-squad-id"
  }
}

Regression tests

Keep Evals for important behavior and issues you have fixed. For example, create separate Evals for formal, casual, and multilingual greetings, each with its own expected response or judge criterion.

Run each relevant Eval after changing the greeting behavior. You can select individual Evals in the dashboard or use the API to run them from your own automation. See test decisions with Evals for guidance on turning real failures into regression coverage.

Troubleshooting

Common errors

“mockConversation.done” not reached:

• Check endedReason for actual error (e.g., “assistant-error”, “pipeline-error-openai-llm-failed”)
• Verify assistant configuration (model, voice, tools)
• Check API key validity and rate limits

Judge validation fails unexpectedly:

• Review actual vs expected output in failureReason
• For exact match: Check for extra spaces, punctuation, or case differences
• For regex: Test pattern with online regex validators
• For AI judge: Verify prompt clarity and binary pass/fail logic

Tool calls not validated:

• Ensure tool is properly configured in assistant
• Check argument types match exactly (string “14:00” vs number 14)
• Verify tool function names are spelled correctly

Next steps

Tips for success

Get help

Need assistance? Try these resources:

• Eval API Reference
• Discord Community
• Support