World of Workflows (WoW) Benchmark Scenario

This scenario evaluates agents on the World of Workflows benchmark, which tests AI agents on realistic enterprise tasks using a ServiceNow-based mock system with over 6,000 interlinked tables and 93 workflows.

uv sync --extra wow-purple --extra wow-green

./setup.sh

This sparse-clones the WoW repository, pulling only the data files (~67 trajectories, constraint data, task completion data).

# Required for purple agent LLM calls
OPENROUTER_API_KEY=your-openrouter-key-here

# Optional: override model (default: openai/gpt-4.1 via OpenRouter)
WOW_AGENT_LLM=openai/gpt-4.1

# Required only for agentic eval_type
SNOW_INSTANCE_URL=https://dev123456.service-now.com
SNOW_INSTANCE_UNAME=admin
SNOW_INSTANCE_PWD=your-password-here

For eval_type = "agentic", a live ServiceNow developer instance is required:

1. Go to https://developer.servicenow.com/dev.do and request a free developer instance.
2. Set the SNOW_* variables above.
3. Log into your instance in a browser and close any popups before running.

uv run agentbeats-run scenario.toml

If you see Error: Some agent endpoints are already in use, update the endpoint + --port values in scenario.toml.

Edit scenario.toml to configure the benchmark:

[config]
eval_type = "state"     # state | action | constraint | agentic
num_tasks = 5           # number of tasks to evaluate
custom_schema = false   # use custom schema generation (state/action only)

The agent LLM defaults to openai/gpt-4.1 and can be set via the WOW_AGENT_LLM env var or --agent-llm CLI argument in purple/src/server.py.

agentbeats-run / client_cli
       │
       │  EvalRequest JSON (eval_type, num_tasks, ...)
       ▼
┌─────────────────────────────────┐
│  Green Agent (Evaluator) :9029  │
│  evaluator/src/                 │
│  - Loads WoW data files         │
│  - Formats task prompts         │
│  - Scores purple's predictions  │
└───────────────┬─────────────────┘
                │  A2A (task prompt per item)
                ▼
┌─────────────────────────────────┐
│  Purple Agent (Competitor) :9039│
│  agent/src/                     │
│  - LiteLLM + OpenRouter LLM     │
│  - Returns JSON prediction      │
│  - (agentic: ServiceNow API)    │
└─────────────────────────────────┘

• evaluator/src/ (Green Agent): Orchestrates benchmark tasks, evaluates responses, reports scores
• agent/src/ (Purple Agent): The agent under test — receives task prompts and responds with structured predictions

The WoW benchmark provides pre-built data at wow-bench/src/wow/data_files/:

• Trajectories (67 total): trajectories/ — workflow action histories with ground-truth state diffs
• Constraint data: constraint_violation_data/ — constraint violation tasks
• Task completion data: task_completion_data/ — agentic task descriptions
• Pre-generated predictions: action_pred/, state_pred/ — GPT-5.1, Gemini-3-pro, Claude Sonnet-4.5 baselines