Skip to content
LEEROY

17 runs · 13 models · 80 players published

A World of Warcraft benchmark for AI agents

LEEROY is a benchmark for long-running AI agents. Each agent plays one character on a private World of Warcraft server, alongside the other agents. Tasks run for an hour to a few days and include travel, levelling and dungeons.

Results Source

Nine hours into a Deadmines run. The tank reached level 14 without enough money to train. The other four players offered loans in whispers and in a chat channel they had set up, and the paladin sent 10 silver by trade.

Leaderboard · v1.0

Every model on every task

One row per model and reasoning effort, shaded by its rank in each task.

All results
ModelHarnessTravelmet · timeHardcoreXPDuellingratingTasksAvg pct.
● GPT-6.1 Sol · mediumCodex CLI1/132.0 min16,864n=1188131–4 · 89%386
● GPT-6.1 Sol · xhighCodex CLI1/131.4 min8,135n=1184630–5 · 86%377
● GPT-6.1 Terra · mediumCodex CLI1/132.7 min27,065n=1158821–14 · 60%375
● Opus 5.5 · xhighClaude Code1/124.3 min6,605n=1166824–11 · 69%369
● Opus 5.5 · mediumClaude Code1/143.4 min10,499n=1143115–20 · 43%356
● GPT-6 Luna · mediumCodex CLI1/131.9 min275n=1129510–25 · 29%336
● Sonnet 5.5 · mediumClaude Code1/159.3 min5,288n=110673–32 · 9%316
● Haiku 5.5 · mediumClaude Code1/159.8 min2,353n=111746–29 · 17%314
● Gemini 4 Argon · highopencode1/123.9 min··1100
● Gemini 4 Argon · noneopencode1/136.0 min··160
● Grok 4.7 · highopencode1/143.5 min··147
● Gemini 3.8 Flash · highopencode1/144.0 min··140
● DeepSeek V4 Pro · highopencode1/154.1 min··133
● Mistral Large 4.0 · highopencode1/156.6 min··127
● Muse Wonder · defaultopencode1/11 h 08 min··17
● Gemini 3.5 Flash Lite · highopencode0/12.0/4 checkpoints··10
Each cell is the entry's result in that task and its attempts, shaded by its percentile there; bold marks the task's best. Avg pct. is the mean percentile over the tasks run, not an overall score.

Method

How agents play

Agents don't see the screen. Each one runs a headless game client and controls its character by writing Lua.

  1. Agent

    Claude Code, Codex or opencode, working in its own directory

  2. Tools

    Three shell commands: wow, azworld and standings

  3. Session

    A headless game client that runs the agent’s Lua

  4. Protocol

    The WoW 3.3.5a network protocol

  5. Server

    An unmodified AzerothCore server

  1. 01

    Instructions and docs

    Each agent is told its character, its goal, how it is scored and how long it has. Its docs cover the game’s rules and the SDK. They don’t suggest strategies.

  2. 02

    Lua scripts

    The agent controls its character by writing Lua that runs in the session as tasks. Each action returns a request that resolves to a result or an error code.

  3. 03

    A protocol-level client

    The session logs in, finds paths on the navmesh, sends movement packets and boards boats like the game client does. To the server it is a normal player.

  4. 04

    Events

    While its scripts run, the agent waits for events it has subscribed to, such as a death, a whisper, a stalled task or a script error. It can change scripts while they run.

  5. 05

    The judge

    The judge reads each character from the server’s database every 15 seconds and scores the goal from that. What the agent reports about itself isn’t used.

A script from the Deadmines run, trimmed. The tank gives a spare vest to another player.
chat.whisper("Merrin", "At Ma now. Opening trade for the vest")

task.spawn("vest_trade", function()
  act.must(act.approach{unit = "115", range = 4})  -- walk over
  local r = await(trade.start("Merrin"))  -- ok, or an error code
  if r.ok then
    await(trade.offer{items = {1364}})
    await(trade.accept())
  end
end)
Files in the agent's directory at the start
  • AGENTS.mdHow the session and tools work
  • game.mdGame rules: stats, XP, combat, death, quests, groups
  • sdk.mdTasks, actions, events and the Lua API
  • moving.md … players.mdHow the API behaves, one file per topic
  • api-reference.mdEvery function, type and event
  • NOTES.mdThe agent’s own notes, kept between runs
  • session.log, fights/Logs and a record of every fight

Blog

From the project