WindTunnel

Benchmark WebMCP against other methods browser agents use to interact with websites.

WebMCP solves 100% of tasks, with 2.5–7.5× faster median execution, 3–47× lower median cost, and 27–50% higher scores than the median of other methods.

Quick start · Results · Run data · Methodology · Cost · WebMCP spec

WindTunnel compares WebMCP with other ways browser agents interact with websites. It runs the same tasks on the same sites and measures success rate, execution time, token usage, and cost.

Just want the results? Jump to them.

Run the harness for free (Node 20.11+, no key, no Docker):

npm ci
WT_FAKE_LIFECYCLE=1 npm run bench    # no LLM — scores 0/7 by design, just proves it runs

Run the real benchmark — needs Docker, Linux or macOS, and a key (setup):

npx playwright install chromium
export ANTHROPIC_API_KEY=sk-ant-...
npm run bench -- --arms wm-claude,cu-claude --budget 2

Runs the 7 tasks on the three lightweight sites once each, same model on two interfaces — WebMCP against screenshots. 14 attempts, well under $1; --budget stops launching further attempts once accumulated spend exceeds it.

A browser agent can operate a website through four main interfaces:

1. Screenshots (computer use) — it reads rendered images of the page and acts by coordinate.
2. Page structure — it reads the page's DOM and accessibility tree.
3. Code execution — the model writes browser-automation code (Playwright) that reads and drives the page; OpenAI's recommended mode for GPT-6 Astra.
4. WebMCP — the website exposes direct actions (add_to_cart(id), book_slot(time)) for the agent to call. (What is WebMCP?)

Canonical board v1.2: 2026-09-18 — 21 configurations × 49 tasks across 8 sites × 3 attempts = 3,087 attempt rows and 1,029 majority verdicts, with a 600s per-attempt cap. What changed since v1.1: CHANGELOG.md.

Ten configurations solve 49/49 tasks: all nine WebMCP configurations and GPT-6 Astra on code execution, OpenAI's recommended computer-use mode. Raw task-solve rate therefore does not separate WebMCP from the best screen-driving configuration; cost and time do.

New: Jev + Mercury 2.5 solves 49/49 tasks with WebMCP and 25/49 with ultrafast DOM controls, without WebMCP. WebMCP leads the composite score and is 1.7× faster for this pair. The page setup costs less per median attempt ($0.0008 vs. $0.0011), but succeeds less often.

Five configurations pass every attempt (147/147): Gemini 3.6 Flash, Sonnet 5 (native and Stagehand v4) and Claude Opus 5 via WebMCP, and GPT-6 Astra via code execution. For the same model, Astra's code-execution attempts cost $35.53 total vs. $3.33 for native WebMCP (10.7×), while median agent time was 16.4s vs. 6.3s.

_{Regenerate with node scripts/readme-charts.mjs — it reads results/canonical and fails if any label would overflow its column.}

Canonical artifacts — CSV, run JSON, provenance, changelog, and interactive explorer.

_{Attempts are successful attempts out of 147; Turn-cap counts attempts that used every turn of their model-turn budget (whether or not the final turn produced an answer). Medians are rounded by the chart generator (JavaScript toFixed; an exact 6.85 s prints as 6.8 s). Median tokens are total processed: uncached input + cache reads + cache writes + output. The table reports tasks solved by a majority of three attempts; infrastructure rows are excluded.}

_{Headline ranges compare each WebMCP configuration with the median of all non-WebMCP configurations: 24.03s, $0.05024, and a composite score of 64.4. Each configuration has equal weight; time and cost include failed attempts.}

Turn budgets are a material separator. None of the 1,323 WebMCP attempts hit the turn budget, versus 211 of 1,764 screen-driving attempts. Matched like-for-like — each model's native WebMCP run against its own computer-use run, 882 attempts per side — the count is 0 vs. 124. The screen-driving arms are given roughly 3× larger budgets because a screenshot agent needs about three model turns per journey step.

Model snapshot reporting has a harness limitation. Stagehand and browser-use cannot report the exact model snapshot they were served, so those rows use snapshot_source: unavailable:<harness>. The configured model is confirmed out of band with scripts/verify-model.mjs.

The following table covers the six native WebMCP/screenshot model pairs; it excludes Jev's page-control setup.

On long journeys the efficiency gap widens sharply. Across the 12 long and sensitive-action tasks, WebMCP solved 72/72 model-task cells vs. computer use's 69/72 (96%), using 12.3× lower cost, 5.8× less agent time, and 13.0× fewer tokens. Until board v1.1 the sensitive-action row was a deliberate coverage boundary: the store's WebMCP tools handed guest checkout back to the page before the final purchase (task md-8), so no WebMCP configuration could complete it while screen-driving agents could. v1.1 added a complete_checkout tool and re-measured that cell — see CHANGELOG.md.

_{How these numbers are computed: docs/SPEC.md.}

Every task runs on the same site, from the same seeded state, scored by the same check, for every method — so a difference in outcome isn't the site or the data. What differs is the whole method configuration: interface, model, framework, and turn budget.

• Real, self-hosted applications. Production open-source apps, not synthetic pages. Each runs locally in Docker, pinned to a fixed version.
• Outcome-based scoring. No human or model judges a run: WindTunnel inspects the resulting application state — is the item in the cart, does the appointment exist — and records pass or fail, plus what the attempt cost.
• Integrity. Checked values are generated fresh from a seed, state-changing tasks are scored by inspecting the application, and published final answers are redacted. The Jev release includes redacted transcripts; the combined result rows stay compact. Memorization risk and its limits: docs/SPEC.md.
• Scorer correction. Independent review found that the canonical merge had been built before corrected predicates were applied. Re-scoring fixed 19 false negatives — 19 promotions and 0 demotions — and the corrected rows are marked in the canonical CSV.

Eight applications across public and authenticated pages and different stacks — six live targets, two read-only controls.

49 benchmark tasks across the eight sites, plus 10 calibration tasks for cost measurement. Four difficulty tiers by journey length:

Each task is attempted N times (3 by default). A task is solved when a majority of attempts pass; the headline is solved ÷ total, reported next to median time, tokens, and cost.

Each task also carries a per-interface turn budget — a screenshot agent needs ~3 turns per step, a tool-calling agent ~1 (defaults: docs/SPEC.md).

The board spans 13 implementations and 21 configurations: nine with WebMCP and twelve using screenshots, page structure, or code execution. Models include Sonnet 5, Opus 5, GPT-5.6 Luna, GPT-5.6 SOL, GPT-6 Astra, Gemini 3.6 Flash, and Jev + Mercury 2.5. Every configuration uses the same 49 tasks and scoring.

Jev makes decisions; Mercury writes arguments or field values and the final answer. The WebMCP and ultrafast page setups use separate frozen harnesses, so this is a comparison of complete setups. Their runner code and run instructions are published separately from the standard CLI. Versions, accounting and redacted transcripts.

Where the tokens go. How much an agent reads each turn is set by the interface:

• Screenshots (computer use) — a full page image every turn: thousands of tokens each, refreshed nearly every step.
• Page structure (DOM / accessibility tree) — the page's text every turn, the whole page re-read each step. DOM + vision sends the text and a screenshot, and is often the heaviest configuration of all.
• WebMCP — a short list of tool schemas plus small JSON results. No page text, no screenshots. Usually less context.

Task length multiplies it. Page-reading interfaces grow fastest because they re-read the page each step. Across the native model pairs, WebMCP's median cost advantage grows from about 6× on short tasks to 11–13× on longer journeys.

Per-task medians across the current 21-configuration leaderboard (tokens include cache reads and cache writes; pricing detail in docs/SPEC.md):

PROJECTED cost per 1,000 task attempts — extrapolated from the 147 observed attempts in each configuration, not an observed 1,000-run experiment:

These ranges overlap. Costs use reported usage; unknown usage and excluded infrastructure spend for the new setups are disclosed in their provenance.

What a run costs:

In the former 2026-07-27 reference flight, the three WebMCP methods were ~9% of the bill; historical breakdown: docs/CALIBRATION.md.

Spending less. The levers, cheapest first:

• Fewer sites — --sites lite (3 lightweight sites, no databases).
• Fewer / cheaper methods — WebMCP medians are $0.0011–$0.017/task; computer use and DOM + vision carry most of the cost.
• Fewer attempts — --preset smoke or --n 1 instead of the default 3 (you lose majority voting, so one run decides each task).

Setup. You need Docker, Node 20.11+, and an API key for the model under test:

npm install
npx playwright install chromium                       # browser for the agents (or set WT_CHROME)
python3 -m venv .venv-browseruse \
  && .venv-browseruse/bin/pip install browser-use==0.12.7   # only for the dom-browseruse method

Real site boots use Linux-oriented capsule tooling. Native Linux is the lowest-overhead host; macOS works through Docker Desktop when GNU tar and coreutils are installed and placed first on PATH (brew install gnu-tar coreutils). Docker Desktop's VM and filesystem layer add boot/reset and wall-clock overhead, so report the host and avoid comparing absolute latency with bare-metal Linux. You can also dry-run the pipeline (WT_FAKE_LIFECYCLE=1) or point it at a site you booted yourself (WT_MANUAL_BASEURL=http://localhost:PORT).

Site profiles size the run to the machine at hand:

Commands. --arms picks the methods; the default (scripted) is a free, no-LLM baseline that only exercises the pipeline:

npm run bench -- --preset smoke --sites lite --arms scripted    # free pipeline check, no API key
npm run bench -- --preset smoke --sites lite --arms wm-claude,cu-claude   # cheapest paid run, 1 attempt per task
npm run bench -- --preset lite --sites lite --arms wm-claude,cu-claude,dom-browseruse,a11y-stagehand   # every task on the three lite sites × 3 attempts
npm run bench -- --preset smoke --sites lite --arms wm-claude --seed 7 --budget 2 --label check
npm run bench -- --preset smoke --sites idurar-erp-crm --arms cu-openai --task-ids id-6,id-6,id-8 --n 1

Keys. Runs read the provider key straight from your environment:

export ANTHROPIC_API_KEY=sk-ant-...   # cu-claude, dom-browseruse, a11y-stagehand, wm-claude, wm-stagehand
export OPENAI_API_KEY=sk-...          # cu-openai, wm-gpt, code-openai
export GEMINI_API_KEY=...             # cu-gemini, wm-gemini
export WT_SECRET=...                  # optional capsule secret
export WT_SECRET_KEY=...              # optional key used to protect it

A method whose key is absent is skipped with a notice, never an error. --task-ids selects exact task IDs; repeated IDs intentionally repeat a task, which is useful for targeted diagnostics.

Switching models. --model <method>=<model> overrides the model for one method:

npm run bench -- --preset lite --sites lite --arms wm-claude --model wm-claude=claude-opus-5
npm run bench -- --preset smoke --sites lite --arms cu-openai,wm-gpt,code-openai --model cu-openai=gpt-6-astra --model wm-gpt=gpt-6-astra --model code-openai=gpt-6-astra

Computer use needs a computer-use-capable model. Switching provider takes that provider's own method and key, not just a model name.

docs/       design spec and methodology
sites/      sites under test + subset configuration
capsules/   self-contained boot recipes per site (Docker, pinned commits, WebMCP tools)
fixtures/   per-site seed data and runtime patches
goldens/    the reference WebMCP tool implementations, one patch per site
tasks/      task definitions
harness/    the runner (bin/ has the site-boot CLIs)
scoring/    per-task checks and result formats
arms/       interface implementations
results/    finished runs and reports
tests/      harness test suite (npm test)

WindTunnel's own code — harness, boot recipes, and WebMCP patches — is Apache-2.0. It does not vendor any site's source tree: each site is cloned from its upstream at a pinned commit and patched locally at run time, so the copyleft (AGPL/GPL) sites run locally only, and Hi.Events' required "Powered by Hi.Events" footer is preserved. A few reference patches modify upstream files; the upstream lines those hunks carry remain under the upstream project's license. Upstreams, licenses, and pinned commits: ATTRIBUTION.md (canonical) and each capsules/<site>/capsule.yaml.