The bench/ tree contains runnable evaluations for routing quality, reasoning mode, session continuity, hallucination detection, Router Flow, and signal runtime performance. It is a contributor workspace rather than a collection of published benchmark results. Generated artifacts belong in an output directory and are ignored by Git.

For an interpretation of maintained benchmark coverage, see the benchmarking overview.

Start with the smallest dataset or --dry-run mode supported by the selected runner. Run python <script> --help or the installed command's --help before launching a full evaluation; the CLI is the source of truth for flags and defaults.

From the repository root:

python3 -m venv .venv-bench
. .venv-bench/bin/activate
python -m pip install -e 'bench[dev]'

Router Flow's EvalScope runner has additional dependencies:

python -m pip install -e 'bench[real_eval]'

Most live evaluations require one or more OpenAI-compatible endpoints. Keep API keys in environment variables; do not put credentials in commands, result files, or committed configuration.

List the datasets registered by the installed package, then run a small comparison:

vllm-semantic-router-bench list-datasets
vllm-semantic-router-bench test \
  --dataset mmlu \
  --samples 5 \
  --mode both

test is a quick sample. compare accepts explicit router and direct-vLLM endpoints, and comprehensive runs multiple registered datasets. Use vllm-semantic-router-bench <command> --help for their endpoint and output options.

reasoning-mode-eval sends the same dataset samples with reasoning disabled and enabled. It reports accuracy, token use, response time, and time per output token.

reasoning-mode-eval \
  --datasets mmlu gpqa \
  --model qwen3-14b \
  --reasoning-family qwen3 \
  --endpoint http://127.0.0.1:8000/v1 \
  --samples-per-category 10 \
  --output-dir results/reasoning-mode

The run writes vsr_canonical_patch.yaml and vsr_canonical_patch_recommendation.json alongside its detailed results. The patch is a review aid, not an automatically safe deployment change. Confirm the model name, reasoning family, measured trade-offs, and existing config entries before merging it.

A known-family patch uses the current v0.3 provider and Model Card fields:

providers:
  defaults:
    reasoning_effort: medium
  models:
    - name: qwen3-14b
      reasoning:
        family: qwen3

Reasoning is enabled per decision reference, after the evaluated model has been merged into the existing configuration:

routing:
  decisions:
    - name: math-reasoning
      rules:
        operator: AND
        conditions:
          - type: domain
            name: math
      modelRefs:
        - model: qwen3-14b
          use_reasoning: true
          reasoning_effort: high

Supported generated family mappings are qwen3, deepseek, and gpt-oss. When --reasoning-family is omitted or unknown, the recommendation records a manual follow-up instead of inventing a provider mapping.

The session tools cover two distinct boundaries:

• agentic_routing_experiment.py is deterministic and does not call a live model. Use it for scenario matrices, policy ablations, replay input, and the Router Learning architecture gate.
• agentic_routing_live_benchmark.py calls a router and can compare it with a direct backend. It measures success, latency, selected diagnostic headers, continuity, context portability, and recovery behavior.

Run the deterministic gate with the maintained PR thresholds:

make bench-router-learning

Use PROFILE=release only when performing a release evaluation. Profile semantics and output files are documented in profiles/router_learning/.

Before sending traffic, inspect a live scenario without network calls:

python bench/agentic_routing_live_benchmark.py \
  --scenario balanced \
  --sessions 2 \
  --turns 3 \
  --dry-run \
  --output-dir results/session-routing-dry-run

For a live comparison, provide both endpoints and set explicit acceptance thresholds. --require-router-diagnostics checks the maintained x-vsr-* response-header contract.

The related tools are intentionally separate:

The branch-image and GA tools validate evidence; they do not build, deploy, or approve an image. Use immutable image tags or digests and record the reviewed source ref whenever those artifacts are used for a release decision.

Use a new output directory for each run. At minimum, record:

• source revision and immutable image identity;
• endpoint implementation and served model name;
• dataset revision, sample count, seed, and filters;
• generation settings and concurrency;
• hardware and relevant runtime versions;
• every threshold used to produce a pass/fail verdict.

Do not present a smoke limit, synthetic fixture, or dry run as a full benchmark. Do not commit raw prompts, responses, headers, or environment captures until they have been checked for credentials and user data.

Run the lightweight benchmark tests without starting model servers:

python -m pytest \
  bench/test_*.py \
  bench/grounded_fusion/test_*.py \
  bench/router_flow/real_eval/test_*.py

When changing a CLI, update its --help, tests, and this index only if the reader's choice of runner or first command changes. Detailed experiment design belongs with the runner that implements it.

The fault proxy forwards SSE events as they arrive from upstream. Optional --stream-frames shaping forwards the first requested content events live and repeats them if needed before forwarding the terminal event. Events after the terminal event, including usage and [DONE], are preserved. --stream-interval-ms adds pacing between forwarded events; it does not remove upstream delays.

Run make soak-test and python -m pytest bench/test_openai_fault_proxy.py -q for buffered and streaming behavior coverage. The PR core-quality workflow runs both commands in Soak behavior (PR head) on the exact PR head commit and records that SHA in its job summary.