RouterLab

A runnable reference implementation of a data-driven LLM model router inspired by the architecture Cursor publicly described for Cursor Router on August 6, 2026.

This is not Cursor source code and does not claim to reproduce proprietary details. It implements the public behavioral design:

1. infer whether a price-efficient model is likely to satisfy the turn (Compass),
2. classify harder work by domain / task / modifiers,
3. learn model-specific strengths from real outcomes,
4. require evidence of uplift over the cheap model,
5. optimize the traffic-weighted model mix under a cost budget,
6. retain explicit exploration probabilities so new policies can be evaluated from logged bandit feedback.

The repository includes a synthetic oracle environment so the entire learning loop can be tested before connecting production model APIs.

Incoming turn
    |
    +--> Compass: P(cheap model succeeds | context)
    |       |
    |       +--> high enough --> cheap model
    |       |
    |       +--> low enough ---> taxonomy classifier
    |                           domain / task / modifiers
    |                                  |
    |                                  v
    |                         Bayesian strength table
    |                         P(model uplift > 0)
    |                                  |
    |                         one-sided 75% gate
    |                                  |
    |                                  v
    |                         cost-constrained LP policy
    |                                  |
    +----------------------------------+--> chosen model
                                               |
                                               v
                                      exact route propensity
                                               |
                                               v
                                         outcome telemetry
                                               |
                    +--------------------------+---------------------+
                    |                                                |
             retrain models                              IPS / SNIPS / DR OPE

• Compass — propensity-corrected TF-IDF + logistic model estimating cheap-model success probability.
• Taxonomy — domain, task and multi-label modifier classifiers.
• Model strengths — propensity-aware beta posteriors with hierarchical pooling over global/domain/task/modifier labels.
• Uplift gate — candidate frontier model must have P(uplift > 0) >= 0.75.
• Cost + latency predictors — learned regressors using turn context and model identity.
• Budget optimizer — scipy.optimize.linprog solves a traffic-weighted quality maximization under average frontier-turn cost.
• Exploration — 2% epsilon exploration by default; every selected model has a known, non-zero propensity.
• Offline policy evaluation — IPS, self-normalized IPS, and doubly robust reward estimation.
• Telemetry service — FastAPI /route and /telemetry endpoints, local SQLite sink, plus a production-oriented PostgreSQL schema.
• Synthetic oracle — known P(success | x, model) and expected cost for every counterfactual model, used only to verify that learning/evaluation recover ground truth.

Python 3.11+ is recommended.

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\\Scripts\\activate
pip install -e ".[dev]"

make validate ROWS=30000

make validate runs:

1. generate synthetic logged-bandit traffic
2. split train / held-out test data
3. train Compass + taxonomy + strength + cost/latency + reward models
4. solve Balance and Intelligence routing policies
5. evaluate both policies on held-out traffic
6. run pytest
7. print example route decisions

You can run the stages separately:

make generate ROWS=30000
make train
make evaluate
make test
make demo

The checked-in artifacts were trained on 14,000 synthetic turns and evaluated on 6,000 held-out turns from a 20,000-turn run.

Because the simulator retains the unobserved counterfactual probabilities, we know the true policy outcome rather than merely trusting the estimator.

The important result is Intelligence: in this known synthetic environment it slightly exceeds always-Opus quality while costing less than always-Opus, and comes within ~0.35 percentage points of always-Fable while costing about 28% less.

Raw IPS is deliberately shown in artifacts/evaluation.json; with limited overlap it is visibly noisier. The doubly robust estimator is substantially closer to the oracle in this run.

The synthetic taxonomy is intentionally easy to classify, so its 100% held-out accuracy is not a claim about real developer traffic. Real taxonomy performance will depend on your labeling scheme and traffic diversity.

After training:

make serve

or:

PYTHONPATH=src uvicorn routerlab.api:app --host 0.0.0.0 --port 8000

curl http://localhost:8000/health

curl -X POST http://localhost:8000/route \
  -H 'content-type: application/json' \
  -d '{
    "mode": "intelligence",
    "context": {
      "request_id": "9ae51039-2f71-4dfd-a888-c72c760c9bbb",
      "request_text": "Debug the failing frontend behavior. A previous attempt failed. The change spans several files and the visual interaction must match precisely.",
      "context_tokens": 42000,
      "conversation_turns": 18,
      "files_in_context": 19,
      "tool_calls_recent": 7,
      "previous_failed": true,
      "user_corrected_recent": true,
      "repo_size_bucket": 5,
      "requires_visual": true,
      "requires_terminal": true,
      "previous_model": "sol"
    }
  }'

The response includes:

{
  "model": "opus",
  "selection_probability": 0.985,
  "model_probabilities": {
    "cheap": 0.005,
    "sol": 0.005,
    "opus": 0.985,
    "fable": 0.005
  },
  "compass_score": 0.68,
  "escalated": true,
  "taxonomy": {
    "domain": "frontend",
    "task": "debugging",
    "modifiers": ["multi_file", "visual_heavy"]
  }
}

Exact numeric values depend on the trained artifacts and request ID.

After the selected provider finishes, send the exact selection propensity returned by /route:

curl -X POST http://localhost:8000/telemetry \
  -H 'content-type: application/json' \
  -d '{
    "request_id": "9ae51039-2f71-4dfd-a888-c72c760c9bbb",
    "chosen_model": "opus",
    "selection_propensity": 0.985,
    "reward": 1,
    "cost_usd": 0.19,
    "latency_ms": 2810,
    "input_tokens": 41000,
    "output_tokens": 1900,
    "cache_hit": false,
    "user_corrected": false,
    "code_survival_24h": 0.97,
    "metadata": {}
  }'

Local telemetry lands in data/router_telemetry.sqlite3. infra/postgres_schema.sql is the recommended starting point for production telemetry.

RouterLab deliberately treats model names as logical aliases:

cheap
sol
opus
fable

See config/model_aliases.yaml.

In your agent harness, the normal flow is:

route = router.route(turn_context, mode="intelligence")
response = providers[route.model].run(turn_context)

telemetry.write(
    chosen_model=route.model,
    selection_propensity=route.selection_probability,
    reward=infer_reward(response, next_user_action),
    cost_usd=response.actual_cost,
    ...
)

Do not hard-code these aliases to a specific vendor in the learning code. Model IDs change; the router should learn against stable logical/versioned actions such as opus-5-2026-08 or gpt-x-2026-09 and retire them explicitly when the provider changes the model behind an alias.

The synthetic CSV intentionally mirrors the production training contract. Minimum useful columns are:

request_text
context_tokens
conversation_turns
files_in_context
tool_calls_recent
previous_failed
user_corrected_recent
repo_size_bucket
requires_visual
requires_terminal
previous_model

domain
task
modifiers

chosen_model
propensity
reward
cost_usd
latency_ms

The following synthetic-only columns must not exist in real training data:

difficulty_oracle
oracle_p_*
oracle_cost_*

For production I would avoid a single brittle heuristic. Construct a reward classifier from signals such as:

Positive

• user moves to a distinct next task,
• generated edit remains in the working tree,
• tests pass,
• commit follows,
• generated lines survive 24h / 7d.

Negative

• user corrects the agent,
• immediate retry,
• undo/revert,
• same failure is described again,
• user manually replaces the generated code,
• user switches model immediately after the turn.

A useful eventual target is:

P(turn satisfied | response + next-user behavior + code survival)

rather than “thumbs up.”

Once routing depends on context, the observed data are selected:

frontend debugging -> Fable often
planning           -> Sol often
Git                -> cheap often

Naively comparing observed success rates confuses model strength with the old router's selection policy.

RouterLab therefore logs:

chosen_model
propensity = P(old policy chose chosen_model | context)

and uses controlled exploration to maintain support. The strength estimator uses stabilized inverse-propensity weighting; offline evaluation exposes IPS, SNIPS and a doubly robust estimate.

Never deploy a contextual router without storing the action probability that produced each observation. You cannot reconstruct it reliably after the fact.

For taxonomy bucket b and model m, RouterLab estimates:

Q[b,m] = expected success
K[b,m] = expected cost
w[b]   = traffic share

and solves:

maximize    sum_b w[b] * sum_m p[b,m] * Q[b,m]

subject to  sum_b w[b] * sum_m p[b,m] * K[b,m] <= budget
            sum_m p[b,m] = 1                for every bucket
            p[b,m] >= 0

Only models that pass the uplift-confidence rule are eligible, except the cheap fallback.

The resulting fractional solution is a traffic mix. Runtime sampling realizes that mix and the returned probability becomes the logged routing propensity.

Cursor publicly describes Compass as a complexity score learned from satisfaction. This implementation uses the operationally cleaner quantity:

Compass(x) = P(cheap model succeeds | x)

so routing is:

if Compass(x) >= threshold:
    cheap
else:
    taxonomy/budget router

The cheap-model training examples themselves were selected by the previous behavior policy, so RouterLab corrects Compass training with stabilized inverse-propensity weights.

Balance currently uses a 0.70 threshold; Intelligence uses 0.76. These are demonstration defaults, not universal constants. Tune them on your own held-out/off-policy evaluation data.

src/routerlab/
  api.py             FastAPI service
  bayes.py           propensity-aware beta posteriors + uplift probability
  compass.py         cheap-success predictor
  config.py          model/mode defaults
  features.py        serving/training feature contract
  ope.py             IPS / SNIPS / doubly robust evaluation
  policy.py          traffic-weighted LP budget optimizer
  predictors.py      cost + latency models
  reward_model.py    counterfactual reward model used by DR
  router.py          runtime router
  schemas.py         Pydantic API contracts
  simulator.py       synthetic contextual-bandit oracle
  storage.py         local outcome store
  taxonomy.py        domain/task/modifier classifiers
  train.py           training orchestration

scripts/
  generate_synthetic.py
  train_router.py
  evaluate_router.py
  demo_routes.py

infra/
  postgres_schema.sql
  docker-compose.yml

artifacts/
  compass.joblib
  taxonomy.joblib
  strengths.json
  cost_model.joblib
  latency_model.joblib
  reward_model.joblib
  policy.json
  evaluation.json

1. Version every model action. Never pool outcomes across materially changed model snapshots.
2. Cross-fit reward/uplift models. The reference implementation uses a clean train/test split; production should use K-fold cross-fitting for OPE and policy search.
3. Tune mode thresholds and budgets jointly. Sweep Compass thresholds and LP budgets on validation traffic, then choose Pareto candidates before online A/B tests.
4. Add confidence intervals to OPE. Bootstrap DR/SNIPS by conversation or user, not individual turns, to respect clustering.
5. Use hierarchical or direct contextual quality models. Replace label tables with Q(x, model) once data volume supports it; retain taxonomy for interpretability and cold start.
6. Model cache state explicitly. previous_model is only a proxy. Log cached tokens, provider cache identifiers, context reuse and switch penalties directly.
7. Separate immediate and durable reward. A patch that survives seven days should matter differently from “user did not complain on the next turn.”
8. Run exploration deliberately. Start small, bound regret, and never silently set propensities to 1.0 for decisions that were actually randomized.
9. Detect distribution shift. A new model, new agent harness, or new tool API changes the data-generating process; stratify/version those changes.
10. Graduate from taxonomy routing to contextual bandits. The natural end state is direct prediction of quality/cost/latency per model with controlled online exploration.

The implementation follows the architecture described publicly in Cursor's How Cursor Router chooses the right model for the task (Aug. 6, 2026). The off-policy evaluator follows the standard contextual-bandit setup in the doubly robust policy-evaluation literature (Dudík, Langford & Li and subsequent work).