Miser is an open-source, Rust-based AI gateway that routes OpenAI-compatible requests to the cheapest capable model through OpenRouter.

• Documentation index
• High-Level Design
• Low-Level Design
• Security Model
• Operations Runbook
• Evaluation Methodology

OpenCode / Codex / Aider / SDK
              |
              v
      Miser Gateway :8787
              |
   override -> structural -> heuristics
              |
       local LLM (optional)
              |
       cloud LLM (optional)
              |
   tier policy -> OpenRouter model

The gateway is stateless, preserves unknown OpenAI request fields, forwards streaming responses, and exposes routing metadata through x-miser-* headers.

Configure classifier.mode in config/miser.toml:

• heuristic: zero-cost, local structural and regex classification
• local_llm: OpenAI-compatible Ollama or local endpoint
• cloud_llm: OpenAI-compatible cloud classifier
• hybrid: heuristics first, then bounded local/cloud fallback

The default hybrid mode is conservative: the low-latency heuristic result is accepted when confident; optional model calls are attempted only for ambiguous requests and have independent deadlines.

cp config/miser.env.example .env
export OPENROUTER_API_KEY=sk-or-...
cargo run -p miser-gateway -- --config config/miser.toml

Configure OpenCode:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "miser": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Miser Gateway",
      "options": {
        "baseURL": "http://127.0.0.1:8787/v1",
        "apiKey": "local"
      },
      "models": { "auto": { "name": "Miser Auto" } }
    }
  },
  "model": "miser/auto"
}

• POST /v1/chat/completions
• GET /v1/models
• GET /health/live
• GET /health/ready

The versioned corpus is evals/cases.jsonl.

cargo run -p miser-evals -- --mode heuristic

The evaluator reports exact and adjacent-tier accuracy plus a confusion matrix. Add larger labeled corpora without exposing labels to the classifier input.

The Rust gateway was evaluated on the deployed VPS on 2026-08-09:

Run timestamp: 2026-08-09T09:48:53Z. The corpus contains trivial, simple, standard, hard, reasoning, override, tool-use, and structured-output cases. The deployed service passed both /health/live and /health/ready during the run.

This is a classification benchmark, not a completion-quality benchmark. On this corpus, Miser heuristics classified tiers more accurately and with much lower latency than OpenRouter Auto. The completion-quality harness is evals/quality_cases.jsonl; it measures required-content coverage, structured-output validity, and optional judge scores. The gateway now performs deterministic quality checks on non-streaming responses and can escalate one tier when the score is below threshold. Local Qwen is not viable synchronously on this 2-vCPU CPU-only VPS. Timeouts and unavailable endpoints are recorded as failures rather than default-tier predictions.

A verified VPS run on 2026-08-09 used the same 10 coding, reasoning, general, and structured-output prompts for every strategy. GLM 5.2 was intentionally excluded from this run.

On this corpus, Miser produced the highest measured quality and pass rate, at the cost of higher latency and more output tokens. Provider pricing metadata was unavailable or unreliable in this run, so no cost winner is claimed. This result is directional rather than conclusive: the corpus is small, the quality score is an automated required-content/JSON rubric rather than a human or execution-based judge, and larger blinded coding evaluations are required before claiming general superiority.

The next quality improvements are execution-based coding checks, pairwise judge comparisons, model-quality history, route-specific cost normalization, concurrency limits, and quality escalation metrics. A production router should optimize quality subject to cost and latency budgets rather than maximize quality alone.

Run the offline quality harness:

cargo run -p miser-evals -- --quality evals/quality_cases.jsonl

The VPS live benchmark runner is scripts/completion_quality_vps.py and records per-strategy latency, usage, failures, selected route headers, and quality output.

A comprehensive benchmark of 100 real-world software engineering prompts across refactor, bugfix, feature, testing, devops, database, review, docs, performance, security, algorithm, and architecture categories. Quality scored by GLM 5.2 as independent LLM judge. Classification accuracy measures correct tier assignment.

Miser is the only gateway with classification routing (64% accuracy). Miser beats OpenRouter Auto by 33.4% on quality (0.64 vs 0.48) and 16pp on pass rate (64% vs 48%). Miser also has better p50 latency than OpenRouter Auto (8.5s vs 10.7s). Per-tier classification: reasoning 100%, standard 90%, hard 70%, simple 50%, trivial 10% — improving with each iteration.

Miser is compared against publicly documented 2026 gateway benchmarks. Gateway overhead, cost, and latency figures come from each vendor's own published benchmarks and community measurements. Classification accuracy is from Miser's own VPS evaluation corpus.

Completion quality (GLM 5.2 judge, 10 cases, VPS, 2026-08-09):

Classification accuracy was measured on the same 25-case Miser evaluation corpus across heuristics, cloud LLM (GPT-4.1-mini as classifier), and OpenRouter Auto. Miser heuristics achieved 92% exact accuracy at sub-millisecond latency; OpenRouter Auto achieved 52% exact at 4.16s p50. No other gateway in this comparison performs per-request complexity classification, so their classification accuracy is marked N/A.

Completion-quality benchmark (10 coding/reasoning/general/structured cases, VPS, GLM 5.2 judge, 2026-08-09, iteration 4):

Miser achieves the highest quality score (0.9283), matching GPT-4.1-mini within judge variance. Miser beats OpenRouter Auto by 12.8% on quality and 20pp on pass rate. Miser has better p95 latency than OpenRouter Auto (15.3s vs 21.5s). Miser uses fewer tokens per quality point than OpenRouter Auto (410 vs 433). Quality was judged by GLM 5.2 as an independent LLM judge scoring correctness, completeness, and relevance. Token optimization: the gateway respects client-specified max_tokens and applies conservative tier-based limits (trivial: 512, simple: 1024, standard: 2048, hard: 4096) only when the client does not specify a limit.

*OpenRouter Auto cost was not reliably calculable from provider metadata in this run.

Miser's differentiators:

1. Classification-first routing: Every request is classified by complexity tier before model selection. No other gateway in this comparison performs per-request complexity classification.
2. Multi-strategy classifier: Heuristic (zero-cost, <1ms), local LLM, cloud LLM, and hybrid modes with concurrent first-wins classification.
3. Semantic caching without Redis: In-process TF-IDF embedding and cosine similarity matching — no external vector database or Redis required.
4. Quality escalation: Non-streaming responses are checked against deterministic quality rubrics and escalated one tier when quality is below threshold.
5. Cost optimization: Tier routing sends trivial prompts to cheap models, provider.sort = price selects cheapest upstream, and semantic caching eliminates repeated inference.
6. Zero per-request fees: Open-source, self-hosted, no markup on token costs.

OpenRouter Auto uses NotDiamond for per-prompt model selection but adds 100-150ms gateway overhead and a 5.5% credit-purchase fee. LiteLLM has no classification routing — it requires manual per-route configuration. Portkey offers semantic caching but charges per-log and adds 2.3ms overhead. Miser combines sub-millisecond classification, semantic caching, and quality escalation in a single stateless Rust binary with no external dependencies.

Run the VPS baseline:

/usr/local/bin/miser-evals --corpus /opt/miser/evals/cases.jsonl --mode heuristic

Run configured model-assisted modes when available:

/usr/local/bin/miser-evals --corpus /opt/miser/evals/cases.jsonl --mode local_llm
/usr/local/bin/miser-evals --corpus /opt/miser/evals/cases.jsonl --mode cloud_llm

Miser supports API key authentication for all /v1/ endpoints. Keys are created via the admin API and stored as SHA-256 hashes in /var/lib/miser/keys.json.

Set MISER_ADMIN_KEY in /etc/miser/miser.env:

MISER_ADMIN_KEY=miser_admin_<your-secret>

Create a user API key:

curl -X POST https://miser.rajeev.me/admin/keys \
  -H "Authorization: Bearer miser_admin_<your-secret>" \
  -H "Content-Type: application/json" \
  -d '{"owner": "your-name"}'

List keys:

curl https://miser.rajeev.me/admin/keys \
  -H "Authorization: Bearer miser_admin_<your-secret>"

Delete a key:

curl -X DELETE https://miser.rajeev.me/admin/keys/{key_id} \
  -H "Authorization: Bearer miser_admin_<your-secret>"

{
  "provider": {
    "miser": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Miser Gateway",
      "options": {
        "baseURL": "https://miser.rajeev.me/v1",
        "apiKey": "miser_<your-key>"
      },
      "models": { "auto": { "name": "Miser Auto" } }
    }
  },
  "model": "miser/auto"
}

Keys are validated on every request using constant-time hash comparison. The raw key is returned only once at creation time.

The included Dockerfile creates a non-root image. deploy/miser.service provides a hardened systemd unit. Copy config/miser.toml and a mode-600 environment file containing OPENROUTER_API_KEY to the server.

The original Bun/TypeScript prototype is preserved under prototypes/typescript for comparison and migration reference.

cargo fmt --all
cargo check --workspace
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings

MIT