llama-router

An LLM router that aggregates multiple Ollama and llama.cpp backends behind unified APIs.

• Unified Ollama API — Ollama-compatible API on port 11434 that routes to the best available backend
• Unified llama.cpp / OpenAI API — OpenAI-compatible API on port 8080 for llama.cpp server backends
• Web Dashboard — Bootstrap-based management UI on port 80 with provider status, model inventory, benchmarks, request log, and cache management
• Auto-Discovery — Queries backends for models and capabilities on registration
• Health Checking — Periodic monitoring of backend availability, including per-address liveness for multi-address providers
• Benchmarking — Measure startup time and tokens/sec per model per provider
• Smart Routing — Routes to the least-busy, most-capable provider based on active requests and benchmark data
• Model Fallbacks — Configure cascading fallback models so requests are transparently rerouted when a model is unavailable
• API Key Routing Policies — Per-key latency/throughput/chaos routing, fallback control, and optional model-to-provider pinning
• OCI Registry Cache — Built-in pull-through cache for Ollama model downloads, serving cached layers at LAN speed

docker compose up -d

Open http://localhost to access the dashboard and add your backends.

After you create at least one account under Users, the dashboard requires sign-in. Passwords are stored hashed; they cannot be read back. If you lose the password, clear the dashboard user table and start over (you will briefly have an open dashboard until you create a new user).

Docker Compose (default database path from docker-compose.yml):

docker compose exec llama-router python -c "import sqlite3; c=sqlite3.connect('/app/data/llama_router.db'); c.execute('DELETE FROM dashboard_users'); c.commit(); c.close()"
docker compose restart llama-router

The first command removes all dashboard accounts. The restart clears the in-process user-count cache so the UI immediately returns to bootstrap mode (no login required). Then open Users and create a new administrator.

If you set LLAMA_ROUTER_DATABASE_PATH to something else, use that path inside the sqlite3.connect(...) call. For a local SQLite file (no Docker), run the same DELETE FROM dashboard_users against your database file with any SQLite client.

To remove a single user by name instead of wiping everyone:

docker compose exec llama-router python -c "import sqlite3; c=sqlite3.connect('/app/data/llama_router.db'); c.execute('DELETE FROM dashboard_users WHERE username = ?', ('myuser',)); c.commit(); c.close()"
docker compose restart llama-router

Prerequisites: wg-quick and wg wherever llama-router applies config (apt install wireguard-tools on Debian/Ubuntu). The default Dockerfile does not install WireGuard tools; add them in a derived image if you want wg-quick inside the container, or use legacy volume mode and run WireGuard on the host.

Docker networking: You do not need network_mode: host if everything that must use the VPN (llama-router APIs, backends you configure with tunnel URLs, etc.) runs inside the same container and remote peers only need to reach that container. Use cap_add: [NET_ADMIN], map 51820:51820/udp (or your listen port), and set the dashboard Public endpoint to whatever address:port peers use to reach the host (Docker forwards UDP into the container). Use network_mode: host when you want the WireGuard interface and service ports on the host network namespace (typical when the tunnel must behave like a bare-metal router on the LAN). Docker Desktop for macOS/Windows is awkward for in-container kernel WG; use a Linux host, legacy volume mode with WG outside the container, or run llama-router on the host.

Defaults: LLAMA_ROUTER_WIREGUARD_CONFIG_PATH defaults to /etc/wireguard/wg0.conf. The dashboard renders peers from the database and applies the config with wg syncconf when possible (falling back to wg-quick up).

Connecting two routers: On each host, open the dashboard WireGuard tab. On the hub, enable Inbound peering, save to obtain a peering API key, and share that key with the spoke operator. On the spoke, use Connect to remote llama-router with the hub URL, key, and chosen tunnel IPs (or use the peering HTTP API documented below). The flow adds reciprocal WireGuard peers and optional providers in one step.

Peering API key: Remote routers must send X-Peering-Key: <secret> on GET /api/wireguard/peer-info and POST /api/wireguard/peer-request. Treat it like a password.

Isolation (peers vs your LAN): WireGuard only carries traffic you route into the tunnel. To not give the remote side access to your site network, keep AllowedIPs minimal: typically the peer’s tunnel address only (e.g. 10.8.0.2/32 on your side for that peer). Do not add office/home LAN subnets (e.g. 192.168.0.0/24) to AllowedIPs unless you intend to expose them. The built-in connect/peering flow uses /32 tunnel addresses. On the remote peer, they should likewise aim AllowedIPs at your llama-router tunnel IP (and ports), not 0.0.0.0/0, unless you explicitly want full tunnel routing. Avoid PostUp/iptables in wg0.conf that forward or NAT arbitrary traffic from wg0 into other interfaces unless you have reviewed it. With Docker bridge networking and no host routing tricks, peers still only reach what the tunnel and your published ports allow—but narrow AllowedIPs remains your main control.

Legacy sidecar: Set LLAMA_ROUTER_WIREGUARD_LEGACY_VOLUME=true to only write wg0.conf and let an external process (or old sidecar layout) apply it.

See docs/wireguard.md for a short pointer on cache access over the tunnel (LLAMA_ROUTER_CACHE_EXTERNAL_HOST).

All settings are configured via environment variables with the prefix LLAMA_ROUTER_.

Use the bundled CLI after creating an empty PostgreSQL database and granting a user DDL rights (Alembic will create tables).

1. Install the Postgres extra: pip install 'llama-router[postgres]' (provides psycopg for this tool and optional runtime use).
2. Run:

llama-router-migrate-sqlite-pg \
  --sqlite /path/to/llama_router.db \
  --postgres postgresql+psycopg://USER:PASSWORD@HOST:5432/DATABASE

The tool applies Alembic on the target, truncates all application tables (not alembic_version), copies rows in foreign-key order so IDs stay stable, then resets sequences. Your SQLite file should already match the current schema (run the app against SQLite once so migrations are applied there, or use a backup from a recent version).

You can pass postgresql+asyncpg://… and it will be rewritten to postgresql+psycopg://… for the sync copy step.

Set a DSN to enable Sentry reporting for unhandled exceptions and performance telemetry.

The cache acts as a pull-through proxy for the Ollama model registry (registry.ollama.ai). When enabled, model pulls are routed through the cache so that large blob layers are stored locally on first download. Subsequent pulls of the same model — on any provider — are served from the local cache at LAN speed instead of re-downloading from the internet.

1. You pull a model through the dashboard or the Ollama API (e.g. llama3:8b).
2. llama-router rewrites the pull request so Ollama fetches layers from http://<cache-external-host>:9200/... instead of registry.ollama.ai.
3. The cache checks its local blob store (content-addressed by SHA256 digest):
   • Cache hit — the blob is streamed directly from disk at LAN speed.
   • Cache miss — the blob is fetched from the upstream registry, streamed to the Ollama client, and simultaneously saved to disk for future requests.
4. Manifests are cached with a configurable TTL (default 10 days). Blobs never expire because they are immutable content-addressed data.

• The cache runs as an HTTP server on port 9200 (configurable). This port does not need to be exposed externally — it only needs to be reachable from the Ollama backends on your LAN.
• In Docker, the cache directory should be on a persistent volume with enough space for the models you pull (models range from ~2 GB to ~200+ GB).

Set LLAMA_ROUTER_CACHE_ENABLED=false to disable the cache entirely. Pulls will go directly to the upstream registry as usual.

Use the Cache tab on the dashboard, or call the API directly:

curl -X POST http://localhost/api/cache/clear

All three API ports expose a GET /health endpoint that returns JSON:

{"status": "ok", "version": "0.7.8", "providers": 3, "providers_online": 2}

The dashboard port (80) includes provider counts; the Ollama (11434) and llama.cpp (8080) ports return a minimal response. The Docker Compose file includes a health check configuration using this endpoint.

You can configure fallback models so that when a requested model is unavailable (no provider has it or all providers with it are offline), the router transparently tries an alternative model.

Fallbacks cascade: if model A falls back to model B, and model B falls back to model C, a request for model A will try A → B → C in order.

Configure fallbacks through:

• The Models tab on the dashboard (click the arrow icon next to any model, or use the Fallbacks section)
• The REST API:

# Set a fallback
curl -X POST http://localhost/api/fallbacks \
  -H 'Content-Type: application/json' \
  -d '{"model": "llama3:70b", "fallback": "llama3:8b"}'

# List all fallbacks
curl http://localhost/api/fallbacks

# Remove a fallback
curl -X DELETE http://localhost/api/fallbacks/llama3:70b

API keys can optionally pin specific models to specific providers. When a pinned model is requested with that key, llama-router always chooses the configured provider for that model (if available), bypassing latency/throughput scoring.

You can configure pins from the API Keys tab in the dashboard, or with the REST API:

# Pin a model to a provider for API key 1
curl -X POST http://localhost/api/keys/1/pins \
  -H 'Content-Type: application/json' \
  -d '{"model_name":"llama3.2:latest","provider_id":2}'

# List pins for API key 1
curl http://localhost/api/keys/1/pins

# Remove a pin
curl -X DELETE http://localhost/api/keys/1/pins/llama3.2:latest

pip install -e '.[dev]'
pytest -v
black src/ tests/

Or with Docker:

docker compose -f docker-compose.dev.yml run test