inebrio/routerly repository overview

⁠Routerly

One gateway. Any AI model. Total control.

Self-hosted LLM gateway with intelligent routing, real-time cost tracking, and budget enforcement. Drop-in compatible with the OpenAI and Anthropic APIs — change one environment variable, nothing else in your code changes.

Your App  ──────────────────────▶  Routerly  ──▶  OpenAI / Anthropic
                                      │       ──▶  AWS Bedrock
  POST /v1/chat/completions           │       ──▶  Azure OpenAI
  (same API, any SDK)                 │       ──▶  Google Vertex AI
                                      │       ──▶  Gemini / Mistral / Groq / Cohere
                                      │       ──▶  Ollama (local)
                              Auth · Route · Budget · Track

⁠Quick start

docker run -d   --name routerly   -p 3000:3000   -v routerly_data:/data   -e ROUTERLY_HOME=/data   --restart unless-stopped   inebrio/routerly:latest

Open http://localhost:3000⁠ to access the dashboard.

⁠docker-compose

services:
  routerly:
    image: inebrio/routerly:latest
    ports:
      - "3000:3000"
    volumes:
      - routerly_data:/data
    environment:
      - ROUTERLY_HOME=/data
    restart: unless-stopped

volumes:
  routerly_data:

docker compose up -d

⁠What Routerly does

• Intelligent routing — every request is scored across up to 9 pluggable policies (cheapest, capability, health, performance, context, budget-remaining, fairness, …) and sent to the best available model. Automatic fallback if a provider fails.
• Content guardrails and PII scrubbing — block or log requests and responses based on keyword, regex, or semantic judge scoring. Strip PII before it reaches the provider. All evaluations appear in the request trace.
• Notification system — Slack, Teams, PagerDutt, Discord, email, and in-app inbox. Per-channel routing rules, cooldowns, and budget alerts.
• Real-time cost tracking — token-level pricing per model, per project. Set hard limits (hourly / daily / monthly / per-request) that block overspending before it happens. Hierarchical budget cascade from org to team to API key.
• 2FA / TOTP — per-user authenticator enrollment with QR code. Enforce 2FA for all users from Settings.
• Prompt playground — test prompts and compare two models side-by-side from the dashboard, with per-panel parameters and debug traces.
• Dynamic model catalog — models are discovered at runtime from configurable git repositories. Model Discovery page in the dashboard with provider and capability filters.
• Project isolation — separate Bearer tokens per project, each with its own model pool, routing rules, and budget envelope.
• Zero database — config and usage data live in plain JSON files. No PostgreSQL, no Redis, no migrations.
• Web dashboard — live view of spend, call volume, error rates, provider health, and per-model breakdown with a sortable usage leaderboard.
• Admin CLI — manage models, projects, users, notifications, and pull usage reports from any terminal.
• Prometheus metrics — /metrics endpoint with request counts, latency histograms, token usage, and cache hit rates.

⁠Works with any OpenAI or Anthropic client

Cursor, Continue.dev, Open WebUI, LibreChat, LangChain, LlamaIndex, or plain curl — anything that speaks the OpenAI or Anthropic protocol works without modification.

# OpenAI SDK
client = OpenAI(
    base_url="http://localhost:3000/v1",
    api_key="your-project-token"
)

⁠Configuration

All data and configuration are stored in /data inside the container. Mount a volume there to persist settings across restarts.

⁠Supported platforms

• linux/amd64
• linux/arm64 — Apple Silicon, AWS Graviton, Raspberry Pi

⁠Tags

⁠Links

• Documentation⁠
• GitHub⁠
• Issues and Feedback⁠