Terminal dashboard for Shepherd Model Gateway — monitor workers, route traffic, and chat with models, all from your terminal.

cargo build -p smg-tui --release
# Binary at: target/release/smg-tui

# Auto-start gateway and connect
 ./target/release/smg-tui --auto-start

# Connect to an existing gateway
 ./target/release/smg-tui --gateway-url http://localhost:30000

# With API key for OpenAI/Anthropic models
OPENAI_API_KEY=sk-...  ./target/release/smg-tui --auto-start

When --auto-start is passed and the gateway is not reachable:

1. Launches smg launch with --enable-igw --policy round_robin
2. Falls back to cargo run -p smg if the binary isn't in PATH
3. Polls the health endpoint until the gateway is ready (up to 120s)
4. On q: TUI exits but gateway keeps running. On Ctrl+C ×2: full shutdown kills the gateway

Switch views with number keys 1-7.

The top header displays four metric cards updated every poll cycle:

    WORKERS          CIRCUIT BREAKERS          REQ/S            AVG LATENCY
       5                    5                  250.3              450ms
   all healthy          all closed          51 in-flight        ▓▓▓░░░░░

Two-column real-time dashboard:

Left column:

• Worker Health — workers grouped by model (local) or provider (external), each with health dot, hostname, runtime, connection mode
• GPUs — GPU utilization, memory bar, temperature via nvidia-smi (auto-hides when unavailable)

Right column:

• Throughput — req/s sparkline over 60s window with tok/s when available
• Request Stats — avg latency (with sparkline), active connections, in-flight requests

Press Enter to toggle. Shows:

Config: URL, runtime, mode, health, models served

Stats (HTTP sglang/vllm):

• Running requests with gauge bar (current / max)
• Waiting requests
• KV cache usage with gauge bar (tokens used / total)
• Gen throughput (when available)

Stats (gRPC / External):

• Req/s from Prometheus
• Circuit breaker state (open/closed)

Interactive chat with any model through the SMG gateway.

Features:

• Streaming responses with live cursor
• Markdown rendering — bold, italic, code, headings, bullets, fenced code blocks
• Multi-turn conversation:
  • Chat completions (/v1/chat/completions): sends full conversation history each turn
  • Responses API (/v1/responses): uses previous_response_id for efficient multi-turn
• Title bar shows current model, endpoint, and multi-turn mode
• Auto-selects first available model on send
• Model filtering: local models shown as-is, OpenAI filtered to gpt-5.4* to keep the list manageable

Sub-tab system for viewing different log sources:

 a:TUI  b:SMG  w:Llama-37595  w:Qwen2-39223          j/k scroll  G bottom  w worker

• TUI logs: Application events (worker add/delete, status changes)
• SMG logs: Gateway logs from /tmp/smg-gateway.log (ANSI stripped)
• Worker logs: Per-worker logs from /tmp/smg-worker-{port}.log, labeled as model(5chars)-port

Press a → 1. External → select provider → enter API key (or press Enter to use env var):

The API key is stored on the worker for model discovery (required in IGW mode).

Press a → 2. Local → select runtime → connection mode → model preset:

GPU management:

• Auto-selects free GPUs (>2GB free via nvidia-smi)
• Sets CUDA_VISIBLE_DEVICES to avoid conflicts
• Tracks claimed GPUs to prevent double-allocation before memory shows in nvidia-smi
• Releases GPU claims on worker deletion
• Kills backend process on worker deletion

Press a → 3. Custom URL → enter worker URL directly.

Press : for command mode:

:add <url> [--provider <p>] [--runtime <r>]
:delete <id>
:priority <number>
:cost <number>
:flush-cache
:toggle-health
:quit

The TUI polls the SMG gateway every --poll-interval seconds via:

Chat uses POST /v1/chat/completions and POST /v1/responses with SSE streaming.