grokbot-BYOK

Route Grok Bot's AI inference through your own LLM API keys.

Grok Bot BYOK is a local proxy that intercepts Grok Bot's inference calls (chat, tab-completion, agent mode) and routes them to any OpenAI-compatible LLM provider you choose: OpenRouter, OpenCode, Kilo, a local model server, or all of the above with automatic failover. No more paying for the hosted backend.

You bring the keys (BYOK = Bring Your Own Key). Grok Bot BYOK handles the rest: protocol translation, routing, failover, and automatic re-patching when Grok Bot updates itself.

• What it does
• Quick start
• How it works
• Providers
• Configuration
• Managing daemons
• Uninstall
• Routing strategies
• Failover and circuit breaking
• Latency-based routing
• Session affinity
• Vision fallback
• Tool call handling
• Observability
• Guardrails
• Deploy
• Troubleshooting
• Tests
• Build
• Project structure
• License

Grok Bot talks to Grok Bot BYOK (this project) using Connect-RPC, a gRPC-over-HTTP streaming protocol with binary-framed envelopes. The backends themselves are OpenAI-compatible chat completion endpoints, but the wire format is different. Grok Bot sends Connect. Providers expect OpenAI SSE.

Grok Bot BYOK sits between them:

graph LR
    GrokBot["Grok Bot (Connect-RPC, binary)"] -->|"POST /Stream"| BYOK
    BYOK["Grok Bot BYOK (localhost:8788)"] -->|"OpenAI SSE (JSON)"| Provider1["OpenRouter"]
    BYOK -->|"OpenAI SSE (JSON)"| Provider2["OpenCode Go"]
    BYOK -->|"OpenAI SSE (JSON)"| Provider3["Kilo"]
    BYOK -->|"OpenAI SSE (JSON)"| Provider4["Local"]
    Provider1 -->|"your keys"| Keys1["sk-or-xxx"]
    Provider2 -->|"your keys"| Keys2["sk-xxx"]
    Provider3 -->|"keyless"| Keys3["200 req/hr"]
    Provider4 -->|"your keys"| Keys4["sk-local"]

Grok Bot BYOK:

1. Receives Connect-RPC streaming requests from Grok Bot
2. Translates them to OpenAI chat completion format
3. Routes to the best available provider (priority, latency, or round-robin)
4. Fails over automatically if a provider errors, rate-limits, or stalls
5. Translates the OpenAI SSE response back to Connect frames
6. Streams the response back to Grok Bot

Grok Bot doesn't know the proxy is there. It thinks it's talking to its normal backend. Grok Bot BYOK patches the host bundle (host-main.cjs) to redirect inference calls to localhost:8788, and a background watcher re-applies the patch automatically whenever Grok Bot updates itself.

• Node.js 18+
• A Grok Bot account
• At least one LLM API key, or none at all. Kilo is keyless.

# 1. Clone
git clone https://github.com/jroth1111/grokbot-BYOK.git
cd grokbot-BYOK

# 2. One-command setup (installs deps, builds, patches Grok Bot, starts daemons)
npm run setup -- --openrouter-key sk-or-xxx --opencode-key sk-xxx

# Or with zero API keys (Kilo is keyless, works out of the box):
npm run setup

Grok Bot's inference now routes through your providers.

npm status                 # Check daemon status (all three should be "running")
tail -20 /tmp/inference-shim.log  # Check the Grok Bot BYOK log for errors
node dist/scripts/live-test.js    # Send a test request through Grok Bot BYOK

If npm status shows all daemons running and the live test returns text, Grok Bot BYOK is working. If something failed, check Troubleshooting.

Keys go in .env in the project root. You can pass them at setup time or edit .env manually. See Providers for which keys you need.

# Via CLI flags (written to .env automatically)
npm run setup -- --opencode-key sk-xxx --openrouter-key sk-or-xxx

# Via env vars (written to .env automatically)
OPENCODE_API_KEY=sk-xxx OPENROUTER_API_KEY=sk-or-xxx npm run setup

# Or just edit .env directly

The setup script is fully non-interactive and does everything in one step:

1. Checks prerequisites (Node version, dependencies installed)
2. Copies config/config.example.json to config/config.json (if not present)
3. Copies .env.example to .env (if not present)
4. Writes API keys from CLI flags / env vars into .env
5. Builds Grok Bot BYOK + scripts (esbuild to standalone bundles)
6. Starts three background daemons (see Managing daemons)
7. Patches Grok Bot's host-main.cjs to redirect inference to Grok Bot BYOK
8. Runs a smoke test
9. Exits 0 on success, 1 on failure

Re-running npm run setup is safe. It stops existing daemons first, then restarts. It acts as a restart command too.

Every inference request from Grok Bot follows this path through Grok Bot BYOK:

sequenceDiagram
    participant C as Grok Bot
    participant S as Grok Bot BYOK
    participant P as Provider

    C->>S: POST /Stream (Connect-RPC, binary)
    S->>S: Parse Connect envelopes
    S->>S: Translate to OpenAI ChatRequest
    S->>S: Route to provider (priority/latency/round-robin)
    S->>S: Vision check + budget guardrail
    S->>P: POST /v1/chat/completions (OpenAI SSE)
    alt Provider accepts
        P-->>S: SSE chunks (text, tool calls, usage)
        S-->>C: Connect frames (streamed)
    else Provider rejects or stalls
        S->>S: Failover to next provider
        S->>P: Retry with next provider
    end
    S-->>C: End stream

 1. HTTP receive    Grok Bot POSTs a Connect-RPC streaming request to
                    /aiserver.v1.InferenceService/Stream
 2. Parse           Connect envelopes are decoded; the data frame is
                    extracted as proto3 JSON (InferenceStreamRequest)
 3. Translate       The Connect request is converted to an OpenAI
                    ChatCompletion request (messages, tools, images,
                    model config)
 4. Route           The provider registry selects a provider based on
                    the routing strategy and model id
 5. Vision check    If the request has images but the model doesn't
                    support vision, re-route to VISION_FALLBACK_MODEL
 6. Budget check    Pre-flight token budget guardrail (optional)
 7. Connect         Fetch the upstream provider (with retry/backoff,
                    key rotation, circuit breaker checks)
 8. Failover        If the provider rejects or stalls, try the next
                    provider in the failover chain
 9. Stream          Read the OpenAI SSE response, translate each chunk
                    to Connect frames (text, tool calls, usage, thinking)
10. Retry           If the stream produces no content (empty completion),
                    failover to the next provider
11. Respond         Write Connect frames back to Grok Bot

Grok Bot BYOK writes nothing to the outbound response until a provider accepts the request, so failover between providers is transparent to Grok Bot.

Grok Bot's inference client lives in a bundled file called host-main.cjs (in the sand-host directory). Grok Bot BYOK patches this file to:

• Set the default model to your configured model
• Replace the createCursorInferencePromptSession function with a "routing client" that proxies all .stream() calls through the local Grok Bot BYOK instance (reading the proxy URL from inference-proxy.url)

A backup of the original file is saved as host-main.cjs.bak so the uninstall script can restore it.

Grok Bot updates its own host-main.cjs periodically (on restart or update). A background watcher daemon monitors the file and re-applies the patch automatically when it changes. Grok Bot BYOK keeps working across Grok Bot updates without manual intervention.

Grok Bot BYOK routes requests through providers in priority order with automatic failover. The registry skips providers with missing API keys automatically (except keyless ones).

Default failover order. openrouter then opencode-go then kilo then opencode-zen then local.

All keys go in .env in the project root. Grok Bot BYOK sources .env automatically at startup. See .env.example for the full template.

# .env
OPENCODE_API_KEY=sk-your-opencode-key
KILO_API_KEY=your-kilo-key          # optional (keyless works without this)
OPENROUTER_API_KEY=sk-or-v1-your-key
LOCAL_API_KEY=sk-local-test         # must match WindsurfAPI's API_KEY

String values in config/config.json support ${ENV_VAR} interpolation, so the config references keys by env var name.

Config is a single JSON file (config/config.json), validated by a zod schema at startup. Copy config/config.example.json to get started.

{
  "port": 8788,
  "host": "127.0.0.1",
  "failover": true,
  "requestTimeoutMs": 30000,
  "routingStrategy": "latency",
  "sessionAffinity": { "enabled": false, "ttlMs": 3600000 },
  "providers": {
    "priority": ["openrouter", "opencode-go", "kilo", "opencode-zen", "local"],
    "configs": { ... }
  },
  "hostConfig": {
    "sandHostDir": "${SAND_HOST_DIR}",
    "defaultModel": "Ox Alpha Free"
  }
}

Each provider in configs supports:

A provider can hold multiple keys for load distribution and failover:

{
  "openrouter": {
    "baseUrl": "https://openrouter.ai/api/v1",
    "keys": [
      { "value": "${OPENROUTER_KEY_1}", "weight": 2 },
      { "value": "${OPENROUTER_KEY_2}", "weight": 1 }
    ],
    "defaultModel": "stealth/ox-alpha",
    "models": { ... }
  }
}

Keys rotate round-robin. A key that returns 401/403 is marked failed and removed from rotation. When all keys fail, the set resets so they get another chance.

npm run setup              # Start/restart everything (builds, stops old, starts new)
npm run setup -- --status  # Check daemon status (exit 0 if all running, 1 if not)
npm run setup -- --stop    # Stop all daemons
npm run setup -- --no-build # Restart without rebuilding (same as npm start)
npm run setup -- --no-watch # Start without host watcher
npm run setup -- --quiet   # Minimal output (agent-friendly)
npm start                  # Start/restart all daemons (no rebuild, re-patches host)
npm stop                   # Stop all daemons
npm status                 # Check daemon status

The host watcher is what makes re-attachment robust. When Grok Bot's supervisor updates or reinstalls host-main.cjs, the watcher detects the change, re-applies the routing-client patch, and restarts the host process. This happens automatically. No manual intervention needed after Grok Bot updates.

The health watchdog parses Grok Bot BYOK's log to compute an error rate over a rolling 5-minute window. If the error rate exceeds 50%, it runs the deploy script to rebuild and restart. A deploy cooldown (5 minutes) prevents deploy storms.

tail -f /tmp/inference-shim.log        # Live shim log (requests, routing, errors)
tail -f /tmp/grokbot-watch-host.log    # Host watcher log (patch events)
tail -f /tmp/grokbot-health-check.log  # Health watchdog log (health status)

All logs are JSON lines. Pipe through jq for readable output:

tail -100 /tmp/inference-shim.log | jq 'select(.level=="error")'

# Stop daemons, remove PID/log files, restore Grok Bot's host-main.cjs
npm run uninstall

# Full removal: also delete .env, config.json, dist/, node_modules/
npm run uninstall -- --purge

# Quiet mode (agent-friendly)
npm run uninstall -- --quiet

# Leave the host patch in place (don't restore host-main.cjs)
npm run uninstall -- --keep-host

The uninstall script:

• Stops all three daemons (Grok Bot BYOK, host watcher, health watchdog)
• Removes all PID files and log files from /tmp/
• Restores Grok Bot's host-main.cjs from the .bak backup (if one exists)
• Removes the inference-proxy.url file and deployed Grok Bot BYOK symlink
• With --purge: also removes .env, config/config.json, dist/, node_modules/
• The git repository itself is preserved. Delete the directory manually if needed.

Set routingStrategy in config/config.json:

If no provider claims the model id, the first provider in the priority list is used as the default fallback.

When failover: true (default), Grok Bot BYOK tries providers in priority order until one succeeds. Each provider has an independent circuit breaker:

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: 3 consecutive failures
    Open --> HalfOpen: cooldown elapsed
    HalfOpen --> Closed: probe succeeds
    HalfOpen --> Open: probe fails
    Open --> Open: cooldown not elapsed

The circuit states:

• After 3 consecutive failures (configurable via failureThreshold), the circuit opens and the provider is skipped
• After the cooldown elapses (10s for rate limits, 30s for server errors), one half-open probe is allowed
• If the probe succeeds, the circuit closes
• If the probe fails, the circuit stays open for another cooldown period

Error classification determines whether to retry, failover, or stop:

If a provider returns a 200 response but produces no content (empty completion), Grok Bot BYOK fails over to the next provider instead of returning an empty response. This handles providers that accept the request but silently produce nothing, a common failure mode for free-tier endpoints.

If the client disconnects during a stream, the failover loop stops immediately. No point burning upstream provider quota on a response no one will receive.

When routingStrategy: "latency", Grok Bot BYOK maintains per-provider performance scores using exponentially-weighted moving averages (EWMA) of three signals:

• Prefill speed. TTFB divided by prompt tokens (ms per prompt token).
• Decode speed. Completion tokens divided by generation duration (tokens/sec).
• Error rate. Fraction of failed requests.

The combined score projects each provider's rates onto a reference workload (rolling average of actual request sizes) to answer one question: "If I sent my average-sized request to each provider, which would finish first?"

score = (prefillMsPerPromptToken × refPromptTokens
       + 1000 / tokensPerSec × refCompletionTokens)
       × (1 + errorRate × 0.2)
       × stalenessDecay

Lower is better. Four things keep the scores honest:

• Epsilon-greedy exploration. 10% of requests go to a random provider to keep scores fresh and discover faster providers.
• Shadow probes. Every 50th request, a concurrent probe goes to the second-best provider to calibrate scores with apples-to-apples data.
• Staleness decay. Providers with old data get a reduced score to encourage re-sampling.
• Client-disconnect awareness. Client-side aborts don't count as provider failures, so error rates stay clean.

When sessionAffinity.enabled: true, Grok Bot BYOK binds a session ID (derived from the request's invocationId) to a provider so subsequent requests from the same conversation go to the same provider. This preserves server-side prompt caching and conversation continuity.

Bindings expire after ttlMs (default: 1 hour). A cleanup sweep runs every 5 minutes.

When a request contains images but the resolved model doesn't support vision, Grok Bot BYOK re-routes to VISION_FALLBACK_MODEL (default: qwen3.8-max) instead of silently stripping the images. If no fallback model is configured, Grok Bot BYOK replaces images with a placeholder text and sends the request to the original model.

Vision support is auto-detected per model based on the model id and provider (see detectSupportsImages in src/providers/compat.ts).

Grok Bot BYOK handles tool calls across the Connect to OpenAI translation boundary:

• Tool call accumulation. OpenAI streams tool calls as deltas spread across multiple SSE chunks. The accumulator merges them into complete tool calls and emits them as Connect toolCallPart frames.
• Argument repair. Streaming JSON arguments that are split mid-token are repaired using a streaming JSON parser.
• Inline tool-call rescue. When a model emits tool calls as text in a non-standard dialect (Kimi <|tool_call_begin|>, DeepSeek DSML, Qwen <tool_calls> XML), Grok Bot BYOK detects and re-parses them into structured tool calls.
• Markup healing. A streaming filter strips leaked chat-template markup from visible content and reconstructs tool calls and reasoning from it.
• Schema validation (optional, VALIDATE_TOOL_ARGUMENTS=true). Validates flushed tool calls against their JSON schemas. Definite violations trigger failover to the next provider.

All logs are JSON lines with a timestamp, level, message, and structured fields. Every request gets a unique requestId injected into all log lines for that request, so you can trace a single request through the full lifecycle.

API keys are redacted from all log output at the console level. No sk-*, gsk_*, AIza*, Bearer token, or JWT ever reaches stdout, even from code that hasn't been written yet.

Grok Bot BYOK logs per-provider metrics every 5 minutes:

{"msg":"provider metrics","provider":"openrouter","requests":16,
 "successes":16,"errors":0,"errorRate":0,"avgLatencyMs":33962,
 "avgTtfbMs":13754,"latencyHistogram":{...}}

When routingStrategy: "latency", it also logs per-provider performance scores:

{"msg":"provider performance","provider":"openrouter","samples":16,
 "avgPrefillMsPerPromptToken":0.12,"avgTokensPerSec":89.2,
 "errorRate":0.0,"score":1234.56,"ageMs":5000,
 "refPromptTokens":10000,"refCompletionTokens":500}

Set CAPTURE_BODIES=true to log truncated request/response bodies for debugging. Bodies are truncated to CAPTURE_MAX_BYTES (default 4096). Tool-call arguments are NOT redacted. This is a debugging tool, not a production log.

A process-level safety net catches unhandled errors and rejections. Transient transport errors (ECONNRESET, UND_ERR_SOCKET) are swallowed. A CDN edge resetting a socket must never crash the process. Everything else (programming bugs) stays fatal (exit 1) to surface loudly.

Two optional hard limits, both off by default:

1. Token budget (REQUEST_MAX_TOKENS_BUDGET). Pre-flight check that estimated input plus requested output fits within a per-request token ceiling. Oversized requests get a 413 before any provider is tried. Requests without an explicit max_tokens have their output capped to the budget remainder instead of being rejected.

2. Consecutive failure breaker (MAX_CONSECUTIVE_UPSTREAM_FAILS). Stops the failover chain after N consecutive upstream failures, returning a 503 instead of grinding through a pool that's failing across the board.

For advanced deployment (atomic rebuild with symlink swap and host restart):

# Full deploy: typecheck, build, stop old, symlink, start, patch host, test
node dist/scripts/deploy.js

# Copy instead of symlink
node dist/scripts/deploy.js --copy

# Skip host restart
node dist/scripts/deploy.js --no-restart

The health watchdog calls this script automatically when Grok Bot BYOK's error rate exceeds 50%. You rarely need to run it manually. Use npm run setup for normal restarts instead.

Check the setup output for the specific error. Common causes:

• Missing API keys. The setup script exits with a message showing which keys are needed and how to provide them. Kilo is keyless, so npm run setup with no keys still works.
• Node version too old. Requires Node 18+. Check with node --version.
• Port 8788 in use. Another process is holding the port. Stop it or set SHIM_PORT to a different value in .env.
• Grok Bot not found. The setup script looks for host-main.cjs in $HOME/sand-host. If your Grok Bot host-main.cjs is in a different location, set SAND_HOST_DIR in .env.

npm status                                    # Are the daemons running?
tail -50 /tmp/inference-shim.log | jq 'select(.level=="error")'  # Any errors?

If Grok Bot BYOK crashed, the health watchdog should auto-redeploy within 5 minutes. If it didn't, run npm run setup to restart everything.

If the host patch was lost (Grok Bot updated and the watcher didn't catch it), run npm run setup to re-patch.

tail -100 /tmp/inference-shim.log | jq 'select(.msg=="all providers failed")'

Check which providers are in the failover chain and whether their API keys are valid. A 401/403 means the key is bad. A 429 means rate limited (wait for the cooldown). A 5xx means the provider is down (the circuit breaker will skip it after 3 failures).

The registry skips providers whose API key is empty or unresolved (the ${VAR} literal remains because the env var isn't set). Check that the key exists in .env and that the env var name in config.json matches.

The host patch may have been lost. Verify:

# Check if the proxy URL file exists
ls -la $HOME/sand-host/inference-proxy.url

# Check if host-main.cjs has the routing client patch
grep -c "routingClient" $HOME/sand-host/host-main.cjs

If either is missing, run npm run setup to re-patch.

npm test              # Run all 359 tests
npm run test:watch    # Watch mode
npm run typecheck     # Type check only (tsc --noEmit)

The test suite covers:

• Protocol encoding/decoding (Connect envelopes, edge cases)
• Request/response translation (Connect to OpenAI and back)
• Provider routing (priority, latency, round-robin, weighted)
• Circuit breaker and failover logic
• Retry with backoff
• Session affinity
• Stream timeout handling
• Performance tracker scoring (EWMA, sentinel handling, staleness)
• Server integration (full HTTP lifecycle with mock upstreams)
• End-to-end (real Grok Bot BYOK server, real HTTP requests)
• Client-disconnect failover behavior
• Config validation

npm run build         # Build Grok Bot BYOK only (dist/shim.js)
npm run build:all     # Build Grok Bot BYOK + scripts (dist/shim.js + dist/scripts/*.js)
npm run clean         # Remove dist/

The build uses esbuild to produce standalone bundled .js files with no runtime dependencies (except ajv and zod, which are bundled in). Grok Bot BYOK is a single dist/shim.js file that can run anywhere Node 18+ is available.

src/
  shim.ts                       Entry point (PID lock, startup, shutdown)
  server.ts                     HTTP server, the request lifecycle
  config.ts + config/schema.ts  Config loader + zod validation
  log.ts                        Structured JSON logger
  types.ts                      Shared types (Provider, ShimConfig, etc.)
  protocol/                     Connect-RPC codec, SSE parser, proto3 helpers
  translate/                    Connect to OpenAI translation (request, response,
                                tools, vision, think-tags, markup healing)
  providers/                    Provider registry, routing, circuit breaker,
                                retry, latency scoring, session affinity
  observability/                Metrics, logging, error classification,
                                guardrails, credential redaction
  utils/                        Daemon management, .env loader, JSON repair
scripts/
  setup.ts                      One-command setup
  start-shim.ts                 Grok Bot BYOK launcher (WindsurfAPI auto-start)
  deploy.ts                     Atomic deploy (build, symlink, restart)
  patch-host.ts                 Host bundle patcher
  watch-host.ts                 Bundle watcher (auto re-patch)
  health-check.ts               Health watchdog (auto-deploy on failure)
  uninstall.ts                  Uninstall
config/
  config.example.json           Example config with all providers
test/
  13 test files, 359 tests

MIT