The Lightweight OpenAI API Server

🔒 Local Inference Without Dependencies 🚀

Languages: 简体中文 · 繁體中文

Shimmy will be free forever. No asterisks. No "free for now." No pivot to paid.

🚀 If Shimmy helps you, consider sponsoring — 100% of support goes to keeping it free forever.

• $5/month: Coffee tier ☕ - Eternal gratitude + sponsor badge
• $25/month: Bug prioritizer 🐛 - Priority support + name in SPONSORS.md
• $100/month: Corporate backer 🏢 - Logo placement + monthly office hours
• $500/month: Infrastructure partner 🚀 - Direct support + roadmap input

🎯 Become a Sponsor | See our amazing sponsors 🙏

• What Is Shimmy?
• 🔥 Airframe Engine (v2.0)
• ⚡ TurboShimmy INT4 KV (v2.1)
• 🎯 Supported Models
• 📦 Migrating from v1.x
• ⚡ Quick Start (30 seconds)
• 🚀 OpenAI SDK Compatibility
• 🔧 Extended Context
• 📥 Download & Install
• 🔗 Integration Examples
• 📖 API Reference
• ❓ FAQ
• 🏛️ Technical Architecture
• 📚 Documentation Hub
• 🌍 Community & Support
• ⚡ Performance
• License

Shimmy is a single-binary that provides 100% OpenAI-compatible endpoints for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free.

🎉 NEW in v2.0.0: Shimmy now runs on Airframe, a pure-Rust WGSL GPU engine. No C++ toolchain, no backend flags, no compilation required.

⚡ NEW in v2.1.0: TurboShimmy INT4 KV — ~7× less KV cache VRAM with one flag. Run Llama-3.2-3B on 4 GB GPUs.

Starting in v2.0.0, Shimmy's default inference engine is Airframe — a pure-Rust WebGPU (WGSL) transformer runtime built from scratch.

Why this matters:

• No C++ toolchain required — Rust only, top to bottom
• F32 precision throughout for deterministic, high-quality output
• WGSL compute shaders work on any GPU via WebGPU (NVIDIA, AMD, Intel, integrated)
• Model spec auto-derived from GGUF metadata — no hardcoded per-model constants
• YaRN RoPE scaling for extended context via SHIMMY_MAX_CTX — engine allocates KV cache and sets RoPE scale automatically (see Extended Context below)

Quick start with Airframe (v2.0.0+):

# Default: 2048-token context
SHIMMY_BASE_GGUF=/path/to/TinyLlama-1.1B-Chat-v1.0.Q4_0.gguf ./shimmy serve

# Extended context (4096 tokens — YaRN RoPE enabled automatically, KV cache resized)
SHIMMY_BASE_GGUF=/path/to/model.gguf SHIMMY_MAX_CTX=4096 ./shimmy serve

TurboShimmy is Shimmy's on-GPU INT4 KV-cache compression system, shipping in v2.1.0. It squeezes the KV cache from 32-bit floats down to per-head-vector 4-bit integers — entirely in WGSL compute shaders with no CPU roundtrips — delivering ~7× less KV VRAM with no measurable quality loss at normal context lengths.

One flag. ~7× less KV VRAM. Same output quality.

# Enable TurboShimmy on any GGUF model
./shimmy serve --kv-quant int4

# Or via environment variable (docker-compose, systemd, etc.)
SHIMMY_KV_QUANT=int4 ./shimmy serve

# Windows GPU + long prompts: reduce per-dispatch work to prevent TDR resets
./shimmy serve --kv-quant int4 --prefill-chunk 8

Why it matters — TurboShimmy changes what fits on your GPU:

VRAM comparison (Llama-3.2-3B, ctx=2048):

VRAM comparison (TinyLlama 1.1B, ctx=2048):

When to use TurboShimmy:

How it works: Each K/V head vector is independently quantized to 4-bit integers with a per-vector F32 scale factor, encoded as packed nibbles by WGSL compute shaders. Dequantization happens on-the-fly when computing attention scores — also on GPU. The Airframe engine's helical context-shift operates directly on the packed INT4 representation. No CPU roundtrips at any step. Full architecture details in the Airframe engine.

Quality validation: Needle-in-a-haystack benchmarks on Llama-3.2-3B show zero retrieval degradation vs F32 at ctx≤2048 across all tested depths (15%, 50%, 85%). Full benchmark data and setup guide: TurboShimmy on the wiki.

Windows stability: Airframe v0.2.1 ships a device.on_uncaptured_error handler so GPU validation errors surface as clean HTTP 500 responses instead of crashes. Use --prefill-chunk 8 to prevent Windows TDR resets during long prefills on older GPUs (GTX 10xx/16xx series).

Airframe v2.0 ships with GPU-verified support across 7 model architectures and 5 quantization types, covering the models most commonly run on consumer hardware. Context window is read directly from each model's GGUF metadata — no hardcoded limits.

* Llama-3.2's native context is 131072 tokens. Airframe reads this from GGUF and allocates KV cache accordingly. Use SHIMMY_MAX_CTX=8192 for a practical 8K window on consumer hardware (~256 MB KV cache for the 1B model).

GPU Math Verified means the Airframe GPU dequantization shader produces results matching the CPU reference implementation, independently confirmed for every tensor type in each model. This is done via quant_verify, which tests 512 elements per quantization type per model.

All types are implemented in both the GPU inference shader and a CPU reference implementation. GPU vs CPU agreement is validated for every type.

Auto-discovery is enabled. If Shimmy finds GGUF models in your HuggingFace cache, Ollama directory, LM Studio cache (~/.cache/lm-studio/models), or local ./models/ folder, it will register and serve them automatically. See docs/MODEL_EXPANSION.md for the full compatibility matrix.

The llama.cpp backend is removed in v2.0.0. The Airframe engine is the only inference path. See docs/MIGRATION_v2.md for the step-by-step migration guide.

Whether you're forking Shimmy or integrating it as a service, we provide complete documentation and integration templates.

# 1) Download pre-built binary
# Windows:
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe
set SHIMMY_BASE_GGUF=C:\path\to\model.gguf && ./shimmy.exe serve &

# Linux / macOS:
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-x86_64 -o shimmy && chmod +x shimmy
SHIMMY_BASE_GGUF=/path/to/model.gguf ./shimmy serve &

# 2) See registered models
./shimmy list

# 3) Smoke test the OpenAI API
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
        "model":"tinyllama-1.1b",
        "messages":[{"role":"user","content":"Say hi in 5 words."}],
        "max_tokens":32
      }' | jq -r '.choices[0].message.content'

No code changes needed - just change the API endpoint:

• Any OpenAI client: Python, Node.js, curl, etc.
• Development applications: Compatible with standard SDKs
• VSCode Extensions: Point to http://localhost:11435
• Cursor Editor: Built-in OpenAI compatibility
• Continue.dev: Drop-in model provider

• Node.js (openai v4)

import OpenAI from "openai";

const openai = new OpenAI({
  baseURL: "http://127.0.0.1:11435/v1",
  apiKey: "sk-local", // placeholder, Shimmy ignores it
});

const resp = await openai.chat.completions.create({
  model: "REPLACE_WITH_MODEL",
  messages: [{ role: "user", content: "Say hi in 5 words." }],
  max_tokens: 32,
});

console.log(resp.choices[0].message?.content);

• Python (openai>=1.0.0)

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="sk-local")

resp = client.chat.completions.create(
    model="REPLACE_WITH_MODEL",
    messages=[{"role": "user", "content": "Say hi in 5 words."}],
    max_tokens=32,
)

print(resp.choices[0].message.content)

• Automatically finds models from Hugging Face cache, Ollama, LM Studio (~/.cache/lm-studio/models), and local dirs
• Auto-allocates ports to avoid conflicts
• Auto-detects LoRA adapters for specialized models
• Just works - no config files, no setup wizards

Note: MoE (Mixture of Experts) CPU offloading is on the Airframe roadmap. See docs/AIRFRAME_MOE_ROADMAP.md for the implementation plan.

Run 70B+ models on consumer hardware — coming to the Airframe engine. Track progress on the roadmap.

Perfect for: Large models (70B+), limited VRAM systems, cost-effective inference

• Privacy: Your code never leaves your machine
• Cost: No API keys, no per-token billing
• Speed: Local inference, sub-second responses
• Reliability: No rate limits, no downtime

v2.0.0: Download pre-built binaries with Airframe WebGPU engine included!

Pick your platform and download — no compilation needed, GPU acceleration included:

# Windows x64 (Airframe WebGPU engine)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe

# Linux x86_64 (Airframe WebGPU engine)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-x86_64 -o shimmy && chmod +x shimmy

# macOS ARM64 (Airframe with Metal backend via wgpu)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-arm64 -o shimmy && chmod +x shimmy

# macOS Intel
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-intel -o shimmy && chmod +x shimmy

# Linux ARM64 (huggingface engine; Airframe cross-compilation not yet supported)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-aarch64 -o shimmy && chmod +x shimmy

That's it! The Airframe WebGPU adapter is selected automatically at runtime.

# Install from crates.io
cargo install shimmy

# Build from source (huggingface engine, no GPU)
git clone https://github.com/Michael-A-Kuykendall/shimmy
cd shimmy
cargo build --release

Note: The Airframe GPU engine is a private dependency and cannot be built from source by public users. The pre-built release binaries already include Airframe compiled in — download those to get full GPU acceleration. cargo install shimmy installs the huggingface engine variant from crates.io.

v2.0.0: Airframe uses WebGPU (wgpu) for GPU acceleration. No backend flags, no driver installation beyond standard OS graphics drivers.

Release binaries include the Airframe engine with WebGPU support compiled in:

Airframe uses wgpu's adapter enumeration. On first launch it selects the best available GPU adapter for your system — discrete GPU preferred over integrated, integrated over CPU fallback. No configuration needed.

# Check selected adapter
shimmy gpu-info

# Start serving (GPU adapter auto-selected)
shimmy serve

SHIMMY_MAX_CTX overrides the context window at the engine level. When set above the model's native window, Airframe automatically engages YaRN RoPE scaling and resizes the KV cache accordingly.

# 4096-token context with YaRN (2x native window for TinyLlama)
SHIMMY_BASE_GGUF=/path/to/model.gguf SHIMMY_MAX_CTX=4096 shimmy serve

# 8192 tokens (4x native, higher RoPE compression)
SHIMMY_BASE_GGUF=/path/to/model.gguf SHIMMY_MAX_CTX=8192 shimmy serve

Note: Extended context beyond 4096 is functional but not yet as deeply validated as the native 2048-token window. Accepted range is 512–131072. Values outside that range are silently ignored and 2048 is used.

Airframe allocates VRAM at load time: weights + KV cache. The KV cache is F32 and scales linearly with context length (n_layers × n_kv_heads × head_dim × ctx × 2 × 4 bytes).

TinyLlama 1.1B Q4_0 — the v2.0 validated path:

Integrated graphics (Intel Iris, Apple M-series unified memory, AMD Vega) running at 2048 context is ~800 MB — comfortably inside the 2 GB allocation most integrated GPUs share with system RAM.

Scaling up to larger models (architecture and quant support required — see docs/MODEL_EXPANSION.md):

The KV cache formula for any model: n_layers × n_kv_heads × head_dim × ctx × 2 × 4 bytes. Multiply the 2048 baseline by your SHIMMY_MAX_CTX multiplier to get the extended context allocation.

Shimmy auto-discovers models from:

• Hugging Face cache: ~/.cache/huggingface/hub/
• Ollama models: ~/.ollama/models/
• Local directory: ./models/
• Environment: SHIMMY_BASE_GGUF=path/to/model.gguf

# Primary validated model for Airframe v2.0
huggingface-cli download TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF \
  --include "tinyllama-1.1b-chat-v1.0.Q4_0.gguf" --local-dir ./models/

# Alternative 1B — also fits in the same hardware envelope
huggingface-cli download bartowski/Llama-3.2-1B-Instruct-GGUF \
  --include "*Q4_K_M*" --local-dir ./models/

# Auto-allocates port to avoid conflicts
shimmy serve

# Or use manual port
shimmy serve --bind 127.0.0.1:11435

Point your development tools to the displayed port — VSCode Copilot, Cursor, Continue.dev all work instantly.

• Rust: cargo install shimmy (installs huggingface engine; for Airframe GPU, use GitHub Releases binaries)
• VS Code: Shimmy Extension
• npm: npm install -g shimmy-js (planned)
• Python: pip install shimmy (planned)

• GitHub Releases: Latest binaries
• Docker: docker pull shimmy/shimmy:latest (coming soon)

Full compatibility confirmed! Shimmy works on macOS with Metal GPU acceleration via wgpu.

# Install from crates.io (huggingface engine)
cargo install shimmy

# For Airframe GPU engine, download the macOS binary from GitHub Releases:
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-arm64 -o shimmy && chmod +x shimmy

✅ Verified working:

• Intel and Apple Silicon Macs
• Metal GPU acceleration via wgpu (automatic on Apple Silicon)
• Xcode 17+ compatibility

{
  "github.copilot.advanced": {
    "serverUrl": "http://localhost:11435"
  }
}

{
  "models": [{
    "title": "Local Shimmy",
    "provider": "openai",
    "model": "your-model-name",
    "apiBase": "http://localhost:11435/v1"
  }]
}

Works out of the box - just point to http://localhost:11435/v1

I built Shimmy to retain privacy-first control on my AI development and keep things local and lean.

This is my commitment: Shimmy stays MIT licensed, forever. If you want to support development, sponsor it. If you don't, just build something cool with it.

💡 Shimmy saves you time and money. If it's useful, consider sponsoring for $5/month — less than your Netflix subscription, infinitely more useful for developers.

• GET /health - Health check
• POST /v1/chat/completions - OpenAI-compatible chat (streaming supported)
• POST /v1/completions - OpenAI-compatible text completions
• GET /v1/models - List available models
• POST /api/generate - Shimmy native API
• GET /ws/generate - WebSocket streaming

shimmy serve                              # Start server (auto port allocation)
shimmy serve --bind 127.0.0.1:8080        # Manual port binding
shimmy serve --gpu-backend auto           # WebGPU adapter auto-select (default)
shimmy serve --gpu-backend cpu            # Force CPU (disable GPU)
shimmy serve --kv-quant int4              # Enable TurboShimmy INT4 KV cache compression
shimmy serve --kv-quant int4 --prefill-chunk 8  # INT4 + Windows TDR prevention
shimmy list                               # Show available models
shimmy discover                           # Refresh model discovery
shimmy generate --name X --prompt "Hi"   # Test generation
shimmy probe model-name                   # Verify model loads
shimmy gpu-info                           # Show selected WebGPU adapter

• Rust + Tokio: Memory-safe, async performance
• Airframe engine: Pure-Rust WGSL GPU inference — no C++ toolchain, deterministic output, GGUF-native
• OpenAI API compatibility: Drop-in replacement
• Dynamic port management: Zero conflicts, auto-allocation
• Zero-config auto-discovery: Just works™

• 🧠 MOE CPU Offloading: Hybrid GPU/CPU processing for large models (70B+)
• 🎯 Smart Model Filtering: Automatically excludes non-language models (Stable Diffusion, Whisper, CLIP)
• 🛡️ 6-Gate Release Validation: Constitutional quality limits ensure reliability
• ⚡ Smart Model Preloading: Background loading with usage tracking for instant model switching
• 💾 Response Caching: LRU + TTL cache delivering 20-40% performance gains on repeat queries
• 🚀 Integration Templates: One-command deployment for Docker, Kubernetes, Railway, Fly.io, FastAPI, Express
• 🔄 Request Routing: Multi-instance support with health checking and load balancing
• 📊 Advanced Observability: Real-time metrics with self-optimization and Prometheus integration
• 🔗 RustChain Integration: Universal workflow transpilation with workflow orchestration

Does Shimmy work on my GPU? Shimmy uses WebGPU (via the Airframe engine) which runs on Vulkan, D3D12, and Metal — covering NVIDIA, AMD, Intel, and Apple Silicon. No CUDA required. See GPU requirements in TROUBLESHOOTING.md if you hit adapter errors.

What's the difference between Shimmy and llama.cpp / Ollama? Shimmy is written in pure Rust with no C++ toolchain dependency. The Airframe engine runs WGSL compute shaders compiled at startup — no pre-built binaries, no driver version pinning. The result is faster startup, lower memory overhead, and deterministic output. See the GPU Pipeline doc for internals.

Why do I need SHIMMY_BASE_GGUF or LIBSHIMMY_MODEL_PATH? If you don't set these, Shimmy auto-discovers models in standard directories (~/.cache/huggingface, ~/.ollama, ~/lm-studio/models, ~/.cache/lm-studio/models, ~/Library/Application Support/LMStudio). Set SHIMMY_BASE_GGUF to override and point directly at a specific GGUF file.

Can I run multiple models at once? Not currently — Shimmy loads one model per server instance. To serve multiple models, run multiple server instances on different ports. Hot-swapping models (reload without restart) is on the roadmap.

Why does generation stop before max_tokens? The model reached a natural end-of-sequence token. For chat models this is expected behavior — the model signals it's done. If you want to force longer output, increase max_tokens and set temperature > 0. If generation stops on the wrong token, the model may be using the wrong chat template — see CHAT_TEMPLATES.md.

Is there streaming support? Set "stream": true in your request. Shimmy returns Server-Sent Events in the standard OpenAI streaming format.

Q4_K_M vs Q4_0 — which should I use? Q4_K_M (K-quant) is consistently better quality than Q4_0 for the same file size. Use Q4_0 only when you need maximum compatibility or the model isn't available in K-quant. See QUANTIZATION.md for the full analysis.

Can I extend the context window beyond what the model was trained on? Yes — set SHIMMY_MAX_CTX to any value. Airframe applies YaRN scaling automatically when the requested context exceeds the model's native context. Quality degrades gradually beyond 2× the native context. See EXTENDED_CONTEXT.md.

Full documentation lives in docs/. Use this table to find what you need:

• 🐛 Bug Reports: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 📖 Documentation: Full Documentation Hub • Migration Guide v1→v2 • Engineering Methodology • OpenAI Compatibility Matrix • Architecture • GPU Pipeline • Troubleshooting
• 💝 Sponsorship: GitHub Sponsors

🌟 stars and climbing fast ⏱ <1s startup 🦀 100% Rust, no Python

🔥 Hacker News • Front Page Again • IPE Newsletter

Companies: Need invoicing? Email michaelallenkuykendall@gmail.com

Shimmy maintains high code quality through comprehensive testing:

• Comprehensive test suite with property-based testing
• Automated CI/CD pipeline with quality gates
• Runtime invariant checking for critical operations
• Cross-platform compatibility testing

Run the complete test suite:

# Using cargo aliases
cargo test-quick           # Quick development tests

# Using Makefile  
make test                  # Full test suite
make test-quick            # Quick development tests

See our testing approach for technical details.

MIT License - forever and always.

Philosophy: Infrastructure should be invisible. Shimmy is infrastructure.

Testing Philosophy: Reliability through comprehensive validation and property-based testing.

Forever maintainer: Michael A. Kuykendall Promise: This will never become a paid product Mission: Making local model inference simple and reliable