#3333rd of 55 in Model serving
llama-swap
Go proxy that starts and swaps local inference servers per request
- Stars
- 6.1k
- License
- MIT
- Last commit
- Oct 2026
- Last release
- Oct 2026
- Language
- Go
2,853 on 26 Mar 20266,062 on 22 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
llama-swap is a single Go binary that sits in front of local inference servers such as llama.cpp, vllm, stable-diffusion.cpp and audio.cpp. It reads the `model` field of each OpenAI- or Anthropic-style request, starts the matching server from a YAML `cmd`, and stops the previous one. It includes a web UI, API keys, TTL unloading and a matrix DSL for running models concurrently.
Who it is for: Self-hosters running several local models on one machine behind one API
Strengths
- One binary and one YAML file, no external dependencies
- Works with any OpenAI- or Anthropic-compatible upstream server, not only llama.cpp
- Matrix DSL runs chosen models concurrently; ttl unloads idle ones
- Built-in web UI with playground, token metrics, log streaming and Prometheus /metrics
Weaknesses
- Does not run inference itself; you supply and configure the upstream servers
- Originally designed for llama-server; Python servers like vllm should run in Docker or Podman
- Swapping reloads the model, so the first request after a switch is slow
- Auth is limited to API keys; no user accounts mentioned
What it needs
- GPU optional
- Docker
- Needs llama.cpp or another OpenAI-compatible inference server
- Models: GGUF via llama.cpp, vllm, stable-diffusion.cpp, whisper.cpp, audio.cpp
- port 8080
Quick answers
Is llama-swap open source?
Yes. Its code is on GitHub under the MIT license.
Can you run llama-swap with Docker?
Yes. Its repository has a Docker setup: a Dockerfile, a Compose file or a published image.
Does llama-swap need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Also in Model serving
See all 55| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 93 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 92 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 90 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 88 out of 100 |
| #5 | colibriC inference engine that streams MoE experts from disk to run huge models | 85 out of 100 |