#66th of 55 in Model serving
strata
Runs a 125B-parameter MoE model on one consumer GPU plus system RAM
- Stars
- 31.4k
- License
- MIT
- Last commit
- Oct 2026
- Last release
- Oct 2026
- Language
- C++
127 on 25 Sept 202631,433 on 23 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Strata is a C++ inference engine, built on parts of llama.cpp/ggml, that runs Qwen3.8-Flash-Next on a PC with a 12 GB or larger NVIDIA or AMD card. It keeps the most-used experts in VRAM, holds all of them in RAM and uses the CPU for the rest. It serves a browser chat app plus OpenAI-, Anthropic- and Responses-style APIs on port 8080.
Who it is for: Self-hosters with a 12 GB+ gaming GPU and 32 GB+ RAM wanting a large local model
Ranked among open-source alternatives to LM Studio (#10 of 10) and OpenAI API (#6 of 10).
Strengths
- Serves OpenAI /v1, Anthropic /v1/messages and Responses /v1/responses on 127.0.0.1:8080
- Reports 53-94 tokens/s decode on an RTX 5070 12 GB, depending on quantization
- Installer picks a model size by RAM; interrupted downloads resume
- Ships a BENCH script, an MCP server for setup, and multi-GPU support
Weaknesses
- Built around one model family (Qwen3.8-Flash-Next), not arbitrary GGUFs
- Needs 32 GB or more RAM and about 80 GB disk; PC may freeze 1-3 minutes at load
- Answers one request at a time by default; parallel mode slows each answer on 12 GB cards
- Intel Arc, Mac, older GPUs and Strix Halo support is experimental and community-tested
What it needs
- RAM ≥ 32 GB
- GPU required
- Docker + Compose
- Needs NVIDIA or AMD GPU driver, Qwen3.8-Flash-Next model download (about 70 GB)
- Models: Qwen3.8-Flash-Next GGUF (Q2_0, IQ2_XS, IQ3_XXS, IQ3_S, Coder, Swift 1.5), Unsloth UD-IQ4_XS to Q8_0
- port 8080
Quick answers
Is strata open source?
Yes. Its code is on GitHub under the MIT license.
Can you run strata with Docker?
Yes. Its repository has a Docker setup: a Dockerfile, a Compose file or a published image.
Does strata need a GPU?
Yes, it needs a GPU. Plan for at least 32 GB of RAM.
Also in Model serving
See all 55| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 93 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 92 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 90 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 88 out of 100 |
| #5 | colibriC inference engine that streams MoE experts from disk to run huge models | 85 out of 100 |