33rd of 55 in Model serving
LocalAI
Self-hosted inference server with OpenAI-compatible API and pluggable backends
- Stars
- 49.9k
- License
- MIT
- Last commit
- Oct 2026
- Last release
- Oct 2026
- Language
- Go
43,793 on 26 Mar 202649,902 on 22 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
LocalAI serves LLMs, vision, speech, image and video models behind OpenAI, Anthropic, ElevenLabs and Ollama-compatible APIs. Each backend (llama.cpp, vLLM, whisper.cpp, diffusers, MLX and others) is a separate OCI image pulled on demand. It also ships a web UI, API-key auth with quotas, built-in agents with MCP, and a distributed mode using PostgreSQL and NATS.
Who it is for: Self-hosters wanting a local, OpenAI-compatible server for many model types
Ranked among open-source alternatives to LM Studio (#4 of 10) and OpenAI API (#3 of 10).
Strengths
- Covers text, vision, speech, image and video models behind one API
- Runs on CPU, CUDA 12/13, ROCm, Intel oneAPI, Vulkan, Apple Silicon and Jetson
- Backends install on demand, so only the engines a model needs are pulled
- Includes API-key auth, user quotas and a distributed mode for horizontal scaling
Weaknesses
- Distributed mode requires PostgreSQL and NATS
- macOS DMG is not signed by Apple and needs a manual xattr command
- Wide scope (agents, fine-tuning, UI) makes the feature surface large to evaluate
- vllm.cpp backend is described as alpha development builds
What it needs
- GPU optional
- Docker + Compose
- Needs PostgreSQL (distributed mode), NATS (distributed mode)
- Models: GGUF, llama.cpp, vLLM, SGLang, transformers
- port 8080
Quick answers
Is LocalAI open source?
Yes. Its code is on GitHub under the MIT license.
Can you run LocalAI with Docker?
Yes. Its repository has a Docker setup: a Dockerfile, a Compose file or a published image.
Does LocalAI need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Head to head
Also in Model serving
See all 55| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 93 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 92 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 88 out of 100 |
| #5 | colibriC inference engine that streams MoE experts from disk to run huge models | 85 out of 100 |
| #6 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 85 out of 100 |