LlamaStash

Zero-overhead, terminal-native local-LLM manager.

A fast TUI and CLI with init wizard for managing local LLMs. One Rust binary that's a TUI, a CLI, a daemon, and an OpenAI-compatible proxy. llama.cpp is the direct, zero-overhead default backend (vs raw llama-server), plus Lemonade for NPU / multi-engine inference, vLLM and SGLang for safetensors, and ds4 for DeepSeek-V4. See benchmarks.

• Why
• Install
• Quickstart
• Agent Skills
• Features
• Benchmarks
• Screenshots
• Configuration
• CLI exit codes
• Platforms
• Roadmap
• Contributing

Heavy abstractions (Ollama, LM Studio) hide llama.cpp; raw llama-server use is tedious. LlamaStash is a fast, transparent manager that is also a first-class shell-tool surface for agents — one binary, daemon on demand, same primitives in the TUI and the CLI.

AI agents installing this for a user: jump to INSTALL.md § For AI agents. The non-interactive install + verify contract, and exit-code branching live there.

Pick whichever channel you prefer — all install the same binary. Full per-platform notes, troubleshooting, and the agent-friendly non-interactive path live in INSTALL.md.

# macOS + Linux, one-shot
curl -fsSL https://llamastash.dev/install.sh | sh

# Windows 11 (PowerShell, no admin elevation)
irm https://llamastash.dev/install.ps1 | iex

# Homebrew (macOS + Linuxbrew)
brew install llamastash/llamastash/llamastash

# Arch Linux (AUR — pick one)
yay -S llamastash       # source build

# From crates.io (any platform with a Rust toolchain)
cargo install llamastash

# Windows via Scoop bucket
scoop bucket add llamastash https://github.com/llamastash/scoop-llamastash && scoop install llamastash

Windows: 64-bit Windows 10 1809+ / Windows 11, PowerShell 5.1+, Windows Terminal recommended for the TUI; the bundled llama-server needs the VC++ 2015–2022 Redistributable (x64). See INSTALL.md.

Then run llamastash init — the interactive wizard installs llama-server for your hardware, downloads a starter GGUF, writes a tuned config, and smoke-launches it.

# Open the TUI. Scans default caches; daemon auto-spawns on demand.
llamastash

# List discovered models. TTY → padded + table; piped or
# `--no-colors` → TSV bytes. `--json` is the agent contract.
llamastash list
llamastash list --json | jq

# Launch a model by name, name substring, path, or canonical id.
llamastash start qwen-coder --ctx 16384 --reasoning on

# `run` is an alias for `start`, and the positional takes a YAML
# launch file: one model, its presets, nothing saved to config.yaml.
llamastash run qwen-coder.yml

# Drive a smoke-test request against the running endpoint.
curl -s http://127.0.0.1:41100/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen-coder", "messages": [{"role": "user", "content": "hi"}]}'

# Stop it.
llamastash stop qwen-coder

Tip — mouse focus. Mouse capture is off by default so the terminal keeps native click-and-drag text selection. To opt in on every TUI run, alias the binary in your shell rc:

# bash / zsh
alias llamastash='llamastash --mouse-focus'

# fish
alias llamastash 'llamastash --mouse-focus'

Or set it permanently in config.yaml:

mouse_focus: true

Either source flips on click-to-focus for the Models list, the right pane, and the tab labels (Settings/Logs/Chat/Embed/Rerank). Most terminals still expose a bypass modifier (Shift on iTerm2 / Alacritty / foot / wezterm, Option on Apple Terminal) so ad-hoc selection stays reachable.

Full subcommand reference: docs/usage.md. Proxy client setup (including an OpenCode example): docs/usage.md#opencode-setup. Prefer a Vulkan llama-server build on AMD/NVIDIA hosts: docs/usage.md#preferring-a-vulkan-llama-server-build. Architecture and IPC contract: docs/architecture.md. When things go wrong: docs/troubleshooting.md.

The CLI ships with an Agent Skills manifest so supported agents can load repo-specific instructions for using llamastash as a local model-management CLI.

• Canonical skill bundle: skills/llamastash/

Any supported agent (recommended): install the bundled skill with the Agent Skills CLI. It auto-detects your harness and writes the bundle to the right path:

npx skills add llamastash/llamastash

Claude Code plugin marketplace: install the repo as a plugin, then install the bundled skill:

/plugin marketplace add llamastash/llamastash
/plugin install llamastash@llamastash
/reload-plugins

Manual install examples:

# OpenClaw
mkdir -p ~/.openclaw/skills && cp -r skills/llamastash ~/.openclaw/skills/

# OpenCode
mkdir -p ~/.config/opencode/skills && cp -r skills/llamastash ~/.config/opencode/skills/

The skill teaches agents to prefer --json, branch on LlamaStash's documented exit codes, reuse exact discovered model names, and read status --json proxy.listen before configuring an OpenAI-compatible client.

Full detail per feature in FEATURES.md — including trade-offs, contracts, and links into docs/usage.md.

• llamastash init — first-run wizard that detects hardware, installs llama-server, picks + downloads a starter GGUF, writes a tuned config, and smoke-launches.
• Hardware-aware model recommender with a VRAM-fit filter + composite ranking over a CI-refreshed benchmark snapshot.
• llamastash doctor — typed, agent-branchable findings; always exits 0.

• Auto-scans HuggingFace, Ollama, and LM Studio caches, plus user paths.
• Rich GGUF intelligence — architecture, params, quant, native context, chat template, KV-cache-aware memory estimates.
• Multimodal projector detection — auto-pairs an mmproj GGUF beside a model, loads it with --mmproj, flags vision (◉) / audio (♪) in the TUI.
• MTP speculative decoding — auto-detects MTP-capable GGUFs (embedded or separate mtp-*.gguf head) and enables --spec-type draft-mtp by default for ~2x decode with no quality loss; --mtp auto\|on\|off, a ↯ TUI badge, live draft-acceptance in status. pull fetches the mmproj / MTP-head companions too.
• Smart deduplication — symlinks collapsed, split GGUFs unified, Ollama blobs named.
• Live filesystem watching — new GGUFs appear without a restart.

• Daemon-on-demand — one binary as TUI + CLI + daemon; running models survive TUI close.
• Multi-model concurrency — per-model port from a configurable range, /health-probed state machine.
• GPU-aware built-in arch defaults — sensible flags per (architecture, gpu_backend) with zero YAML.
• Intelligent context auto-fit — when ctx is unset, llamastash picks the largest context that fits free VRAM (or RAM, CPU-only) from the GGUF attention geometry. Sidesteps llama.cpp --fit's 4096 collapse on Linux 7+ iGPUs (AMD Strix Halo) where unified-memory free space is mis-reported.
• Typed launch-knob editor with (source) chips and a layered preset → last-params → arch-defaults → built-ins resolver.
• Multi-GPU device selection — pin a model to one card (--device) instead of letting llama.cpp split it across every GPU; the picker lists exactly what llama-server --list-devices reports.
• Named presets, favorites, last-params recall — presets live in config.yaml (per-model or per-arch), edited from the CLI, the TUI Ctrl+P save dialog, or by hand; a preset cycle row in the Settings form picks one in a keystroke.

• Keyboard-driven everywhere — vim-style hjkl + Ctrl+F/Ctrl+B paging, 0/$ top/bottom, gt/gT tab cycling; / filter, u/c/p yank, ? help.
• Right pane is your smoke test — Logs / Chat / Embed / Rerank over the same OpenAI-compatible endpoints external clients use.
• In-TUI HuggingFace browser — search, sort, paginate, per-file hardware fit, download strip with cancel.
• Theming and rebinding — five themes + custom palette; every action rebindable.
• Accessible by default — dual-encoded status (color + glyph), readable on mono terminals.
• Adaptive layout — works from 60 cells up — below 100 cells the right pane goes drill-in-only; list columns and hint chips drop by priority rank as the pane shrinks so the model name stays readable.

• Subcommands cover every TUI capability with --json as the stable agent contract.
• Documented exit codes per failure class — pin numbers, not message text.
• Colored TTY output, byte-stable TSV when piped — existing awk / column pipelines keep working.
• llamastash pull <hf-repo> — same primitive as the wizard, with disk-space prechecks.
• llamastash recommend — the recommender on its own, agent-friendly.
• llamastash integrations — point OpenCode, Aider, Continue.dev, Zed or pi.dev at the proxy, with your favorites registered.
• Reproducible pulls via --revision <SHA>.

• OpenAI-compatible endpoint at http://127.0.0.1:11435/v1 by default (or the next free port up to 11440) — drives every discovered model through one URL; OpenCode, Pi (pi.dev), Cline, llm-cli, the OpenAI SDKs all work as-is. Auto-starts the requested model; falls back to a Ready peer with audit headers (x-llamastash-served-by, x-llamastash-fallback-reason) when launch fails. The default port is 11435 (one above Ollama's well-known 11434) so a llamastash daemon and an Ollama install can co-exist without a port collision.
• Anthropic Messages API on the same proxy — /v1/messages + /v1/messages/count_tokens forward to llama-server's native endpoints, so Claude Code and the Anthropic SDK attach via ANTHROPIC_BASE_URL (key sent as x-api-key). llamastash init's Claude Code integration drops a sourceable claude-code.sh so you opt in per-shell (source claude-code.sh && claude) without hijacking its global config. Tool calling works out of the box (--jinja on by default).
• Browser web UI at http://127.0.0.1:11435/ui/ — opens the running model's stock llama.cpp web UI through the proxy on one port-stable origin, so you never chase the ephemeral backend port. A chooser (plus /ui/switch) handles several running models; reachable over LAN behind the same key via a browser Basic-auth prompt.
• Ollama discovery surface — GET /api/tags / /api/version / /api/ps, POST /api/show so tools that auto-detect Ollama via OLLAMA_HOST recognise llamastash and fall through to the OpenAI-compat endpoints for inference.
• Ollama drop-in mode is opt-in. Enable with --ollama-compat (or proxy.ollama_compat: true / LLAMASTASH_OLLAMA_COMPAT=1) and the proxy claims port 11434, answers GET / with the byte-exact "Ollama is running" handshake string, and works as a transparent replacement for the official ollama CLI and other Ollama-Go-based clients. Leaving compat off keeps the safe coexistence default (port 11435, "LlamaStash is running" identity).
• Loopback by default, opt-in LAN with auth — the proxy binds 127.0.0.1 and runs keyless for the same-machine threat model. Expose it on the LAN with --proxy-host 0.0.0.0 (or proxy.host) and llamastash auto-generates a bearer key, requires it on every request, and refuses to bind a routable address with no key unless you pass --insecure-no-auth. TLS is on the roadmap; LAN mode is plaintext for now (trusted network or front with a reverse proxy). The control plane and llama-server children always stay loopback.

• ⚠️ Experimental — new and lightly road-tested; behaviour and config may change. llama.cpp stays the stable default.
• A pluggable backend seam. llama.cpp is the direct, zero-overhead default; Lemonade (lemond) plugs in as a second backend for engines llama.cpp can't reach — NPU inference on AMD Ryzen AI / XDNA, plus ROCm / ONNX / others. Default-on when the lemond binary resolves (like ds4); force via --lemonade / LLAMASTASH_LEMONADE=1, or set backend.lemonade.enabled: false to opt out. Zero footprint when the binary is absent.
• You install Lemonade; LlamaStash drives it. No auto-install — LlamaStash finds lemond (PATH or backend.lemonade.servers), supervises the shared umbrella, discovers its models, routes inference through the proxy, and evicts idle models by API unload. See Lemonade setup.

• ⚠️ Experimental — validated against vLLM 0.27.1 on a single Strix Halo / ROCm box; behaviour and config may change.
• The non-GGUF half of your cache. Safetensors repos sitting in ~/.cache/huggingface were invisible to LlamaStash before; now they appear in the catalog and launch through vllm serve. No competition with llama.cpp — a GGUF still binds llama.cpp (or ds4), and vLLM claims safetensors repos only.
• You install vLLM; LlamaStash drives it. Default-on when a vllm launcher resolves (PATH or backend.vllm.servers); force with --vllm / LLAMASTASH_VLLM=1, opt out with backend.vllm.enabled: false. Zero footprint when absent. On ROCm, where vLLM ships only as a container, point the config at a small wrapper script — the recipe is in vLLM setup.
• Nine native knobs (--kv-cache-memory-bytes, --gpu-memory-utilization, --tensor-parallel-size, --dtype, --kv-cache-dtype, --quantization, --max-num-seqs, --enforce-eager, --trust-remote-code) in the launch picker and presets; --ctx maps to --max-model-len.

• ⚠️ Experimental — validated against SGLang 0.5.18 on a single DGX Spark (GB10, unified memory); behaviour and config may change.
• The same rows vLLM serves. Safetensors repos launch through sglang serve; a GGUF still binds llama.cpp (or ds4). With both engines installed a repo lists both and auto picks vLLM — --backend sglang selects SGLang.
• You install SGLang; LlamaStash drives it. Default-on when a sglang launcher resolves (PATH or backend.sglang.servers); force with --sglang / LLAMASTASH_SGLANG=1, opt out with backend.sglang.enabled: false. Zero footprint when absent.
• A token cap, not a byte cap, on unified-memory hosts. SGLang's only deterministic bound on its KV pool is --max-total-tokens, so the launcher divides the shared byte budget by the model's KV bytes per token, read from config.json. Eight native knobs in the launch picker and presets; --ctx maps to --context-length.

• ⚠️ Experimental — new and lightly road-tested (validated on a single Strix Halo / ROCm box); behaviour, config, and defaults may change. llama.cpp stays the stable default and runs DeepSeek-V4 too on a current build (llama.cpp b9840+), so nothing depends on ds4.
• A third backend for antirez's ds4. ds4-server is the purpose-built engine for the DeepSeek-V4 Flash/PRO GGUFs (disk KV cache, SSD streaming). A ds4-compatible GGUF auto-routes to ds4 when the ds4-server binary is found, and falls back to llama.cpp when it isn't — a current llama.cpp (b9840+, the first release with DeepSeek-V4 support) runs these GGUFs too, so ds4 is preferred, never required. Older llama.cpp builds can't load them (unknown model architecture: 'deepseek4'). Default-on when the binary resolves; enable/force via backend.ds4 config, --ds4, or LLAMASTASH_DS4=1. An SSD-streaming launch knob runs the 81–300+ GB models on below-floor RAM. See ds4 backend.

• Bearer-token loopback control plane (runtime.json 0600) — the per-daemon token + URL live in $XDG_STATE_HOME/llamastash/runtime.json; same-UID trust, no network exposure.
• Hardened fetch substrate — HTTPS-only, host allowlist, redirect/body-size caps, IP-literal refusal.
• Archive-bomb defenses on installers — entry/size/ratio caps; SHA-256 verified before extract.
• Atomic, mode-checked config + state writes — 0600 final mode; corrupt state quarantined, not fatal.
• Side-by-side daemons — isolated instances via LLAMASTASH_*_DIR (state / config / cache); each daemon publishes its own runtime.json.

Note: This is beta software. Rough edges are to be expected. Windows and macOS support is not as well-tested as Linux; Same goes for non-AMD GPUs. Please report issues if you hit them. The llama-server builds are unmodified upstream binaries; any bugs in them are out of scope for LlamaStash.

LlamaStash spawns the unmodified upstream llama-server. Three suites track what that means in practice — Suite A asserts the wrapper adds no measurable overhead vs raw llama-server, Suite B compares LlamaStash-as-shipped against Ollama + LM Studio on the same hardware through their OpenAI-compatible endpoints, Suite C measures the proxy hop vs hitting llama-server directly (TTFT p50 +0.45 ms, decode unchanged). Full write-up + per-workload tables: docs/benchmarks.md.

Each cell below is decode tok/s / TTFT ms on the chat_turn workload (50-token prompt → 64 tokens decode). LlamaStash matches raw llama-server within ≤1% in normalized mode on every platform. Re-run on your hardware: make bench-end-to-end (Suite B) or make bench-overhead (Suite A).

chat_turn normalized mode, decode tok/s / TTFT ms. One bench JSON per row (no averaging).

¹ LM Studio's bundled ROCm vendor libraries (v6.4) abort in ggml_cuda_error during backend init on gfx1151 (Strix Halo) across all LMS-shipped runtimes. System ROCm 7.2.3 loads the same models cleanly via raw llama-server, so this is an LMS vendor-bundle limitation. LMS Vulkan numbers are in the benchmark blog and in the final report.

Same backend (Vulkan b9440 / vulkan-avx2@2.18.0), same GGUF bytes. raw llama-server and Ollama omitted: the wrapper-overhead claim already covered by the HIP table; Ollama mainline has no Vulkan support.

✦ LlamaStash leads raw llama-server in defaults mode (12–16% decode, 33–49% TTFT) via the hardware-aware defaults overlay; normalized mode: within ≤0.5 tok/s. Vulkan decode ≥ CUDA on this hardware in 26 of 28 cells (median +5%). Curated report with six findings: linux-nvidia-final.md.

✦ LlamaStash leads raw llama-server on M1 in defaults mode (99.0 vs 92.3 tok/s, 15 vs 19 ms TTFT) because its Metal defaults overlay injects hardware-optimal knobs at startup. Normalized mode: 92.2 vs 91.5 — within 1%. Curated report: macos-m1-final-report.md.

LlamaStash reads $XDG_CONFIG_HOME/llamastash/config.yaml on Linux (fallback ~/.config/llamastash/config.yaml), ~/Library/Application Support/llamastash/config.yaml on macOS, and %APPDATA%\llamastash\config\config.yaml on Windows. A fully-annotated sample lives at config.example.yaml — copy it to the path above and edit. Run llamastash config to open the active file in $EDITOR, or llamastash config bindings to print every effective keybinding as YAML. The full schema reference is in docs/usage.md.

Quick tour of the top-level keys:

Environment variables:

When model_paths and --model-path are empty, LlamaStash walks these caches automatically. Each bucket is independently toggleable via disable_default_cache_paths.<bucket>: true in config.yaml, or globally via --no-scan / LLAMASTASH_NO_SCAN=1.

Files anywhere under these roots that end in .gguf (and aren't .gguf.part) get parsed and added to the catalog.

Every non-interactive subcommand returns a documented exit code so agent scripts can branch on failure class. Pin against numbers, not message text — they are the public contract.

Note on sysexits.h: the numbers above are deliberately reused from <sysexits.h> for familiarity, but LlamaStash's meanings diverge from the standard ones. Scripts that import EX_NOHOST (68) expecting "host unreachable" will get our "stop failed"; EX_DATAERR (65) is reused for "daemon unreachable", not "data error". Branch on LlamaStash's table above, not the libc constants.

Linux (x86_64, aarch64), macOS (Apple Silicon, Intel), and Windows 11 (x86_64). One binary, one TUI, one CLI — the daemon's control plane is bearer-token-authed HTTP loopback on every platform, and the supervisor uses the OS's native process-group semantics (POSIX setsid + signals, Windows Job Objects + CTRL+BREAK). Windows AMD GPU detection and aarch64-pc-windows-msvc are on the roadmap.

LlamaStash hands GPU/CPU placement and context sizing to llama.cpp's --fit (on by default), so it needs a llama-server that has it.

• Minimum: build b7410 (2025-12-15), the first release carrying --fit / --fit-ctx (llama.cpp PR #16653). Older builds abort on the unknown argument the moment a model launches.
• Recommended: a recent build (b8500+, 2026). The May 2026 AMD GPU-stack updates (kernel, amdgpu firmware, ROCm) materially improved --fit on unified memory; verified on b9245.

llama.cpp has no semantic versioning, no stable branch, and no stability policy (discussion #16111) — it tags ~10-14 rolling builds a day. So the build number is a floor for flag existence, not a behaviour guarantee; LlamaStash's own pre-spawn admission control is what actually prevents out-of-memory launches regardless of build. llamastash init installs a known-good build for your hardware.

Tracked in detail in TODO.md. The headline items on deck:

• MLX backend — if the surface area lands cheaply alongside llama.cpp.
• llama.cpp version pinning — prevent silent drift / incompatibility on brew upgrade.
• MCP and LAN-exposed HTTP surfaces — Model Context Protocol, plus auth + TLS + LAN binding for the proxy. The loopback OpenAI-compatible proxy ships today (see Drop-in OpenAI + Ollama proxy); the rest of the v1 R34 deferral (Anthropic compat, MCP, network exposure) stays on the roadmap.
• Per-PID VRAM attribution via NVML's nvmlDeviceGetComputeRunningProcesses. Today the right pane shows per-model RAM + CPU%; VRAM is reported only at the host level.
• Docker-ready packaging — official images plus a documented docker run path.
• Windows AMD GPU detection — pick a probe path (DXGI / WMI / ADLX). 0.0.2 shows "GPU detection unavailable" on Windows AMD hosts.
• aarch64-pc-windows-msvc — Snapdragon X / Surface Pro coverage. Deferred from 0.0.2.

Bug reports, design discussion, and PRs welcome. Start with CONTRIBUTING.md.

Multiple AI Coding Harnesses and LLMs were heavily used to create this project.

MIT © Deepu K Sasidharan

• The Software shall be used for Good, not Evil.
• This software shall not be used for any military purposes including intelligence agencies.

• kdash — Kubernetes dashboard TUI by the same author. LlamaStash borrows its engineering and release scaffolding from kdash: the org layout (llamastash/llamastash, llamastash/homebrew-llamastash, llamastash/llamastash.github.io), the brew-tap structure, the cli.rs subdomain setup, and the release-on-tag workflow shape. The product itself is independent.
• jwt-ui — JWT decoder / encoder TUI by the same author.

If LlamaStash is useful to you, a star helps other people find it.