Soup

Fine-tune and post-train LLMs in one command. No SSH, no config hell.

Quick Start · Config · Docs · Commands · Models

This is a fork of MakazhanAlpamys/Soup — all credit for the original design, the layer-streaming approach, and the core CLI goes to the upstream project and its contributors. This fork adds a Web UI, RAM-prefetch streaming, additional quantization strategies (AWQ/GPTQ/k-quants/i-quants), an optional compression pipeline (activation scan → merge/SVD compress → distill, all opt-in and declared in the training YAML), multi-dataset training/validation/calibration support, and a pre-training data-verification gate. See README-FORK.md for the full list of fork-specific changes and how they map onto upstream, and please direct anything not specific to this fork's own changes (bug reports against core training behavior, feature requests unrelated to the items above) to the upstream repository instead — this fork tracks upstream but isn't a substitute for it.

Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done. It also ships a Web UI and a set of optional model-compression tools (neuron-importance ranking, similar-neuron merging, SVD compression, a bridge into distillation) on top of the core CLI — all opt-in, all covered below.

git clone https://github.com/serpis172/Soup.git
cd Soup
pip install -e ".[train]"   # add [train] to fine-tune; bare -e "." is the light CLI
soup init --template chat
soup train

Fine-tune an 8B model on a 4 GB laptop GPU. Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one decoder layer at a time. Measured on an RTX 3050 Laptop 4 GB: Llama-3.1-8B-Instruct + NF4 at 119.6 tok/s, 3.32 GB peak — bit-exact against a normal resident run, and reproduced independently on an H100 at 113.00 tok/s in the same 3.32 GB. (The tok/s figure was measured on v0.72.2, before the v0.73.0 correctness repair that cost −4.8% at 32B; it has not been re-run on a 4 GB card since.) Opt-in (stream_layers: true) and still BETA — how it works · all measurements · check it yourself on a free Colab T4 (caps the process to 4 GB, then asserts a streamed model is bit-identical to a normal one)

_{Llama-3.1-8B-Instruct + NF4, LoRA, batch 1, seq 512 on an RTX 3050 Laptop 4 GB — 3.32 GB peak, 119.6 tok/s. Full video (90s)}

Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.

• Zero SSH. Never SSH into a broken GPU box again.
• One config. A simple YAML file is all you need.
• Auto everything. Batch size, GPU detection, quantization — handled.
• Works locally. Train on your own GPU with QLoRA. No cloud required.

v0.73.3 — every pull request in this release came from someone other than the maintainer. All 24 of them, from eight people, five of whom appear here for the first time. What they found is the interesting part: four separate flags that were validated, documented, and then read by nothing.

• Assistant-only masking trained on zero tokens, with a normal loss curve. A tokenizer returning BatchEncoding — which is not a dict — slipped past the guard, so the label mask was built from the mapping's key strings. No exception, no warning, a loss curve that looks like training. Found by reading the type, not by hitting the bug.
• On Apple Silicon, quantization: 4bit was silently rewritten to none. detect_device() did not know MLX, so every run reported "CPU (no GPU detected)" and quietly downgraded. The label was never the harm; the quantization decision is now explicit and testable instead of hidden inside a 900-line function.
• soup train --no-reexec printed a launch command with your own flags missing — follow it literally and you trained without --fsdp, and the run succeeded, so nothing pointed back at the hint. Two hand-maintained copies of "what the user typed"; the printed one is deleted, and the hint now derives from the argv that actually launches the run.
• training.bnb_4bit_use_double_quant was read by nothing. Every 4-bit path hardcoded True, so setting it to false changed your config fingerprint and nothing else. Fixing it correctly also meant not defaulting the field: a plain True breaks round-tripping for 21 of 173 shipped configs.
• On Windows, a process that genuinely exits with code 259 read as alive forever, because that is also STILL_ACTIVE. It defeated run reconciliation and could wedge the MCP execution cap shut with no error an operator could act on.
• New: soup mcp serve --allow-execute runs a planned training or export behind a single-use, server-generated confirmation token — no command, no argv, no client-supplied environment — with the config snapshotted at plan time and protected paths digested by content, so a model cannot be swapped between planning and running.

The measurement record for the earlier VRAM work, published as written — including the three readings withdrawn during it — is benchmarks/gate-v0.73.1-measured-vram-fit.md.

# soup.yaml — then just `soup train --config soup.yaml`
training:
  stream_layers: true      # base streams out of VRAM; only the adapter trains
  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
  batch_size: 4            # bigger batches amortise the weight read
  stream_source: auto      # RAM when it fits, NVMe disk when it does not
  seed: 1234               # new in v0.73.0

Python 3.10–3.12 only. v0.73.0 adds the upper bound that was missing: on 3.13+, pip used to resolve untested PyTorch wheels that crash in the native extension before Soup runs at all.

A Web UI and a set of optional model-compression tools sit on top of the core CLI. Everything below is opt-in — plain soup train on the CLI works exactly as described above whether or not you touch any of this.

chmod +x start_ui.sh
./start_ui.sh              # first run: builds the image (10-20 min, compiles llama.cpp)
                            # later runs: only rebuilds layers that actually changed
./start_ui.sh --rebuild    # force a clean image if you suspect it's stale/corrupted
./start_ui.sh --skip-build # relaunch the existing image, no docker build at all

A FastAPI backend (soup_cli/ui/app.py) serves the dashboard — the same one soup ui runs outside Docker — with a Bearer token gating every action that starts training, downloads a model, or writes a checkpoint (printed to the console on launch). Prerequisites: Docker with NVIDIA Container Toolkit, ~16 GB RAM, an NVIDIA GPU with ≥4 GB VRAM for the Docker path (the plain soup ui CLI path has no GPU requirement of its own — it shells out to soup train, which has the normal requirements below).

Pages: Dashboard (runs, live CPU/RAM/GPU, a doctor-style health banner), New Training (template picker + YAML editor, plus a Streaming & Quantization panel — RAM prefetch slider, quantization format/bit picker, with a live diff of what changed), Data, Chat, Tool Outputs, Model Hub (search + download models/datasets/benchmarks from Hugging Face, with the Hub's own filters), and Compress (below). Background jobs (downloads, compress scans) toast a notification on completion even if you've navigated away.

training:
  stream_layers: true
  ram_cache_gb: 8         # 0 = disabled

A single background thread reads upcoming decoder layers ahead of need into pinned host RAM, direction-aware (it follows the forward-then-backward zigzag gradient checkpointing already produces). Not an LRU cache — that turned out to be the wrong model for a fully deterministic access pattern; see soup_cli/memory/layer_cache.py for why. The requested GB is clamped to a safe fraction of actually available system RAM (via psutil), so a value that doesn't fit gets reduced with a printed warning instead of feeding the OOM killer.

training:
  custom_quant_strategy: awq   # none | awq | gptq | k-quants | i-quants | qat
  custom_quant_detail: "4"     # bits for awq/gptq; a GGUF type name for k/i-quants

AWQ/GPTQ/GGUF are post-training formats — they need the trained checkpoint, not the base model, so setting this just prints the exact soup export command to run once training finishes, rather than pretending to quantize a model that hasn't been trained yet. QAT points at the existing BitNet trainer (soup train --trainer bitnet) instead of a generic, unvalidated torch.ao.quantization hook. custom_quant_detail is validated against the real format registry in soup_cli/utils/gguf_quant.py — the same one the WebUI's dropdown is populated from, so the two can't drift apart.

None of this runs automatically, and nothing writes a checkpoint without --apply.

• soup compress importance --model <path> — ranks output neurons by weight magnitude (default, streamed, no data needed) or --metric wanda (activation-weighted, needs the model loaded + --calibration-data, catches neurons whose large weights cancel out on real inputs).
• soup compress neurons --model <path> [--apply --output-dir <dir>] — finds MLP/FFN intermediate neurons that are near-duplicates (gate_proj and up_proj rows both highly similar — that joint condition is what makes the merge safe without calibration data) and merges them, folding the dropped neuron's output contribution into the kept one's. --eval-data compares perplexity before/after on real text as part of the same run.
• soup compress svd --model <path> --mode denoise|factorize [--apply --output-dir <dir>] — denoise replaces a matrix with its best low-rank reconstruction at the same shape (always loads with stock transformers, doesn't shrink the file, removes low-signal noise — the same idea soup spectrum already uses for SNR-based LoRA targeting, applied to compression). factorize splits a matrix into two smaller ones for a genuine size reduction; the result needs custom loading code (svd_manifest.json documents exactly how) and is not loadable by plain AutoModelForCausalLM.from_pretrained.
• soup compress distill-config --student <compressed> --teacher <original> — Soup already has a full token/sequence-level distillation trainer (task: distill, soup_cli/trainer/distill.py); this generates a ready-to-run config pointing it at your compressed model as the student and the pre-compression original as the teacher, for a quality-recovery pass. Doesn't reimplement training — just wires the existing trainer to the natural next step after pruning/merging.

Scope, stated plainly: MLP/FFN neurons only (attention-head merging has more structure — per-head grouping, GQA, RoPE — and isn't attempted here); merge quality is bounded, not guaranteed (measured on synthetic weights: ~0.3% median output error at 0.998 similarity, ~1.3% at 0.97 — --eval-data lets you check on your own text instead of trusting that number).

soup reward synth references.jsonl -o reward.py --output-report calib.json

soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error

Full history: CHANGELOG.md.

git clone <this-repository-url>
cd soup

# Light core: CLI + config + data tools, no PyTorch
pip install -e "."

# Add the training stack (torch, transformers, peft, trl, datasets, …)
pip install -e ".[train]"

# Everything (train + serve + ui + data) in one shot
pip install -e ".[all]"

The full extras table (fast, mlx, serve, eval, ui, vision, audio, …) lives in docs/models.md.

Double quotes, not single. ".[train]" is the only spelling that works in every shell — cmd.exe, PowerShell, bash and zsh. If you copied '.[train]' from an older tutorial and pip rejected it, that is the reason: why, and the exact error.

soup init, soup data …, and the other data/inspection commands work on the light install. Fine-tuning (soup train) needs the [train] extra.

soup init                       # interactive wizard
soup init --template chat       # or start from a template

Templates: chat, code, tool-calling, medical, reasoning, vision, kto, orpo, simpo, ipo, bco, rlhf, pretrain, moe, longcontext, embedding, audio.

soup train --config soup.yaml                 # LoRA, quantization, batching — all handled
soup chat  --model ./output                    # talk to your model
soup push  --model ./output --repo you/my-model

soup merge  --adapter ./output                              # merge LoRA into the base
soup export --model ./output --format gguf --quant q4_k_m   # GGUF for Ollama / llama.cpp

More export targets (ONNX, TensorRT, AWQ, GPTQ, BitNet) and deployment options live in docs/serving-and-export.md.

A complete soup.yaml:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
# backend: unsloth  # 2-5x faster, pip install -e ".[fast]"

data:
  train: ./data/train.jsonl
  format: alpaca
  val_split: 0.1

training:
  epochs: 3
  lr: 2e-5
  batch_size: auto
  lora:
    r: 64
    alpha: 16
  quantization: 4bit

output: ./output

config/schema.py is the single source of truth for every field. Advanced data, training, and PEFT options are documented under Documentation.

The full feature reference lives in docs/. Start here:

Alpaca, ShareGPT, ChatML, preference pairs (DPO / ORPO / SimPO / IPO / KTO), vision, audio, ASR, plaintext, embedding, RAFT and more — all auto-detected from JSONL, JSON, CSV, Parquet or TXT, so in most cases you point data.train at a file and nothing else changes. Schemas with a worked example per format, plus the data pipeline (remote URIs, streaming, sharding, interleaving, vocab expansion, document ingestion), are in docs/data.md.

soup train  --config soup.yaml        # train (SFT/DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/...)
soup infer  --model ./output --input prompts.jsonl   # batch inference
soup chat   --model ./output          # interactive chat
soup serve  --model ./output          # OpenAI-compatible API server
soup merge  --adapter ./output        # merge LoRA into the base model
soup export --model ./output --format gguf           # export for deployment
soup eval   benchmark --model ./output               # evaluate
soup data   inspect ./data/train.jsonl               # dataset stats
soup recipes list                     # 100+ ready-made model recipes
soup autopilot --model <id> --data d.jsonl --goal chat  # zero-config
soup doctor                           # check GPU / deps / environment

The complete command list is in docs/commands.md.

Soup works with any text-generation model on the HuggingFace Hub — if it loads with AutoModelForCausalLM, it works, zero config changes. Llama 3.x/4, Qwen 2.5/3, Gemma 3, Mistral, Mixtral, DeepSeek R1/V3, Phi-4, and 100+ others ship as ready-made recipes (soup recipes list).

Full model + vision tables and the optional-extras matrix are in docs/models.md.

Run Soup without installing CUDA or PyTorch locally — build the image from this repo (see Dockerfile.ui for the Web UI image, ./start_ui.sh to build and run it in one step):

docker build -f Dockerfile.ui -t soup .
docker run --gpus all -v $(pwd):/workspace soup train --config soup.yaml
docker compose up   # or build locally

• Python 3.10, 3.11 or 3.12 (those are the versions CI tests; 3.13+ is not supported yet because the PyTorch stack has not been validated there)
• GPU with CUDA (recommended), Apple Silicon (MPS), or CPU (experimental — very slow)
• 8 GB+ VRAM for 7B models with QLoRA

All training tasks run on CPU for testing (quantization auto-disabled). Optional extras (train, all, fast, vision, qat, serve, serve-fast, ui, eval, deepspeed, liger, mlx, onnx, tensorrt, …) are listed in docs/models.md.

soup doctor    # GPU, system resources, dependencies, and version in one place

• ImportError: DLL load failed while importing _C (Windows) — reinstall PyTorch for your CUDA version: pip install torch --index-url https://download.pytorch.org/whl/cu121.
• soup version ≠ pip show soup-cli — multiple Python installs; use a virtualenv.

git clone <this-repository-url>
cd soup
pip install -e ".[dev]"

ruff check src/soup_cli/ tests/    # lint
pytest tests/ -v                   # unit tests (fast, no GPU)
pytest tests/ -m smoke -v          # smoke tests (downloads a tiny model, trains)

pre-commit install                 # optional: ruff lint+format on commit

See CONTRIBUTING.md for the full workflow and SECURITY.md to report a vulnerability.

Soup is Apache-2.0 and free — and stays that way. See CONTRIBUTING.md for the full workflow and SECURITY.md to report a vulnerability. Bugs and feature requests belong in the issue tracker; the Code of Conduct applies throughout.

Apache-2.0. Copyright © the Soup contributors.