Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/gpu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ on:
workflow_dispatch:
inputs:
run_recipe:
description: "Also run the full 16 GB recipe (slow, ~10 minutes)"
description: "Also run the full single-GPU recipe (slow, ~10 minutes)"
type: boolean
default: false

Expand Down Expand Up @@ -46,7 +46,7 @@ jobs:
- name: GPU tests
run: pytest -q -m gpu -ra

- name: Full 16 GB recipe
- name: Full single-GPU recipe
if: ${{ inputs.run_recipe }}
env:
MINIVERL_LOG_LEVEL: INFO
Expand Down
28 changes: 28 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,34 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

## [0.2.2] - 2026-07-29

Single-GPU portability and presentation release.

### Added

- A hardware-portability guide for personal NVIDIA GPUs, including honest
starting points for 8–12 GiB, 16–24 GiB and 24–32+ GiB cards, OOM controls,
and a reproducible hardware-result contribution path.
- A prominent PyPI destination in both READMEs and package metadata.
- Visual regression assertions that keep benchmark grid lines below axis labels
and preserve the dark generated figure.

### Changed

- The supported Qwen3 recipe now uses model-agnostic run metadata and
`dtype: auto`, selecting bf16 when available and fp16 on older CUDA cards
such as Titan V. The pinned models, adapter, objective and budgets are
unchanged.
- The repository is positioned as a personal single-GPU training stack rather
than a 16 GiB-specific implementation. RTX 4080 numbers remain explicitly
labeled as the only measured GPU evidence.
- The banner and data-bound protocol benchmark figure use a new dark visual
system. Axis grids no longer cross tick labels, and protocol-incompatible
0% controls are rendered as diagnostic states rather than zero-length bars.
- GPU workflow language now names the portable single-GPU recipe instead of a
particular VRAM tier.

## [0.2.1] - 2026-07-29

Correctness, lifecycle safety and reproducibility release.
Expand Down
10 changes: 6 additions & 4 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ cff-version: 1.2.0
title: "miniVERL: On-policy distillation for tool-using agents on one GPU"
message: "If you use miniVERL in your work, please cite it as below."
type: software
version: 0.2.1
version: 0.2.2
date-released: 2026-07-29
license: Apache-2.0
repository-code: "https://github.com/DaoyuanLi2816/mini-verl"
Expand All @@ -18,8 +18,9 @@ abstract: >-
coarse-graining, enforces per-token provenance so that tool output can never
become a training label, and stores teacher targets in a versioned,
checksummed, pickle-free cache with policy-version enforcement. It is
designed and measured for a single 16 GB consumer GPU and requires neither
Ray nor a cluster.
designed for one personal CUDA GPU, automatically selects bf16 or fp16, and
requires neither Ray nor a cluster. Published performance is measured on one
RTX 4080; other GPU models use the same code path but remain unmeasured.
authors:
- family-names: Li
given-names: Daoyuan
Expand All @@ -32,6 +33,7 @@ keywords:
- agents
- QLoRA
- consumer GPU
- single GPU
- post-training
- reproducibility
references:
Expand Down Expand Up @@ -96,4 +98,4 @@ references:
url: "https://arxiv.org/abs/2603.07079"
notes: >-
Motivates recording per-token teacher entropy. Entropy-aware divergence
mixing is a roadmap item and is not implemented in miniVERL v0.2.1.
mixing is a roadmap item and is not implemented in miniVERL v0.2.2.
15 changes: 14 additions & 1 deletion PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,20 @@ and what it printed.

Last updated: 2026-07-29.

## v0.2.1 correctness release status
## v0.2.2 single-GPU portability release candidate

| item | current state |
| --- | --- |
| integration source | `agent/single-gpu-visuals`, based on synchronized public `main` at `dcded6bd1948eb7421f89e336f611e3b0d3c0ad3` |
| version transition | source, changelog and citation metadata identify `0.2.2`; the intended annotated tag is `v0.2.2` |
| hardware scope | one NVIDIA CUDA GPU with no device-name allowlist; automatic dtype selects bf16 when supported and fp16 otherwise; exact fit remains model/budget dependent |
| measured evidence | the only published real-model GPU measurements remain the RTX 4080 runs; RTX 3070, Titan V, RTX 5090 and other cards are portable code paths, not fabricated benchmark claims |
| presentation | banner and generated benchmark SVG use a dark single-GPU visual system; grid lines begin below tick labels and diagnostic 0% controls are status pills |
| package discovery | both READMEs, project metadata and the GitHub About homepage expose `https://pypi.org/project/miniverl/` |
| frozen scientific artifact | `benchmarks/results/gpu-calc-hard-equal-update-v2.json` remains byte-identical at SHA-256 `53fc1d4d5b7adee09618d77ad62d4086ba56b78569832d6fc7c3bcd5c2695bbc` |
| publication state | authorized for direct release; publish only after local gates, pull-request CI and synchronized default-branch CI pass |

## Historical v0.2.1 correctness release status

| item | current state |
| --- | --- |
Expand Down
36 changes: 27 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,21 @@

</div>

**On-policy distillation for tool-using agents on one GPU.**
<p align="center">
<a href="https://pypi.org/project/miniverl/"><strong>PyPI package</strong></a> ·
<a href="#single-gpu-quickstart">Install &amp; train</a> ·
<a href="docs/single-gpu-guide.md">Bring your own GPU</a> ·
<a href="#measured-result-protocol-aligned-opd-matches-sft">Measured result</a>
</p>

**A compact, auditable training stack for tool-using agents on one personal GPU.**

miniVERL is a compact, auditable training lab for teaching a small language
model from its own multi-turn tool trajectories. It runs real tools, keeps
token provenance explicit, and applies teacher distributional targets only
where they belong — without Ray, a GPU cluster, or a 40 GB accelerator.
where they belong — without Ray or a GPU cluster. There is no device-name
allowlist: use the NVIDIA CUDA card you have, then choose a model pair and
sequence budget that fit it.

```bash
python -m pip install miniverl # lightweight core
Expand All @@ -44,7 +53,7 @@ validate artifacts without downloading a multi-gigabyte ML stack; use
`top-k + tail` objectives are named and reported separately.

[Run the local demo](#local-toy-demo) ·
[Train on a consumer GPU](#consumer-gpu-quickstart) ·
[Train on your GPU](#single-gpu-quickstart) ·
[Inspect the measured result](#measured-result-protocol-aligned-opd-matches-sft) ·
[Read the math](docs/math.md)

Expand All @@ -66,7 +75,7 @@ go wrong in practice, and all four are silent:
cannot run.

miniVERL makes each of those a *checked property* rather than a comment, and
keeps the whole thing on one 16 GB card.
keeps the whole lifecycle in one readable single-GPU process.

## What is implemented

Expand All @@ -78,7 +87,7 @@ keeps the whole thing on one 16 GB card.
| Compressed `top-k + tail` KL and JSD | yes; the unsmoothed coarse-graining has a proven lower-bound relationship to the exact loss |
| Privileged-context teacher with an explicit alignment map | yes |
| Frozen standard PEFT teacher adapters with provenance and competence gates | yes |
| QLoRA (NF4) student, bf16 or quantized teacher | yes, measured on an RTX 4080 |
| Single-GPU CUDA path with automatic bf16/fp16 selection | yes; model-agnostic code path, measured reference on an RTX 4080 |
| `resident` and `swap` memory strategies, `auto` resolution | yes, with an equivalence test |
| Versioned, checksummed, pickle-free teacher-target cache | yes |
| SFT / offline KD / strict OPD / explicitly labeled replay behind one trainer | yes |
Expand All @@ -105,7 +114,7 @@ keeps the whole thing on one 16 GB card.
| diagnostic control | answer-privileged, protocol-naive teacher | 0.0% | 0.0% |

The [public, immutable protocol-teacher adapter](https://huggingface.co/DaoyuanLi/mini-verl-qwen3-1.7b-protocol-teacher)
is the default in the consumer-GPU recipe. It passed an independently
is the default in the single-GPU recipe. It passed an independently
prespecified policy-competence gate before this benchmark was inspected. The
two controls intentionally remove that guarantee: their loss decreased
normally, but they taught the student an incompatible tool policy. This is a
Expand Down Expand Up @@ -200,7 +209,14 @@ The toy backend is a **machinery harness, not a capability demonstration**. Its
models are too small to solve anything beyond the `easy` split. Capability
numbers come from the GPU recipe.

## Consumer-GPU quickstart
## Single-GPU quickstart

The default recipe uses `device: auto` and `dtype: auto`: bf16-capable cards use
bf16, while older CUDA cards such as Titan V use fp16. RTX 3070, Titan V,
RTX 4080 and RTX 5090-class cards all enter the same code path; only the
RTX 4080 result is measured here. Exact fit is governed by VRAM, model sizes,
drivers and token budgets, not the card's marketing name. See the
[`single-GPU guide`](docs/single-gpu-guide.md) before changing the recipe.

```bash
git clone https://github.com/DaoyuanLi2816/mini-verl.git
Expand Down Expand Up @@ -474,8 +490,10 @@ environments, and multi-GPU. For anything at cluster scale, use verl.
The name is a nod to the problem space, not a claim of compatibility. verl is an
excellent, much larger system that also implements on-policy distillation and
multi-turn tool use — at cluster scale, with Ray. If you have a cluster, use it.
miniVERL exists for the case where you have one consumer GPU and want to read
every line of what is happening. See [`docs/comparisons.md`](docs/comparisons.md).
miniVERL exists for the case where you have one personal GPU and want to read
every line of what is happening. That can be an older 12 GiB card or a current
high-end card; the repository claims measured performance only for hardware it
actually ran. See [`docs/comparisons.md`](docs/comparisons.md).

## Citation

Expand Down
30 changes: 22 additions & 8 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,21 @@

</div>

<p align="center">
<a href="https://pypi.org/project/miniverl/"><strong>PyPI 软件包</strong></a> ·
<a href="#个人单卡快速上手">安装与训练</a> ·
<a href="docs/single-gpu-guide.md">适配你的 GPU</a> ·
<a href="#实测结果协议对齐的-opd-追平-sft">实测结果</a>
</p>

> 本文是 [README.md](README.md) 的中文翻译。英文版为准;若两者不一致,请以英文版为准并提交 issue。

**单卡上的工具调用智能体在线策略蒸馏(on-policy distillation)。**
**面向个人单卡、紧凑且可审计的工具调用智能体训练栈。**

miniVERL 是一个紧凑、可审计的训练实验室,让小型语言模型从**它自己生成的
多轮工具轨迹**中学习。它会真实执行工具、显式记录 token 来源,并且只在正确
的位置使用教师分布目标——不需要 Ray、GPU 集群或 40 GB 显存的加速卡。
的位置使用教师分布目标——不需要 Ray 或 GPU 集群。代码没有显卡型号白名单:
使用你现有的 NVIDIA CUDA 显卡,再选择能装进显存的模型组合与 token 预算。

```bash
python -m pip install miniverl # 轻量核心层
Expand All @@ -41,7 +49,7 @@ schema 与 Python API)。`train` extra 会添加 torch、Transformers 与 PEFT
- **预算真实:** 精确全词表目标与压缩的 `top-k + tail` 目标分开命名、分开报告。

[运行本地 demo](#本地玩具演示) ·
[在消费级 GPU 上训练](#消费级-gpu-快速上手) ·
[在你的 GPU 上训练](#个人单卡快速上手) ·
[查看实测结果](#实测结果协议对齐的-opd-追平-sft) ·
[阅读数学说明](docs/math.md)

Expand All @@ -54,7 +62,7 @@ schema 与 Python API)。`train` extra 会添加 torch、Transformers 与 PEFT
3. **其实并不是 on-policy。** 跨策略版本复用教师缓存,做的就是离线 KD,却仍叫它 OPD。
4. **显存放不下 logits。** `[batch, seq_len, 152k]` 的张量在消费级显卡上放不下,于是真正有意思的配置恰好都跑不了。

miniVERL 把上面每一条都变成**被代码检查的性质**,而不是注释里的一句承诺,并且整套流程只用一块 16 GB 显卡
miniVERL 把上面每一条都变成**被代码检查的性质**,而不是注释里的一句承诺,并把整个生命周期放进一个可读的单卡进程

## 已实现的能力

Expand All @@ -66,7 +74,7 @@ miniVERL 把上面每一条都变成**被代码检查的性质**,而不是注
| 压缩的 `top-k + tail` KL / JSD | 支持;未平滑粗粒化与精确散度的下界关系有严格证明 |
| 特权上下文教师模式,带显式对齐表 | 支持 |
| 标准冻结 PEFT 教师适配器,带来源记录与能力门禁 | 支持 |
| QLoRA(NF4)学生,bf16 或量化教师 | 支持,已在 RTX 4080 上实测 |
| 自动选择 bf16/fp16 的单卡 CUDA 路径 | 支持;代码路径不绑定型号,实测参考为 RTX 4080 |
| `resident` / `swap` 显存策略与 `auto` 解析 | 支持,并有等价性测试 |
| 带版本号与校验和、完全不用 pickle 的教师目标缓存 | 支持 |
| SFT / 离线 KD / 严格 OPD / 显式标注 replay 统一在一个 trainer 中 | 支持 |
Expand All @@ -92,7 +100,7 @@ miniVERL 把上面每一条都变成**被代码检查的性质**,而不是注
| 诊断对照 | 获知答案但不懂协议的教师 | 0.0% | 0.0% |

[公开且固定版本的协议教师适配器](https://huggingface.co/DaoyuanLi/mini-verl-qwen3-1.7b-protocol-teacher)
现在是消费级 GPU 配方的默认教师。它在下游 benchmark 被查看之前,已经通过
现在是单卡配方的默认教师。它在下游 benchmark 被查看之前,已经通过
预先指定、独立评估的策略能力门槛。两个负对照故意去掉了这个保证:loss 正常
下降,但教师把不兼容的工具策略教给了学生。这是实测的教师能力问题,不是
trainer 崩溃。
Expand Down Expand Up @@ -165,7 +173,13 @@ tokens by span type (only assistant_* can enter the loss)

玩具后端是**机制验证台,不是能力展示**。它的模型太小,除了 `easy` 难度之外什么都做不了。能力数字来自 GPU 配方。

## 消费级 GPU 快速上手
## 个人单卡快速上手

默认配方使用 `device: auto` 与 `dtype: auto`:支持 bf16 的显卡自动使用 bf16,
Titan V 等较老 CUDA 显卡自动使用 fp16。RTX 3070、Titan V、RTX 4080 和
RTX 5090 级别显卡都走同一条代码路径;本仓库目前只有 RTX 4080 的实测结果。
能否装下取决于显存、模型大小、驱动和 token 预算,而不是显卡的商品名。修改配方前请阅读
[`单卡适配指南`](docs/single-gpu-guide.md)。

```bash
git clone https://github.com/DaoyuanLi2816/mini-verl.git
Expand Down Expand Up @@ -299,7 +313,7 @@ print(result.run_dir, result.global_step, result.eval["success_rate"])

> miniVERL 是一个独立项目,与 verl 项目、字节跳动(ByteDance)或火山引擎(Volcano Engine)没有隶属关系,也未获得其背书。它**不是** verl 的直接替代品。

这个名字只是对问题领域的致意,不代表任何兼容性声明。verl 是一个优秀得多、规模也大得多的系统,它同样实现了在线策略蒸馏和多轮工具调用——只不过是在集群规模上,依赖 Ray。如果你有集群,请用它。miniVERL 面向的是「只有一块消费级显卡、并且希望把每一行发生的事都读懂」的场景。对比见 [`docs/comparisons.md`](docs/comparisons.md)。
这个名字只是对问题领域的致意,不代表任何兼容性声明。verl 是一个优秀得多、规模也大得多的系统,它同样实现了在线策略蒸馏和多轮工具调用——只不过是在集群规模上,依赖 Ray。如果你有集群,请用它。miniVERL 面向的是「只有一块个人显卡、并且希望把每一行发生的事都读懂」的场景:它可以是较老的 12 GiB 显卡,也可以是当前的高端显卡;仓库只对实际跑过的硬件声明实测性能。对比见 [`docs/comparisons.md`](docs/comparisons.md)。

## 引用与许可证

Expand Down
Loading