Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 21 additions & 5 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,29 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

### verl v0.8 OPD config compiler
## [0.8.0] - 2026-08-12

### Single-GPU verl v0.8 OPD runtime

- Added the typed `verl-opd-v0.8-single-gpu-v1` configuration profile and an
offline `bridge compile-opd` command. Resolved YAML and repeatable dotted
overrides compile into deterministic field-by-field compatibility reports.
offline compiler plus weight-free `plan` and executable `run` commands.
Resolved YAML and repeatable dotted overrides compile into deterministic
field-by-field compatibility reports.
- Unsupported policy-gradient OPD, task-reward mixtures, KL penalties,
multi-generation, multi-teacher and distributed dimensions fail closed.
Engine/resource fields are labelled as local reinterpretations rather than
upstream-exact behavior; the config-only command loads no model weights.
upstream-exact behavior.
- Added first-class bounded verl Parquet prompts, padded local-HF rollout,
exact response-only selection, current-policy/teacher/cache bindings and the
pinned verl `forward_kl_topk` loss with token-mean scalar/metric/gradient
conformance.
- Added a packaged Qwen3-0.6B/1.7B NF4 recipe and standard PEFT export. One RTX
4080 runtime-conformance run completed its first update in 12.0224 seconds at
3.1758 GiB peak reserved VRAM. No alignment-quality comparison was run.
- Added OPD v2 import/export. Imports publish a canonical prompt profile without
inventing an environment or reward; exports preserve PEFT, teacher identity,
Parquet bytes and pure OPD overrides without a reward scaffold. Missing base
snapshots and teacher-adapter materialization remain explicit launch blockers.

## [0.7.1] - 2026-08-11

Expand Down Expand Up @@ -833,7 +847,9 @@ Same-tokenizer only; one trajectory per forward pass; `swap` unavailable for
quantized models; only Qwen3 and Qwen2 architectures tested; single-seed GPU
results. The full list is in `docs/limitations.md`.

[Unreleased]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.7.1...HEAD
[Unreleased]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.8.0...HEAD
[0.8.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.7.1...v0.8.0
[0.7.1]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.7.0...v0.7.1
[0.7.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.3...v0.7.0
[0.6.3]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.2...v0.6.3
[0.6.2]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.1...v0.6.2
Expand Down
15 changes: 9 additions & 6 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,17 @@ cff-version: 1.2.0
title: "miniVERL: Auditable single-GPU alignment and distillation runtime"
message: "If you use miniVERL in your work, please cite it as below."
type: software
version: 0.7.1
date-released: 2026-08-11
version: 0.8.0
date-released: 2026-08-12
license: Apache-2.0
repository-code: "https://github.com/DaoyuanLi2816/mini-verl"
url: "https://github.com/DaoyuanLi2816/mini-verl"
abstract: >-
miniVERL is an auditable single-GPU alignment and distillation runtime with a
bounded artifact bridge to one pinned verl profile. Its native core is
miniVERL is a local single-GPU runtime for a documented subset of verl v0.8
on-policy distillation. It consumes typed verl-shaped configuration and
Parquet prompts, executes actor rollout, teacher scoring and actor update in
one inspectable process, and exports standard PEFT, Parquet and configuration
artifacts through a fail-closed pinned bridge. Its native core also supports
multi-turn, tool-aware on-policy distillation: a student language
model samples its own tool-using trajectories against deterministic local
environments, a teacher scores exactly the states the student visited, and
Expand All @@ -28,8 +31,8 @@ abstract: >-
published with their negative and mixed results intact. A fail-closed
artifact bridge exchanges configs, LoRA adapters, tokenizers and Parquet
datasets with one pinned verl profile, verifying what it can and refusing to
imply the rest: it does not launch distributed jobs and does not establish
algorithmic parity with PPO. miniVERL is designed for one personal CUDA GPU,
imply the rest: it does not launch distributed jobs or establish broad
algorithmic parity. miniVERL is designed for one personal CUDA GPU,
automatically selects bf16 or fp16, and requires neither Ray nor a cluster.
Published performance is measured on one RTX 4080; other GPU models use the
same code path but remain unmeasured. The v0.7 external-alignment study
Expand Down
24 changes: 22 additions & 2 deletions PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ Living build log for **miniVERL** (`mini-verl` / `miniverl` / CLI `miniverl`).
A checkbox is not evidence: every completed item names the command that was run
and what it printed.

Last updated: 2026-08-11.
Last updated: 2026-08-12.

Canonical release state: stable `v0.7.1` (`830a4ca5d873bce4cdcc7c43a44d827b096e8c0c`), development `0.8.0.dev0`.
Canonical release state: releasing `v0.8.0`.
Every public version claim is generated from `release-state.yaml` and gated by
`python scripts/release_state.py --check`.

Expand Down Expand Up @@ -84,6 +84,26 @@ from construction start. The standard PEFT adapter load passed. This is a
systems result only; no alignment endpoint or method comparison ran. The
machine-readable record is `benchmarks/results/rtx4080-verl-opd-runtime-v1.json`.

## v0.8.0 single-GPU verl OPD pivot — PR E

`import-verl --profile verl-opd-v0.8-single-gpu-v1 --config ...` now publishes
a transactional, canonical OPD profile plus a field-by-field report; it uses
the same typed compiler as `plan`/`run`, consumes Parquet paths and needs no
ToolEnvironment or reward. The legacy environment-profile importer remains
available under its original profile name.

`export-verl` recognizes a completed compatible OPD run and emits the standard
student PEFT adapter, exact student/teacher identities, byte-preserved Parquet,
source config, compiled plan and pure `forward_kl_topk` overrides. It emits no
reward scaffold. Student/teacher/data loadability, upstream parse/tiny-smoke,
launchability and distributed execution are independent fields. Missing base
snapshots and teacher-adapter materialization keep new bundles fail-closed and
`launchable: false`; no distributed command or execution claim is generated.

The generated `docs/generated/verl-opd-v0.8-compatibility.json` binds 72 source
fields to the pinned compiler fixture. Upstream parse conformance is recomputed
against verl `7aed6b23`; it does not imply a model launch or distributed job.

## v0.7.1 Product correction — RELEASE CANDIDATE

Branch `v0.7.1-product-correction` starts from synchronized main
Expand Down
105 changes: 49 additions & 56 deletions PYPI.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<p align="center">
<img src="https://raw.githubusercontent.com/DaoyuanLi2816/mini-verl/main/docs/banner.svg" alt="miniVERL — single-GPU LLM post-training" width="880">
<img src="https://raw.githubusercontent.com/DaoyuanLi2816/mini-verl/v0.8.0/docs/banner.svg" alt="miniVERL — single-GPU LLM post-training" width="880">
</p>

<div align="center">
Expand All @@ -8,41 +8,41 @@
[![Build](https://github.com/DaoyuanLi2816/mini-verl/actions/workflows/build.yml/badge.svg)](https://github.com/DaoyuanLi2816/mini-verl/actions/workflows/build.yml)
[![PyPI](https://img.shields.io/pypi/v/miniverl.svg)](https://pypi.org/project/miniverl/)
[![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/DaoyuanLi2816/mini-verl/blob/main/LICENSE)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/LICENSE)

</div>

<p align="center">
<a href="https://pypi.org/project/miniverl/"><strong>PyPI</strong></a> ·
<a href="https://daoyuanli2816.github.io/mini-verl/"><strong>Stable docs</strong></a> ·
<a href="https://daoyuanli2816.github.io/mini-verl/dev/">Development docs</a> ·
<a href="https://github.com/DaoyuanLi2816/mini-verl/blob/main/README.zh-CN.md">中文</a>
<a href="https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/README.zh-CN.md">中文</a>
</p>

**miniVERL is a local, inspectable single-GPU alignment and distillation
runtime.** It runs native SFT, DPO, KD and strict OPD recipes, preserves
assistant-only loss masks and policy-version provenance, and exchanges standard
HF/PEFT/Parquet artifacts through a fail-closed bridge to one pinned verl
profile.
**Run a documented subset of verl-style on-policy distillation on one consumer
GPU.** miniVERL accepts a typed verl v0.8 OPD profile and Parquet prompts,
executes actor rollout → teacher scoring → actor update locally, and exports
standard PEFT/Parquet/config artifacts for scale-out. Native SFT, DPO, KD and
tool-agent recipes remain available.

PyPI `v0.7.1` is stable; `main` is development. miniVERL is independent from
PyPI `v0.8.0` is stable; `main` is development. miniVERL is independent from
verl. It does not claim arbitrary verl YAML execution, distributed execution,
or full algorithmic compatibility.

## Install and verify in about a minute
## Pip-only OPD quickstart

```bash
python -m pip install "miniverl[train]"
miniverl doctor
miniverl demo --fast --output runs/quickstart
miniverl inspect runs/quickstart/trajectories.jsonl
miniverl evidence validate alignment-external-v1
miniverl data sample --format verl-parquet --out prompts.parquet
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config builtin:qwen3-0.6b-1.7b-opd \
--set 'data.train_files=["prompts.parquet"]'
miniverl run --profile verl-opd-v0.8-single-gpu-v1 --config builtin:qwen3-0.6b-1.7b-opd \
--set 'data.train_files=["prompts.parquet"]' --dry-run
```

The deterministic demo downloads no model and produces typed trajectories, a
checksummed teacher cache, manifest and report. The evidence command reads
self-contained package data; it works from a wheel without a Git checkout.
For schemas and inspection without the ML stack, install `miniverl` alone.
The sample, plan and dry run need no Git checkout; planning loads no weights.
Remove `--dry-run` on one CUDA GPU to execute the pinned Qwen3-0.6B/1.7B NF4
recipe and produce a loadable PEFT adapter. [Follow the OPD quickstart](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/opd-quickstart.md).

## Supported hardware and runtime boundary

Expand All @@ -51,43 +51,36 @@ device-name agnostic, but fit depends on model pair, context, kernels and VRAM.
Install the matching CUDA-enabled PyTorch build first, then
`miniverl[train,cuda]`; that extra does not select a CUDA PyTorch wheel.
Ray, FSDP, Megatron, PPO, GRPO and distributed launch are outside the runtime.
See the [single-GPU guide](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/single-gpu-guide.md).
See the [single-GPU guide](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/single-gpu-guide.md).

## verl compatibility summary

The bridge targets official verl `v0.8.0` at commit `7aed6b23`. Its verified
boundary is checksummed standard artifacts plus pinned config-parse and
model/data-load smoke—not native checkpoint parity or a completed verl job.
Imports fail closed when dataset, environment, teacher, objective or schedule
semantics are unresolved; they never substitute calculator tasks or invent an
unqualified teacher.
The executable profile targets official verl `v0.8.0` at commit `7aed6b23` and
supports one actor, one teacher, `n=1`, pure GKD `forward_kl_topk`, token-mean
aggregation, LoRA/QLoRA and no reward/KL penalty. PG OPD, task-reward mixtures,
multi-teacher, multimodal and distributed fields fail closed.

Current exports remain `launchable: false`: the base snapshot is absent, the
reward scaffold fails closed and required mappings remain placeholders. The
entry point is `launch.template.sh`; readiness, parse/load evidence,
launchability, distributed execution and semantic parity are separate facts.
[Read the bridge contract](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-bridge.md).
Compatible OPD exports contain no reward scaffold. They preserve student and
teacher identities, Parquet bytes and OPD overrides, but remain
`launchable: false` until exact base snapshots are materialized. Parse status,
artifact loadability, launchability and distributed execution are separate.
[Read the bridge contract](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/verl-bridge.md).

## One measured systems result
## Measured RTX 4080 runtime

On one RTX 4080 with Qwen3-0.6B and eight fixed SQLite trajectories, physical
batch 4 increased dual-model update throughput from 2.369 to 3.866
trajectories/s. Shared-backbone batch 4 used 2.227 GiB peak reserved memory
versus 3.035 GiB for dual model while running 10.1% slower. All 12
preregistered equivalence comparisons passed. This is one workload on one
machine, not a promise for other GPUs.

![Measured throughput and reserved VRAM for dual-model and shared-backbone runtime cells](https://raw.githubusercontent.com/DaoyuanLi2816/mini-verl/main/docs/consumer-runtime-v1-pareto.svg)

[Consumer Runtime v1 methods and caveats](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/consumer-runtime-v1.md)
The packaged Qwen3-0.6B/1.7B recipe completed two 16-token rollouts and one OPD
update with **3.1758 GiB peak reserved VRAM**; the first update completed in
**12.0224 s** and the standard PEFT adapter reloaded successfully. This proves
one runtime/artifact path only—no alignment-quality endpoint or method
comparison ran. [Exact recipe, timings and hashes](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/opd-quickstart.md).

## Three paths

| Path | Start with | Concrete artifact | Next |
| --- | --- | --- | --- |
| **Align** — use SFT, DPO, KD or OPD only when pilot evidence supports the cost | `miniverl pilot recipes/alignment_policy_conditioned_qwen.yaml` | `alignment-card.json` | [Alignment Lab](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/alignment-lab/alignment-lab-v1.md) |
| **Distill locally** — strict OPD, shared backbones and padded updates on one CUDA GPU | `miniverl train recipes/qwen_consumer_gpu_shared.yaml --dry-run` | resolved config and revision-pinned PEFT adapter | [Bring your own GPU](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/single-gpu-guide.md) |
| **Scale out** — convert Parquet, export standard artifacts and inspect the unsupported boundary | `miniverl bridge doctor scaleout-bundle` | `provenance/compatibility-report.json` | [Verified artifact bridge](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-bridge.md) |
| **Run OPD locally** | `miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml` | compiled plan, trajectories, targets and PEFT adapter | [Plan and run](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/opd-quickstart.md) |
| **Bring a verl config** | `miniverl import-verl --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml --out local-opd.yaml` | field report plus round-trippable profile | [Compatibility](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/compatibility.md) |
| **Move data and artifacts** | `miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout` | Parquet + PEFT + OPD override bundle | [Bridge contract](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/verl-bridge.md) |

## Research notes and preserved negative evidence

Expand All @@ -108,7 +101,7 @@ miniverl pilot --builtin-study alignment-external-v1 --json
The result is `do_not_continue_this_study` and `insufficient_evidence`, not a
recommendation among SFT/DPO/KD/OPD. Granite Guardian values are unqualified
selection diagnostics; Granite, PairRM and teacher qualification and the
reserved final test did not run. [Study and limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/alignment-external/alignment-external-v1.md).
reserved final test did not run. [Study and limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/alignment-external/alignment-external-v1.md).

### Earlier measured alignment case study

Expand All @@ -118,17 +111,17 @@ ceiling; continued SFT and both OPD variants retained measured regressions.
The two sandbox safety checks tied at zero while utility still regressed.
IFEval, XSTest, HarmBench and RewardBench were not executed, and “preference
win rate” is a deterministic Minipolicy paired outcome, not human preference.
[Seed-level evidence](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/alignment-lab/alignment-lab-v1.md).
[Seed-level evidence](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/alignment-lab/alignment-lab-v1.md).

- [RecoveryBench v1](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/recoverybench/recoverybench-v1.md): frozen-student KD
- [RecoveryBench v1](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/recoverybench/recoverybench-v1.md): frozen-student KD
beat slower fresh-state OPD on the preregistered primary view; the verifier
gate remained `insufficient_evidence`.
- [Calculator benchmark](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/benchmarking.md): both negative controls completed
- [Calculator benchmark](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/benchmarking.md): both negative controls completed
normally at 0%; the ambiguous historical protocol-v1 prompt prevents
attributing failure solely to intrinsic teacher behavior.
- [Limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/limitations.md), [math](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/math.md),
[reproducibility](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/reproducibility.md) and
[compatibility policy](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/compatibility.md).
- [Limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/limitations.md), [math](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/math.md),
[reproducibility](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/reproducibility.md) and
[compatibility policy](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/compatibility.md).

New runs establish tokenizer compatibility through structural identity. The
legacy behavioral fingerprint is only a migration fallback, not identity proof.
Expand All @@ -142,8 +135,8 @@ python -m pip install -e ".[dev]"
pytest -q -m "not gpu and not network"
```

Apache-2.0 licensed. See [CONTRIBUTING.md](https://github.com/DaoyuanLi2816/mini-verl/blob/main/CONTRIBUTING.md),
[SECURITY.md](https://github.com/DaoyuanLi2816/mini-verl/blob/main/SECURITY.md), the [changelog](https://github.com/DaoyuanLi2816/mini-verl/blob/main/CHANGELOG.md) and
[citation](https://github.com/DaoyuanLi2816/mini-verl/blob/main/CITATION.cff). Project records: [default GPU recipe](https://github.com/DaoyuanLi2816/mini-verl/blob/main/recipes/qwen_consumer_gpu_calc.yaml),
[frozen calculator result](https://github.com/DaoyuanLi2816/mini-verl/blob/main/benchmarks/results/gpu-calc-hard-equal-update-v2.json)
and [license](https://github.com/DaoyuanLi2816/mini-verl/blob/main/LICENSE).
Apache-2.0 licensed. See [CONTRIBUTING.md](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/CONTRIBUTING.md),
[SECURITY.md](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/SECURITY.md), the [changelog](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/CHANGELOG.md) and
[citation](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/CITATION.cff). Project records: [default GPU recipe](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/recipes/qwen_consumer_gpu_calc.yaml),
[frozen calculator result](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/benchmarks/results/gpu-calc-hard-equal-update-v2.json)
and [license](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/LICENSE).
Loading