From 4fe229ebd87d2c6ea2e6007033f4fe0cb7877c98 Mon Sep 17 00:00:00 2001 From: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com> Date: Wed, 12 Aug 2026 01:18:03 -0700 Subject: [PATCH 1/2] Add pure OPD bridge round trip and finalize v0.8.0 --- CHANGELOG.md | 26 +- CITATION.cff | 15 +- PROJECT_STATE.md | 24 +- PYPI.md | 105 ++- README.md | 73 +- README.zh-CN.md | 62 +- TODO.md | 4 +- docs/compatibility.md | 17 + .../verl-opd-v0.8-compatibility.json | 750 ++++++++++++++++++ docs/index.md | 77 +- docs/opd-quickstart.md | 22 + docs/overrides/main.html | 6 +- docs/verl-bridge.md | 79 +- release-state.yaml | 12 +- scripts/publish_verl_opd_compatibility.py | 57 ++ scripts/verify_verl_bridge_smoke.py | 4 +- src/miniverl/__init__.py | 2 +- src/miniverl/bridge/doctor.py | 103 ++- src/miniverl/bridge/export.py | 283 ++++++- src/miniverl/bridge/opd_v08.py | 65 ++ src/miniverl/cli.py | 67 +- tests/cli/test_verl_bridge_cli.py | 34 + .../test_verl_opd_compatibility_artifact.py | 21 + tests/unit/test_verl_opd_export.py | 248 ++++++ 24 files changed, 1910 insertions(+), 246 deletions(-) create mode 100644 docs/generated/verl-opd-v0.8-compatibility.json create mode 100644 scripts/publish_verl_opd_compatibility.py create mode 100644 tests/unit/test_verl_opd_compatibility_artifact.py create mode 100644 tests/unit/test_verl_opd_export.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 27e1080..cf34c9f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,15 +6,29 @@ All notable changes to miniVERL are recorded here. The format follows ## [Unreleased] -### verl v0.8 OPD config compiler +## [0.8.0] - 2026-08-12 + +### Single-GPU verl v0.8 OPD runtime - Added the typed `verl-opd-v0.8-single-gpu-v1` configuration profile and an - offline `bridge compile-opd` command. Resolved YAML and repeatable dotted - overrides compile into deterministic field-by-field compatibility reports. + offline compiler plus weight-free `plan` and executable `run` commands. + Resolved YAML and repeatable dotted overrides compile into deterministic + field-by-field compatibility reports. - Unsupported policy-gradient OPD, task-reward mixtures, KL penalties, multi-generation, multi-teacher and distributed dimensions fail closed. Engine/resource fields are labelled as local reinterpretations rather than - upstream-exact behavior; the config-only command loads no model weights. + upstream-exact behavior. +- Added first-class bounded verl Parquet prompts, padded local-HF rollout, + exact response-only selection, current-policy/teacher/cache bindings and the + pinned verl `forward_kl_topk` loss with token-mean scalar/metric/gradient + conformance. +- Added a packaged Qwen3-0.6B/1.7B NF4 recipe and standard PEFT export. One RTX + 4080 runtime-conformance run completed its first update in 12.0224 seconds at + 3.1758 GiB peak reserved VRAM. No alignment-quality comparison was run. +- Added OPD v2 import/export. Imports publish a canonical prompt profile without + inventing an environment or reward; exports preserve PEFT, teacher identity, + Parquet bytes and pure OPD overrides without a reward scaffold. Missing base + snapshots and teacher-adapter materialization remain explicit launch blockers. ## [0.7.1] - 2026-08-11 @@ -833,7 +847,9 @@ Same-tokenizer only; one trajectory per forward pass; `swap` unavailable for quantized models; only Qwen3 and Qwen2 architectures tested; single-seed GPU results. The full list is in `docs/limitations.md`. -[Unreleased]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.7.1...HEAD +[Unreleased]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.8.0...HEAD +[0.8.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.7.1...v0.8.0 +[0.7.1]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.7.0...v0.7.1 [0.7.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.3...v0.7.0 [0.6.3]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.2...v0.6.3 [0.6.2]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.1...v0.6.2 diff --git a/CITATION.cff b/CITATION.cff index dddbdb6..3f2592f 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -2,14 +2,17 @@ cff-version: 1.2.0 title: "miniVERL: Auditable single-GPU alignment and distillation runtime" message: "If you use miniVERL in your work, please cite it as below." type: software -version: 0.7.1 -date-released: 2026-08-11 +version: 0.8.0 +date-released: 2026-08-12 license: Apache-2.0 repository-code: "https://github.com/DaoyuanLi2816/mini-verl" url: "https://github.com/DaoyuanLi2816/mini-verl" abstract: >- - miniVERL is an auditable single-GPU alignment and distillation runtime with a - bounded artifact bridge to one pinned verl profile. Its native core is + miniVERL is a local single-GPU runtime for a documented subset of verl v0.8 + on-policy distillation. It consumes typed verl-shaped configuration and + Parquet prompts, executes actor rollout, teacher scoring and actor update in + one inspectable process, and exports standard PEFT, Parquet and configuration + artifacts through a fail-closed pinned bridge. Its native core also supports multi-turn, tool-aware on-policy distillation: a student language model samples its own tool-using trajectories against deterministic local environments, a teacher scores exactly the states the student visited, and @@ -28,8 +31,8 @@ abstract: >- published with their negative and mixed results intact. A fail-closed artifact bridge exchanges configs, LoRA adapters, tokenizers and Parquet datasets with one pinned verl profile, verifying what it can and refusing to - imply the rest: it does not launch distributed jobs and does not establish - algorithmic parity with PPO. miniVERL is designed for one personal CUDA GPU, + imply the rest: it does not launch distributed jobs or establish broad + algorithmic parity. miniVERL is designed for one personal CUDA GPU, automatically selects bf16 or fp16, and requires neither Ray nor a cluster. Published performance is measured on one RTX 4080; other GPU models use the same code path but remain unmeasured. The v0.7 external-alignment study diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index c190fe9..8a67123 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -4,9 +4,9 @@ Living build log for **miniVERL** (`mini-verl` / `miniverl` / CLI `miniverl`). A checkbox is not evidence: every completed item names the command that was run and what it printed. -Last updated: 2026-08-11. +Last updated: 2026-08-12. -Canonical release state: stable `v0.7.1` (`830a4ca5d873bce4cdcc7c43a44d827b096e8c0c`), development `0.8.0.dev0`. +Canonical release state: releasing `v0.8.0`. Every public version claim is generated from `release-state.yaml` and gated by `python scripts/release_state.py --check`. @@ -84,6 +84,26 @@ from construction start. The standard PEFT adapter load passed. This is a systems result only; no alignment endpoint or method comparison ran. The machine-readable record is `benchmarks/results/rtx4080-verl-opd-runtime-v1.json`. +## v0.8.0 single-GPU verl OPD pivot — PR E + +`import-verl --profile verl-opd-v0.8-single-gpu-v1 --config ...` now publishes +a transactional, canonical OPD profile plus a field-by-field report; it uses +the same typed compiler as `plan`/`run`, consumes Parquet paths and needs no +ToolEnvironment or reward. The legacy environment-profile importer remains +available under its original profile name. + +`export-verl` recognizes a completed compatible OPD run and emits the standard +student PEFT adapter, exact student/teacher identities, byte-preserved Parquet, +source config, compiled plan and pure `forward_kl_topk` overrides. It emits no +reward scaffold. Student/teacher/data loadability, upstream parse/tiny-smoke, +launchability and distributed execution are independent fields. Missing base +snapshots and teacher-adapter materialization keep new bundles fail-closed and +`launchable: false`; no distributed command or execution claim is generated. + +The generated `docs/generated/verl-opd-v0.8-compatibility.json` binds 72 source +fields to the pinned compiler fixture. Upstream parse conformance is recomputed +against verl `7aed6b23`; it does not imply a model launch or distributed job. + ## v0.7.1 Product correction — RELEASE CANDIDATE Branch `v0.7.1-product-correction` starts from synchronized main diff --git a/PYPI.md b/PYPI.md index 1ed0081..451ff63 100644 --- a/PYPI.md +++ b/PYPI.md @@ -1,5 +1,5 @@

- miniVERL — single-GPU LLM post-training + miniVERL — single-GPU LLM post-training

@@ -8,7 +8,7 @@ [![Build](https://github.com/DaoyuanLi2816/mini-verl/actions/workflows/build.yml/badge.svg)](https://github.com/DaoyuanLi2816/mini-verl/actions/workflows/build.yml) [![PyPI](https://img.shields.io/pypi/v/miniverl.svg)](https://pypi.org/project/miniverl/) [![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org) -[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/DaoyuanLi2816/mini-verl/blob/main/LICENSE) +[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/LICENSE)
@@ -16,33 +16,33 @@ PyPI · Stable docs · Development docs · - 中文 + 中文

-**miniVERL is a local, inspectable single-GPU alignment and distillation -runtime.** It runs native SFT, DPO, KD and strict OPD recipes, preserves -assistant-only loss masks and policy-version provenance, and exchanges standard -HF/PEFT/Parquet artifacts through a fail-closed bridge to one pinned verl -profile. +**Run a documented subset of verl-style on-policy distillation on one consumer +GPU.** miniVERL accepts a typed verl v0.8 OPD profile and Parquet prompts, +executes actor rollout → teacher scoring → actor update locally, and exports +standard PEFT/Parquet/config artifacts for scale-out. Native SFT, DPO, KD and +tool-agent recipes remain available. -PyPI `v0.7.1` is stable; `main` is development. miniVERL is independent from +PyPI `v0.8.0` is stable; `main` is development. miniVERL is independent from verl. It does not claim arbitrary verl YAML execution, distributed execution, or full algorithmic compatibility. -## Install and verify in about a minute +## Pip-only OPD quickstart ```bash python -m pip install "miniverl[train]" -miniverl doctor -miniverl demo --fast --output runs/quickstart -miniverl inspect runs/quickstart/trajectories.jsonl -miniverl evidence validate alignment-external-v1 +miniverl data sample --format verl-parquet --out prompts.parquet +miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config builtin:qwen3-0.6b-1.7b-opd \ + --set 'data.train_files=["prompts.parquet"]' +miniverl run --profile verl-opd-v0.8-single-gpu-v1 --config builtin:qwen3-0.6b-1.7b-opd \ + --set 'data.train_files=["prompts.parquet"]' --dry-run ``` -The deterministic demo downloads no model and produces typed trajectories, a -checksummed teacher cache, manifest and report. The evidence command reads -self-contained package data; it works from a wheel without a Git checkout. -For schemas and inspection without the ML stack, install `miniverl` alone. +The sample, plan and dry run need no Git checkout; planning loads no weights. +Remove `--dry-run` on one CUDA GPU to execute the pinned Qwen3-0.6B/1.7B NF4 +recipe and produce a loadable PEFT adapter. [Follow the OPD quickstart](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/opd-quickstart.md). ## Supported hardware and runtime boundary @@ -51,43 +51,36 @@ device-name agnostic, but fit depends on model pair, context, kernels and VRAM. Install the matching CUDA-enabled PyTorch build first, then `miniverl[train,cuda]`; that extra does not select a CUDA PyTorch wheel. Ray, FSDP, Megatron, PPO, GRPO and distributed launch are outside the runtime. -See the [single-GPU guide](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/single-gpu-guide.md). +See the [single-GPU guide](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/single-gpu-guide.md). ## verl compatibility summary -The bridge targets official verl `v0.8.0` at commit `7aed6b23`. Its verified -boundary is checksummed standard artifacts plus pinned config-parse and -model/data-load smoke—not native checkpoint parity or a completed verl job. -Imports fail closed when dataset, environment, teacher, objective or schedule -semantics are unresolved; they never substitute calculator tasks or invent an -unqualified teacher. +The executable profile targets official verl `v0.8.0` at commit `7aed6b23` and +supports one actor, one teacher, `n=1`, pure GKD `forward_kl_topk`, token-mean +aggregation, LoRA/QLoRA and no reward/KL penalty. PG OPD, task-reward mixtures, +multi-teacher, multimodal and distributed fields fail closed. -Current exports remain `launchable: false`: the base snapshot is absent, the -reward scaffold fails closed and required mappings remain placeholders. The -entry point is `launch.template.sh`; readiness, parse/load evidence, -launchability, distributed execution and semantic parity are separate facts. -[Read the bridge contract](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-bridge.md). +Compatible OPD exports contain no reward scaffold. They preserve student and +teacher identities, Parquet bytes and OPD overrides, but remain +`launchable: false` until exact base snapshots are materialized. Parse status, +artifact loadability, launchability and distributed execution are separate. +[Read the bridge contract](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/verl-bridge.md). -## One measured systems result +## Measured RTX 4080 runtime -On one RTX 4080 with Qwen3-0.6B and eight fixed SQLite trajectories, physical -batch 4 increased dual-model update throughput from 2.369 to 3.866 -trajectories/s. Shared-backbone batch 4 used 2.227 GiB peak reserved memory -versus 3.035 GiB for dual model while running 10.1% slower. All 12 -preregistered equivalence comparisons passed. This is one workload on one -machine, not a promise for other GPUs. - -![Measured throughput and reserved VRAM for dual-model and shared-backbone runtime cells](https://raw.githubusercontent.com/DaoyuanLi2816/mini-verl/main/docs/consumer-runtime-v1-pareto.svg) - -[Consumer Runtime v1 methods and caveats](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/consumer-runtime-v1.md) +The packaged Qwen3-0.6B/1.7B recipe completed two 16-token rollouts and one OPD +update with **3.1758 GiB peak reserved VRAM**; the first update completed in +**12.0224 s** and the standard PEFT adapter reloaded successfully. This proves +one runtime/artifact path only—no alignment-quality endpoint or method +comparison ran. [Exact recipe, timings and hashes](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/opd-quickstart.md). ## Three paths | Path | Start with | Concrete artifact | Next | | --- | --- | --- | --- | -| **Align** — use SFT, DPO, KD or OPD only when pilot evidence supports the cost | `miniverl pilot recipes/alignment_policy_conditioned_qwen.yaml` | `alignment-card.json` | [Alignment Lab](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/alignment-lab/alignment-lab-v1.md) | -| **Distill locally** — strict OPD, shared backbones and padded updates on one CUDA GPU | `miniverl train recipes/qwen_consumer_gpu_shared.yaml --dry-run` | resolved config and revision-pinned PEFT adapter | [Bring your own GPU](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/single-gpu-guide.md) | -| **Scale out** — convert Parquet, export standard artifacts and inspect the unsupported boundary | `miniverl bridge doctor scaleout-bundle` | `provenance/compatibility-report.json` | [Verified artifact bridge](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-bridge.md) | +| **Run OPD locally** | `miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml` | compiled plan, trajectories, targets and PEFT adapter | [Plan and run](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/opd-quickstart.md) | +| **Bring a verl config** | `miniverl import-verl --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml --out local-opd.yaml` | field report plus round-trippable profile | [Compatibility](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/compatibility.md) | +| **Move data and artifacts** | `miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout` | Parquet + PEFT + OPD override bundle | [Bridge contract](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/verl-bridge.md) | ## Research notes and preserved negative evidence @@ -108,7 +101,7 @@ miniverl pilot --builtin-study alignment-external-v1 --json The result is `do_not_continue_this_study` and `insufficient_evidence`, not a recommendation among SFT/DPO/KD/OPD. Granite Guardian values are unqualified selection diagnostics; Granite, PairRM and teacher qualification and the -reserved final test did not run. [Study and limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/alignment-external/alignment-external-v1.md). +reserved final test did not run. [Study and limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/alignment-external/alignment-external-v1.md). ### Earlier measured alignment case study @@ -118,17 +111,17 @@ ceiling; continued SFT and both OPD variants retained measured regressions. The two sandbox safety checks tied at zero while utility still regressed. IFEval, XSTest, HarmBench and RewardBench were not executed, and “preference win rate” is a deterministic Minipolicy paired outcome, not human preference. -[Seed-level evidence](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/alignment-lab/alignment-lab-v1.md). +[Seed-level evidence](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/alignment-lab/alignment-lab-v1.md). -- [RecoveryBench v1](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/recoverybench/recoverybench-v1.md): frozen-student KD +- [RecoveryBench v1](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/recoverybench/recoverybench-v1.md): frozen-student KD beat slower fresh-state OPD on the preregistered primary view; the verifier gate remained `insufficient_evidence`. -- [Calculator benchmark](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/benchmarking.md): both negative controls completed +- [Calculator benchmark](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/benchmarking.md): both negative controls completed normally at 0%; the ambiguous historical protocol-v1 prompt prevents attributing failure solely to intrinsic teacher behavior. -- [Limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/limitations.md), [math](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/math.md), - [reproducibility](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/reproducibility.md) and - [compatibility policy](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/compatibility.md). +- [Limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/limitations.md), [math](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/math.md), + [reproducibility](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/reproducibility.md) and + [compatibility policy](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/docs/compatibility.md). New runs establish tokenizer compatibility through structural identity. The legacy behavioral fingerprint is only a migration fallback, not identity proof. @@ -142,8 +135,8 @@ python -m pip install -e ".[dev]" pytest -q -m "not gpu and not network" ``` -Apache-2.0 licensed. See [CONTRIBUTING.md](https://github.com/DaoyuanLi2816/mini-verl/blob/main/CONTRIBUTING.md), -[SECURITY.md](https://github.com/DaoyuanLi2816/mini-verl/blob/main/SECURITY.md), the [changelog](https://github.com/DaoyuanLi2816/mini-verl/blob/main/CHANGELOG.md) and -[citation](https://github.com/DaoyuanLi2816/mini-verl/blob/main/CITATION.cff). Project records: [default GPU recipe](https://github.com/DaoyuanLi2816/mini-verl/blob/main/recipes/qwen_consumer_gpu_calc.yaml), -[frozen calculator result](https://github.com/DaoyuanLi2816/mini-verl/blob/main/benchmarks/results/gpu-calc-hard-equal-update-v2.json) -and [license](https://github.com/DaoyuanLi2816/mini-verl/blob/main/LICENSE). +Apache-2.0 licensed. See [CONTRIBUTING.md](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/CONTRIBUTING.md), +[SECURITY.md](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/SECURITY.md), the [changelog](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/CHANGELOG.md) and +[citation](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/CITATION.cff). Project records: [default GPU recipe](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/recipes/qwen_consumer_gpu_calc.yaml), +[frozen calculator result](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/benchmarks/results/gpu-calc-hard-equal-update-v2.json) +and [license](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.8.0/LICENSE). diff --git a/README.md b/README.md index 60b24c4..8416ac2 100644 --- a/README.md +++ b/README.md @@ -19,30 +19,30 @@ 中文

-**miniVERL is a local, inspectable single-GPU alignment and distillation -runtime.** It runs native SFT, DPO, KD and strict OPD recipes, preserves -assistant-only loss masks and policy-version provenance, and exchanges standard -HF/PEFT/Parquet artifacts through a fail-closed bridge to one pinned verl -profile. +**Run a documented subset of verl-style on-policy distillation on one consumer +GPU.** miniVERL accepts a typed verl v0.8 OPD profile and Parquet prompts, +executes actor rollout → teacher scoring → actor update locally, and exports +standard PEFT/Parquet/config artifacts for scale-out. Native SFT, DPO, KD and +tool-agent recipes remain available. -PyPI `v0.7.1` is stable; `main` is development. miniVERL is independent from +PyPI `v0.8.0` is stable; `main` is development. miniVERL is independent from verl. It does not claim arbitrary verl YAML execution, distributed execution, or full algorithmic compatibility. -## Install and verify in about a minute +## Pip-only OPD quickstart ```bash python -m pip install "miniverl[train]" -miniverl doctor -miniverl demo --fast --output runs/quickstart -miniverl inspect runs/quickstart/trajectories.jsonl -miniverl evidence validate alignment-external-v1 +miniverl data sample --format verl-parquet --out prompts.parquet +miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config builtin:qwen3-0.6b-1.7b-opd \ + --set 'data.train_files=["prompts.parquet"]' +miniverl run --profile verl-opd-v0.8-single-gpu-v1 --config builtin:qwen3-0.6b-1.7b-opd \ + --set 'data.train_files=["prompts.parquet"]' --dry-run ``` -The deterministic demo downloads no model and produces typed trajectories, a -checksummed teacher cache, manifest and report. The evidence command reads -self-contained package data; it works from a wheel without a Git checkout. -For schemas and inspection without the ML stack, install `miniverl` alone. +The sample, plan and dry run need no Git checkout; planning loads no weights. +Remove `--dry-run` on one CUDA GPU to execute the pinned Qwen3-0.6B/1.7B NF4 +recipe and produce a loadable PEFT adapter. [Follow the OPD quickstart](docs/opd-quickstart.md). ## Supported hardware and runtime boundary @@ -55,39 +55,32 @@ See the [single-GPU guide](docs/single-gpu-guide.md). ## verl compatibility summary -The bridge targets official verl `v0.8.0` at commit `7aed6b23`. Its verified -boundary is checksummed standard artifacts plus pinned config-parse and -model/data-load smoke—not native checkpoint parity or a completed verl job. -Imports fail closed when dataset, environment, teacher, objective or schedule -semantics are unresolved; they never substitute calculator tasks or invent an -unqualified teacher. - -Current exports remain `launchable: false`: the base snapshot is absent, the -reward scaffold fails closed and required mappings remain placeholders. The -entry point is `launch.template.sh`; readiness, parse/load evidence, -launchability, distributed execution and semantic parity are separate facts. -[Read the bridge contract](docs/verl-bridge.md). - -## One measured systems result +The executable profile targets official verl `v0.8.0` at commit `7aed6b23` and +supports one actor, one teacher, `n=1`, pure GKD `forward_kl_topk`, token-mean +aggregation, LoRA/QLoRA and no reward/KL penalty. PG OPD, task-reward mixtures, +multi-teacher, multimodal and distributed fields fail closed. -On one RTX 4080 with Qwen3-0.6B and eight fixed SQLite trajectories, physical -batch 4 increased dual-model update throughput from 2.369 to 3.866 -trajectories/s. Shared-backbone batch 4 used 2.227 GiB peak reserved memory -versus 3.035 GiB for dual model while running 10.1% slower. All 12 -preregistered equivalence comparisons passed. This is one workload on one -machine, not a promise for other GPUs. +Compatible OPD exports contain no reward scaffold. They preserve student and +teacher identities, Parquet bytes and OPD overrides, but remain +`launchable: false` until exact base snapshots are materialized. Parse status, +artifact loadability, launchability and distributed execution are separate. +[Read the bridge contract](docs/verl-bridge.md). -![Measured throughput and reserved VRAM for dual-model and shared-backbone runtime cells](docs/consumer-runtime-v1-pareto.svg) +## Measured RTX 4080 runtime -[Consumer Runtime v1 methods and caveats](docs/consumer-runtime-v1.md) +The packaged Qwen3-0.6B/1.7B recipe completed two 16-token rollouts and one OPD +update with **3.1758 GiB peak reserved VRAM**; the first update completed in +**12.0224 s** and the standard PEFT adapter reloaded successfully. This proves +one runtime/artifact path only—no alignment-quality endpoint or method +comparison ran. [Exact recipe, timings and hashes](docs/opd-quickstart.md). ## Three paths | Path | Start with | Concrete artifact | Next | | --- | --- | --- | --- | -| **Align** — use SFT, DPO, KD or OPD only when pilot evidence supports the cost | `miniverl pilot recipes/alignment_policy_conditioned_qwen.yaml` | `alignment-card.json` | [Alignment Lab](docs/alignment-lab/alignment-lab-v1.md) | -| **Distill locally** — strict OPD, shared backbones and padded updates on one CUDA GPU | `miniverl train recipes/qwen_consumer_gpu_shared.yaml --dry-run` | resolved config and revision-pinned PEFT adapter | [Bring your own GPU](docs/single-gpu-guide.md) | -| **Scale out** — convert Parquet, export standard artifacts and inspect the unsupported boundary | `miniverl bridge doctor scaleout-bundle` | `provenance/compatibility-report.json` | [Verified artifact bridge](docs/verl-bridge.md) | +| **Run OPD locally** | `miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml` | compiled plan, trajectories, targets and PEFT adapter | [Plan and run](docs/opd-quickstart.md) | +| **Bring a verl config** | `miniverl import-verl --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml --out local-opd.yaml` | field report plus round-trippable profile | [Compatibility](docs/compatibility.md) | +| **Move data and artifacts** | `miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout` | Parquet + PEFT + OPD override bundle | [Bridge contract](docs/verl-bridge.md) | ## Research notes and preserved negative evidence diff --git a/README.zh-CN.md b/README.zh-CN.md index 99e651c..6d98a22 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -19,27 +19,28 @@ English

-**miniVERL 是一个本地、可检查的单卡对齐与蒸馏运行时。** 它运行原生 -SFT、DPO、KD 与严格 OPD recipe，保留仅 assistant token 的 loss mask 和 -policy-version 来源，并通过 fail-closed 桥接与一个锁定的 verl profile 交换 -标准 HF/PEFT/Parquet 产物。 +**在一张个人 GPU 上运行有明确边界的 verl 风格在线策略蒸馏。** miniVERL +读取类型化的 verl v0.8 OPD profile 与 Parquet prompt，在本地执行 actor rollout +→ teacher scoring → actor update，并导出标准 PEFT/Parquet/config 产物用于扩展。 +原生 SFT、DPO、KD 与 tool-agent recipe 仍然保留。 -PyPI `v0.7.1` 是稳定版；`main` 是开发版。miniVERL 独立于 verl，不声称可 +PyPI `v0.8.0` 是稳定版；`main` 是开发版。miniVERL 独立于 verl，不声称可 执行任意 verl YAML、分布式任务或具备完整算法兼容性。 -## 大约一分钟完成安装与验证 +## 仅用 pip 的 OPD quickstart ```bash python -m pip install "miniverl[train]" -miniverl doctor -miniverl demo --fast --output runs/quickstart -miniverl inspect runs/quickstart/trajectories.jsonl -miniverl evidence validate alignment-external-v1 +miniverl data sample --format verl-parquet --out prompts.parquet +miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config builtin:qwen3-0.6b-1.7b-opd \ + --set 'data.train_files=["prompts.parquet"]' +miniverl run --profile verl-opd-v0.8-single-gpu-v1 --config builtin:qwen3-0.6b-1.7b-opd \ + --set 'data.train_files=["prompts.parquet"]' --dry-run ``` -确定性 demo 不下载模型，会生成类型化 trajectory、带校验和的 teacher cache、 -manifest 与报告。证据命令读取 wheel 自带数据，无需 Git checkout。若只需 schema -与检查功能，可仅安装 `miniverl`。 +sample、plan 与 dry run 无需 Git checkout，plan 也不加载权重。在一张 CUDA GPU +上去掉 `--dry-run` 即可执行锁定的 Qwen3-0.6B/1.7B NF4 recipe，并生成可加载的 +PEFT adapter。参见 [OPD quickstart](docs/opd-quickstart.md)。 ## 支持的硬件与运行边界 @@ -51,34 +52,29 @@ miniVERL 在 CPU 或一张 NVIDIA CUDA GPU 上运行单个本地进程。CUDA ## verl 兼容性摘要 -桥接锁定官方 verl `v0.8.0`、commit `7aed6b23`。已验证的边界是带校验和的标准 -产物以及锁定版本的配置解析、模型/数据加载冒烟测试，不包括原生 checkpoint -等价或已完成的 verl 作业。若数据集、环境、教师、目标或 schedule 语义未解析， -导入会 fail closed；它不会替换成 calculator task 或虚构教师。 +可执行 profile 锁定官方 verl `v0.8.0`、commit `7aed6b23`，支持单 actor、单 teacher、 +`n=1`、纯 GKD `forward_kl_topk`、token-mean、LoRA/QLoRA，且不使用 reward 或 KL +penalty。PG OPD、task-reward mixture、多 teacher、多模态与分布式字段会 fail closed。 -当前导出仍为 `launchable: false`：缺少 base snapshot，reward scaffold 会失败关闭， -且必要映射仍是占位符。入口名为 `launch.template.sh`；readiness、parse/load 证据、 -launchability、分布式执行与语义等价分别报告。参见[桥接契约](docs/verl-bridge.md)。 +兼容的 OPD 导出不包含 reward scaffold；它保留 student/teacher 身份、Parquet 原始 +字节与 OPD overrides，但在精确 base snapshot 尚未 materialize 时仍为 +`launchable: false`。解析、产物可加载性、launchability 与分布式执行分别报告。 +参见[桥接契约](docs/verl-bridge.md)。 -## 一项系统实测结果 +## RTX 4080 运行时实测 -在一张 RTX 4080、Qwen3-0.6B 与 8 条固定 SQLite trajectory 上，physical batch 4 -将 dual-model update throughput 从 2.369 提高到 3.866 trajectories/s。 -shared-backbone batch 4 的 peak reserved memory 为 2.227 GiB，dual model 为 -3.035 GiB，但前者慢 10.1%。12 项预注册等价比较全部通过。这只是单机单 workload -数据，不是对其他 GPU 的承诺。 - -![dual-model 与 shared-backbone runtime 的实测吞吐和 reserved VRAM](docs/consumer-runtime-v1-pareto.svg) - -[Consumer Runtime v1 方法与限制](docs/consumer-runtime-v1.md) +打包的 Qwen3-0.6B/1.7B recipe 完成了两条 16-token rollout 与一次 OPD update； +**peak reserved VRAM 为 3.1758 GiB**，首次 update 在 **12.0224 秒**完成，标准 +PEFT adapter 成功重新加载。这只证明一个运行时/产物路径；没有运行对齐质量 +endpoint 或方法比较。[精确 recipe、计时与哈希](docs/opd-quickstart.md)。 ## 三条使用路径 | 路径 | 起点 | 真实产物 | 下一步 | | --- | --- | --- | --- | -| **Align** — 仅在 pilot 证据支持成本时使用 SFT、DPO、KD 或 OPD | `miniverl pilot recipes/alignment_policy_conditioned_qwen.yaml` | `alignment-card.json` | [Alignment Lab](docs/alignment-lab/alignment-lab-v1.md) | -| **本地蒸馏** — 在一张 CUDA GPU 上运行严格 OPD、共享 backbone 与 padded update | `miniverl train recipes/qwen_consumer_gpu_shared.yaml --dry-run` | resolved config 与锁定 revision 的 PEFT adapter | [使用自己的 GPU](docs/single-gpu-guide.md) | -| **Scale out** — 转换 Parquet、导出标准产物并检查不支持边界 | `miniverl bridge doctor scaleout-bundle` | `provenance/compatibility-report.json` | [产物桥接](docs/verl-bridge.md) | +| **本地运行 OPD** | `miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml` | compiled plan、trajectory、target 与 PEFT adapter | [Plan 与 run](docs/opd-quickstart.md) | +| **带入 verl config** | `miniverl import-verl --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml --out local-opd.yaml` | 字段报告与可往返 profile | [兼容性](docs/compatibility.md) | +| **移动数据与产物** | `miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout` | Parquet + PEFT + OPD override bundle | [桥接契约](docs/verl-bridge.md) | ## 研究记录与保留的负结果 diff --git a/TODO.md b/TODO.md index 3b6f62d..009ea0f 100644 --- a/TODO.md +++ b/TODO.md @@ -21,8 +21,8 @@ Alignment Lab and pinned verl-bridge work lives in `PROJECT_STATE.md` and ## Runtime scope - [ ] Cross-tokenizer distillation with an explicit alignment contract. -- [ ] Batched or engine-backed rollout decoding; v0.4 batches update forwards, - while rollout generation remains deliberately sequential. +- [ ] Engine-backed decoding beyond the current padded local-HF prompt batches; + tool-environment multi-turn generation remains deliberately sequential. - [ ] Additional tested model families beyond Qwen2/Qwen3. - [ ] Entropy-aware divergence mixing after a prespecified experiment; current code records teacher entropy but does not implement the method. diff --git a/docs/compatibility.md b/docs/compatibility.md index afe8d13..b17e1ac 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -23,6 +23,23 @@ regeneration hint. ## Pinned bridge, not generic compatibility +v0.8 adds the separately versioned `verl-opd-v0.8-single-gpu-v1` executable +profile: one local actor, one teacher, one generation, pure GKD +`forward_kl_topk`, token-mean aggregation, LoRA/QLoRA and Parquet prompts. +`import-verl --config` publishes a canonical profile plus field report; +`export-verl` recognizes a completed compatible run and emits reward-free OPD +overrides. PG OPD, task-reward mixtures, multiple teachers, multimodal and all +distributed semantics fail closed. Local physical phase scheduling is always +labelled as a reinterpretation of upstream resource fields. + +The [machine-readable field matrix](generated/verl-opd-v0.8-compatibility.json) +is regenerated directly from the typed compiler and its pinned resolved +fixture; CI compares the committed bytes with the generator output. + +The older `single-gpu-online-distillation-v1` import/export contract remains +available for migration and retains its historical non-launchable PPO/reward +scaffold semantics. + Compatibility Level 1 covers standard Hugging Face, PEFT, safetensors, tokenizer and Parquet artifacts. Level 2 is a fail-closed 14-field config whitelist. **miniVERL-defined compatibility Level 3** adds the generated diff --git a/docs/generated/verl-opd-v0.8-compatibility.json b/docs/generated/verl-opd-v0.8-compatibility.json new file mode 100644 index 0000000..5fb899f --- /dev/null +++ b/docs/generated/verl-opd-v0.8-compatibility.json @@ -0,0 +1,750 @@ +{ + "classification_counts": { + "exact": 23, + "informational_only": 12, + "locally_reinterpreted": 22, + "semantically_conformant": 15 + }, + "compiled_digest": "44d9392a43fd1c166217d2ff992a4b4202070df08d32f03a767a59c15793a7ce", + "executable": true, + "field_count": 72, + "fields": [ + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "loss.reduction", + "reason": "token-mean is the only executable v0.8 mode", + "semantic_risk": "none", + "source_value": "token-mean", + "upstream_field": "actor_rollout_ref.actor.loss_agg_mode", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "optimizer.lr", + "reason": "same learning rate", + "semantic_risk": "none", + "source_value": "1e-5", + "upstream_field": "actor_rollout_ref.actor.optim.lr", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "optimizer.lr_warmup_steps", + "reason": "same optimizer-step count", + "semantic_risk": "none", + "source_value": 0, + "upstream_field": "actor_rollout_ref.actor.optim.lr_warmup_steps", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "optimizer.weight_decay", + "reason": "same optimizer coefficient", + "semantic_risk": "none", + "source_value": 0.01, + "upstream_field": "actor_rollout_ref.actor.optim.weight_decay", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "batching.update_token_budget", + "reason": "one-GPU physical token budget", + "semantic_risk": "medium", + "source_value": 2048, + "upstream_field": "actor_rollout_ref.actor.ppo_max_token_len_per_gpu", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "batching.update_trajectory_batch_size", + "reason": "used as a logical update batch, not a PPO mini-batch", + "semantic_risk": "high", + "source_value": 8, + "upstream_field": "actor_rollout_ref.actor.ppo_mini_batch_size", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "batching.dynamic_physical_batching", + "reason": "may change only physical execution", + "semantic_risk": "medium", + "source_value": true, + "upstream_field": "actor_rollout_ref.actor.use_dynamic_bsz", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "loss.actor_reference_kl", + "reason": "must remain disabled for this profile", + "semantic_risk": "none", + "source_value": false, + "upstream_field": "actor_rollout_ref.actor.use_kl_loss", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "student.gradient_checkpointing", + "reason": "same memory technique", + "semantic_risk": "none", + "source_value": true, + "upstream_field": "actor_rollout_ref.model.enable_gradient_checkpointing", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "student.adapter.path", + "reason": "same optional adapter artifact identity", + "semantic_risk": "none", + "source_value": null, + "upstream_field": "actor_rollout_ref.model.lora_adapter_path", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "student.lora.alpha", + "reason": "same PEFT scale", + "semantic_risk": "none", + "source_value": 32, + "upstream_field": "actor_rollout_ref.model.lora_alpha", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "student.lora.r", + "reason": "same PEFT rank", + "semantic_risk": "none", + "source_value": 16, + "upstream_field": "actor_rollout_ref.model.lora_rank", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "student.model_id", + "reason": "same model identity", + "semantic_risk": "none", + "source_value": "Qwen/Qwen3-0.6B", + "upstream_field": "actor_rollout_ref.model.path", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "student.lora.target_modules", + "reason": "same module names", + "semantic_risk": "none", + "source_value": [ + "q_proj", + "v_proj" + ], + "upstream_field": "actor_rollout_ref.model.target_modules", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "memory.rollout_fraction", + "reason": "planner hint rather than an inference-server reservation", + "semantic_risk": "high", + "source_value": 0.5, + "upstream_field": "actor_rollout_ref.rollout.gpu_memory_utilization", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "rollout.max_model_len", + "reason": "same context ceiling", + "semantic_risk": "none", + "source_value": 320, + "upstream_field": "actor_rollout_ref.rollout.max_model_len", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "batching.rollout_token_budget", + "reason": "local padded-token budget", + "semantic_risk": "medium", + "source_value": 2048, + "upstream_field": "actor_rollout_ref.rollout.max_num_batched_tokens", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "batching.rollout_batch_limit", + "reason": "local sequence cap", + "semantic_risk": "medium", + "source_value": 8, + "upstream_field": "actor_rollout_ref.rollout.max_num_seqs", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "rollout.n", + "reason": "one generation per prompt", + "semantic_risk": "none", + "source_value": 1, + "upstream_field": "actor_rollout_ref.rollout.n", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "rollout.backend", + "reason": "source engine name is recorded; local execution does not claim vLLM/SGLang equivalence", + "semantic_risk": "high", + "source_value": "vllm", + "upstream_field": "actor_rollout_ref.rollout.name", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "rollout.temperature", + "reason": "same sampling value", + "semantic_risk": "none", + "source_value": 1.0, + "upstream_field": "actor_rollout_ref.rollout.temperature", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "placement.tensor_parallel", + "reason": "must be one on one GPU", + "semantic_risk": "high", + "source_value": 1, + "upstream_field": "actor_rollout_ref.rollout.tensor_model_parallel_size", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "rollout.top_p", + "reason": "same sampling value", + "semantic_risk": "none", + "source_value": 0.95, + "upstream_field": "actor_rollout_ref.rollout.top_p", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "loss.kl_in_reward", + "reason": "must remain disabled; pure OPD has no reward", + "semantic_risk": "none", + "source_value": false, + "upstream_field": "algorithm.use_kl_in_reward", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "source.filter_overlong_prompts", + "reason": "same filtering intent", + "semantic_risk": "none", + "source_value": false, + "upstream_field": "data.filter_overlong_prompts", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "source.max_prompt_length", + "reason": "same token bound", + "semantic_risk": "none", + "source_value": 256, + "upstream_field": "data.max_prompt_length", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "source.max_response_length", + "reason": "same token bound", + "semantic_risk": "none", + "source_value": 64, + "upstream_field": "data.max_response_length", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "source.prompt_key", + "reason": "the same row field is selected", + "semantic_risk": "none", + "source_value": "prompt", + "upstream_field": "data.prompt_key", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "source.seed", + "reason": "same integer seed", + "semantic_risk": "none", + "source_value": 17, + "upstream_field": "data.seed", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "source.shuffle", + "reason": "deterministic local shuffle", + "semantic_risk": "none", + "source_value": true, + "upstream_field": "data.shuffle", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "scheduler.logical_batch_size", + "reason": "logical examples per update; physical one-GPU batches are separately bounded", + "semantic_risk": "medium", + "source_value": 8, + "upstream_field": "data.train_batch_size", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "source.train_files", + "reason": "the same Parquet paths are consumed", + "semantic_risk": "none", + "source_value": [ + "data/train.parquet" + ], + "upstream_field": "data.train_files", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "source.truncation", + "reason": "same named policy", + "semantic_risk": "none", + "source_value": "error", + "upstream_field": "data.truncation", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "source.val_files", + "reason": "the same validation paths are consumed", + "semantic_risk": "none", + "source_value": [ + "data/val.parquet" + ], + "upstream_field": "data.val_files", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": null, + "reason": "coefficient is inactive when task rewards are disabled", + "semantic_risk": "none", + "source_value": 1.0, + "upstream_field": "distillation.distillation_loss.distillation_loss_coef", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "loss.log_prob_min_clamp", + "reason": "same optional log-prob clamp", + "semantic_risk": "none", + "source_value": -10.0, + "upstream_field": "distillation.distillation_loss.log_prob_min_clamp", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "loss.loss_max_clamp", + "reason": "same optional final clamp", + "semantic_risk": "none", + "source_value": null, + "upstream_field": "distillation.distillation_loss.loss_max_clamp", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "loss.mode", + "reason": "dedicated pinned forward_kl_topk path", + "semantic_risk": "none", + "source_value": "forward_kl_topk", + "upstream_field": "distillation.distillation_loss.loss_mode", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "loss.top_k", + "reason": "same teacher top-k", + "semantic_risk": "none", + "source_value": 32, + "upstream_field": "distillation.distillation_loss.topk", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "loss.use_policy_gradient", + "reason": "must remain disabled for direct GKD OPD", + "semantic_risk": "none", + "source_value": false, + "upstream_field": "distillation.distillation_loss.use_policy_gradient", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "loss.use_task_rewards", + "reason": "must remain disabled", + "semantic_risk": "none", + "source_value": false, + "upstream_field": "distillation.distillation_loss.use_task_rewards", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "loss.enabled", + "reason": "must be enabled", + "semantic_risk": "none", + "source_value": true, + "upstream_field": "distillation.enabled", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "placement.teacher_phase", + "reason": "resource pool becomes one-GPU phases", + "semantic_risk": "high", + "source_value": 1, + "upstream_field": "distillation.n_gpus_per_node", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "placement.teacher_phase", + "reason": "resource pool becomes one local node", + "semantic_risk": "high", + "source_value": 1, + "upstream_field": "distillation.nnodes", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": "teacher.routing_metadata", + "reason": "single-teacher local execution does not route", + "semantic_risk": "none", + "source_value": "data_source", + "upstream_field": "distillation.teacher_key", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "placement.teacher_data_parallel", + "reason": "must be one", + "semantic_risk": "high", + "source_value": 1, + "upstream_field": "distillation.teacher_models.teacher_model.inference.data_parallel_size", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "teacher.dtype", + "reason": "same requested numerical dtype", + "semantic_risk": "none", + "source_value": "bfloat16", + "upstream_field": "distillation.teacher_models.teacher_model.inference.dtype", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "memory.teacher_fraction", + "reason": "planner hint, not server reservation", + "semantic_risk": "high", + "source_value": 0.5, + "upstream_field": "distillation.teacher_models.teacher_model.inference.gpu_memory_utilization", + "user_confirmation_required": false + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "teacher.max_model_len", + "reason": "same scoring context ceiling", + "semantic_risk": "none", + "source_value": 321, + "upstream_field": "distillation.teacher_models.teacher_model.inference.max_model_len", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "teacher.backend", + "reason": "recorded without engine-equivalence claim", + "semantic_risk": "high", + "source_value": "vllm", + "upstream_field": "distillation.teacher_models.teacher_model.inference.name", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "placement.teacher_pipeline_parallel", + "reason": "must be one", + "semantic_risk": "high", + "source_value": 1, + "upstream_field": "distillation.teacher_models.teacher_model.inference.pipeline_model_parallel_size", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "placement.teacher_tensor_parallel", + "reason": "must be one", + "semantic_risk": "high", + "source_value": 1, + "upstream_field": "distillation.teacher_models.teacher_model.inference.tensor_model_parallel_size", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "teacher.model_id", + "reason": "same teacher model identity", + "semantic_risk": "none", + "source_value": "Qwen/Qwen3-1.7B", + "upstream_field": "distillation.teacher_models.teacher_model.model_path", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "placement.teacher_phase", + "reason": "one frozen teacher role, no replicas", + "semantic_risk": "high", + "source_value": 1, + "upstream_field": "distillation.teacher_models.teacher_model.num_replicas", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": null, + "reason": "miniVERL-only physical batch", + "semantic_risk": "none", + "source_value": 2, + "upstream_field": "miniverl.batching.rollout_batch_size", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": null, + "reason": "miniVERL-only physical batch", + "semantic_risk": "none", + "source_value": 2, + "upstream_field": "miniverl.batching.teacher_score_batch_size", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": null, + "reason": "miniVERL-only physical batch", + "semantic_risk": "none", + "source_value": 2, + "upstream_field": "miniverl.batching.update_trajectory_batch_size", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": null, + "reason": "miniVERL-only planner headroom", + "semantic_risk": "none", + "source_value": 1.5, + "upstream_field": "miniverl.memory.headroom_gib", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": null, + "reason": "miniVERL-only planner limit", + "semantic_risk": "none", + "source_value": 16, + "upstream_field": "miniverl.memory.vram_limit_gib", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": null, + "reason": "miniVERL-only local extension", + "semantic_risk": "none", + "source_value": "auto", + "upstream_field": "miniverl.runtime.mode", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": "student.revision", + "reason": "miniVERL-only immutable Hub revision", + "semantic_risk": "none", + "source_value": "c1899de289a04d12100db370d81485cdf75e47ca", + "upstream_field": "miniverl.student_revision", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": null, + "reason": "miniVERL-only teacher adapter extension", + "semantic_risk": "none", + "source_value": null, + "upstream_field": "miniverl.teacher_adapter.path", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": null, + "reason": "miniVERL-only teacher adapter extension", + "semantic_risk": "none", + "source_value": null, + "upstream_field": "miniverl.teacher_adapter.revision", + "user_confirmation_required": false + }, + { + "classification": "informational_only", + "executable": true, + "local_target": "teacher.revision", + "reason": "miniVERL-only immutable Hub revision", + "semantic_risk": "none", + "source_value": "70d244cc86ccca08cf5af4e1e306ecf908b1ad5e", + "upstream_field": "miniverl.teacher_revision", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "run.name", + "reason": "same provenance label", + "semantic_risk": "none", + "source_value": "opd-smoke", + "upstream_field": "trainer.experiment_name", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "placement.device_count", + "reason": "always one local CUDA device", + "semantic_risk": "high", + "source_value": 1, + "upstream_field": "trainer.n_gpus_per_node", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "placement.node_count", + "reason": "must be one", + "semantic_risk": "high", + "source_value": 1, + "upstream_field": "trainer.nnodes", + "user_confirmation_required": false + }, + { + "classification": "exact", + "executable": true, + "local_target": "run.project", + "reason": "same provenance label", + "semantic_risk": "none", + "source_value": "mini-verl", + "upstream_field": "trainer.project_name", + "user_confirmation_required": false + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "checkpoint.interval", + "reason": "local optimizer-step interval", + "semantic_risk": "medium", + "source_value": 10, + "upstream_field": "trainer.save_freq", + "user_confirmation_required": true + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "evaluation.interval", + "reason": "local optimizer-step interval", + "semantic_risk": "medium", + "source_value": 10, + "upstream_field": "trainer.test_freq", + "user_confirmation_required": true + }, + { + "classification": "locally_reinterpreted", + "executable": true, + "local_target": "schedule.dataset_passes", + "reason": "local bounded dataset passes", + "semantic_risk": "medium", + "source_value": 1, + "upstream_field": "trainer.total_epochs", + "user_confirmation_required": true + }, + { + "classification": "semantically_conformant", + "executable": true, + "local_target": "schedule.optimizer_steps", + "reason": "explicit global optimizer-step cap", + "semantic_risk": "none", + "source_value": 2, + "upstream_field": "trainer.total_training_steps", + "user_confirmation_required": false + } + ], + "profile": "verl-opd-v0.8-single-gpu-v1", + "schema_version": 1, + "scope": "config conformance for one documented local pure-OPD profile; not full verl compatibility or distributed execution", + "source_fixture": "examples/verl-opd-v0.8-single-gpu.yaml", + "source_sha256": "085805d904136ae271db7629b4810340f54d6aba7c2ed08e7a88d8ac01511b4f", + "upstream": { + "commit": "7aed6b230776f963fa09509c10d9c3a767d1102c", + "repository": "https://github.com/verl-project/verl", + "tag": "v0.8.0" + } +} diff --git a/docs/index.md b/docs/index.md index db8c77c..50f1992 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,26 +1,29 @@ # miniVERL -Auditable single-GPU alignment and distillation runtime with native SFT, DPO, -KD and strict OPD recipes, inspectable artifacts and a bounded bridge to one -pinned verl profile. miniVERL is independent; distributed execution and full -algorithm compatibility are not claimed. +Run a documented subset of verl v0.8 on-policy distillation on one consumer +GPU. Bring a typed OPD profile and Parquet prompts, inspect rollout → teacher +scoring → update locally, then export standard artifacts. miniVERL is +independent; distributed execution and full verl compatibility are not claimed. [Install and run locally](single-gpu-guide.md){ .md-button .md-button--primary } [Read the compatibility boundary](verl-bridge.md){ .md-button } -## Install and verify in about a minute +## Pip-only OPD quickstart ```bash python -m pip install "miniverl[train]" -miniverl demo --fast --output runs/quickstart -miniverl inspect runs/quickstart/trajectories.jsonl -miniverl evidence validate alignment-external-v1 +miniverl data sample --format verl-parquet --out prompts.parquet +miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \ + --config builtin:qwen3-0.6b-1.7b-opd \ + --set 'data.train_files=["prompts.parquet"]' +miniverl run --profile verl-opd-v0.8-single-gpu-v1 \ + --config builtin:qwen3-0.6b-1.7b-opd \ + --set 'data.train_files=["prompts.parquet"]' --dry-run ``` -The deterministic demo downloads no model and produces typed trajectories, a -checksummed teacher cache, manifest and report. Packaged evidence commands need -no repository checkout. For CUDA, install the matching CUDA-enabled PyTorch -wheel first; the `[cuda]` extra does not select one. +Planning loads no weights. Remove `--dry-run` on one CUDA GPU to execute the +pinned Qwen3 recipe and export a standard PEFT adapter. Install the matching +CUDA-enabled PyTorch wheel first; the `[cuda]` extra does not select one. ## Runtime and compatibility boundary @@ -28,20 +31,18 @@ miniVERL runs one local CPU process or one NVIDIA CUDA GPU. Fit depends on the model pair, context, kernels and VRAM; there is no GPU-name allowlist. Ray, FSDP, Megatron, PPO, GRPO and distributed launch are outside the runtime. -The artifact bridge pins verl `v0.8.0` at `7aed6b23`. It verifies standard -artifact interchange and pinned config/model/data parse-load smoke. Current -exports are not launchable and do not establish algorithmic parity. +The executable profile pins verl `v0.8.0` at `7aed6b23`: one actor, one teacher, +`n=1`, GKD `forward_kl_topk`, token-mean and no reward/KL penalty. Unsupported +algorithm or distributed semantics fail closed. Exports remain unlaunchable +until exact base snapshots are materialized. -## Measured systems evidence +## Measured runtime evidence -On one RTX 4080 with Qwen3-0.6B and eight fixed SQLite trajectories, padded -updates increased dual-model update throughput from 2.369 to 3.866 -trajectories/s. Shared-backbone batch 4 used 2.227 GiB peak reserved memory -versus 3.035 GiB for dual model while running 10.1% slower. All 12 -preregistered equivalence comparisons passed. This is one measured workload, -not a hardware-wide promise. +One RTX 4080 run of the pinned Qwen3-0.6B/1.7B recipe completed its first OPD +update in 12.0224 s at 3.1758 GiB peak reserved VRAM, then reloaded its standard +PEFT adapter. No quality endpoint or method comparison ran. -[Consumer Runtime methods and caveats](consumer-runtime/index.md){ .md-button } +[Exact quickstart evidence](opd-quickstart.md){ .md-button } ## Choose a path @@ -49,50 +50,48 @@ not a hardware-wide promise.
-## Align +## Run OPD locally -Choose SFT, DPO, offline KD or OPD from explicit pilot evidence. +Plan, execute, inspect and resume one local pure-OPD run. ```bash -miniverl pilot recipes/alignment_tool_policy_toy.yaml --json +miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml ``` -**Artifact:** an [Alignment Card](alignment-lab/alignment-lab-v1.md#reproducibility-and-artifacts) -with starting checkpoint, metrics, cost and limitations. +**Artifact:** a checksummed local execution plan, trajectories and PEFT adapter. -**Next:** [When should OPD follow SFT?](alignment-lab/when-opd-should-follow-sft.md) +**Next:** [Plan and run](opd-quickstart.md)
-## Distill locally +## Bring a verl config -Use strict OPD, shared-backbone role switching and padded trajectory updates on -one CUDA GPU. +Import the resolved, documented OPD subset with field-by-field classifications. ```bash -miniverl train recipes/qwen_consumer_gpu_shared.yaml --dry-run --json +miniverl import-verl --profile verl-opd-v0.8-single-gpu-v1 \ + --config verl-opd.yaml --out local-opd.yaml ``` -**Artifact:** a resolved recipe and typed provenance plan before model loading. +**Artifact:** a round-trippable profile and `local-opd.import-report.json`. -**Next:** [Consumer-GPU shared runtime](consumer-runtime/index.md) +**Next:** [Supported field boundary](compatibility.md)
-## Scale out +## Move data and artifacts Convert Parquet, export standard artifacts and inspect the unsupported boundary. ```bash -miniverl bridge doctor exports/my-bundle --json +miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout ``` -**Artifact:** `provenance/compatibility-report.json` with separate readiness -flags; current bundles are not launchable. +**Artifact:** PEFT + Parquet + OPD overrides and separate readiness flags. **Next:** [verl bridge contract](verl-bridge.md) diff --git a/docs/opd-quickstart.md b/docs/opd-quickstart.md index 6985b23..45e4468 100644 --- a/docs/opd-quickstart.md +++ b/docs/opd-quickstart.md @@ -45,3 +45,25 @@ reserved but fails closed in v0.8.0 rather than loading weights unexpectedly. Unsupported settings—including policy-gradient OPD, task rewards, reference KL, multiple teachers, multiple generations, multimodal inputs and every distributed dimension—are rejected rather than reinterpreted. + +## Import and export the same bounded profile + +```bash +miniverl import-verl --profile verl-opd-v0.8-single-gpu-v1 \ + --config verl-opd.yaml --out local-opd.yaml +miniverl run --profile verl-opd-v0.8-single-gpu-v1 \ + --config local-opd.yaml --output runs +miniverl export-verl --run runs/ --target-verl v0.8.0 --out scaleout +miniverl bridge doctor scaleout --require-verl +``` + +A compatible export contains the standard student PEFT adapter, tokenizer +metadata, exact student/teacher identities, original Parquet bytes, the source +config, compiled plan and pure OPD overrides. No reward scaffold is generated. +Validation data is exported only when the source declared it; an empty +`data.val_files` remains empty rather than duplicating training rows. + +The bundle stays `launchable: false` until the exact student and teacher base +snapshots are materialized. A local teacher adapter adds an explicit merge +requirement because the pinned upstream profile does not consume that adapter +path directly. The bridge never claims a distributed verl job ran. diff --git a/docs/overrides/main.html b/docs/overrides/main.html index b2e65ec..739aae4 100644 --- a/docs/overrides/main.html +++ b/docs/overrides/main.html @@ -1,12 +1,12 @@ {% extends "base.html" %} {% block announce %} -
+
Stable documentation
{% endblock %} diff --git a/docs/verl-bridge.md b/docs/verl-bridge.md index b6b2554..7595761 100644 --- a/docs/verl-bridge.md +++ b/docs/verl-bridge.md @@ -21,19 +21,40 @@ claimed. | State | Current value | Meaning | | --- | --- | --- | -| `artifact_bundle_complete` | `true` | PEFT, safetensors, Parquet, config and provenance are present and hashed. | -| `upstream_config_parse_passed` | `false` in a new bundle | Set only by a separate pinned upstream smoke record, never inferred at export time. | -| `model_data_load_smoke_passed` | `false` in a new bundle | The export itself does not load the base snapshot or execute a model. | -| `reward_implementation_complete` | `false` | The generated reward function deliberately fails closed. | -| `launchable` | `false` | Base weights, reward logic and confirmed mappings are incomplete. | +| `artifact_complete` | `true` | Required PEFT, Parquet, config, identity and provenance files are present and hashed. | +| `config_semantics_supported` | `true` for a compatible OPD run | The source is the bounded pure-GKD profile, not a PPO reinterpretation. | +| `student_artifact_loadable` | separate check | Standard PEFT structure and payload are checked independently. | +| `teacher_artifact_loadable` | `false` in a new bundle | Teacher identity is preserved; the exact snapshot is not bundled. | +| `dataset_loadable` | separate check | Every exported Parquet footer and required column is checked. | +| `upstream_parse_passed` | `false` in a new bundle | Set only when doctor recomputes a merge under the exact installed pin. | +| `upstream_tiny_smoke_passed` | `false` | No model execution occurs during export or doctor. | +| `launchable` | `false` | Exact student/teacher snapshots are not materialized in the bundle. | | `distributed_execution_tested` | `false` | No distributed job ran. | -| `algorithm_semantic_parity` | `false` | The target is a PPO/reward scaffold, not a continuation of miniVERL OPD. | +| `algorithm_semantic_parity` | `false` | Conformance is scoped to documented config/loss behavior, not an end-to-end distributed algorithm. | The committed [pinned smoke record](generated/verl-bridge-smoke.json) verifies a specific artifact-only upstream parse/load exercise. It remains separate from the readiness state of a newly exported bundle and from any execution claim. -## Import a resolved profile subset +## Import the executable OPD v2 profile + +The v2 importer consumes the same typed source used by `plan` and `run`; it +does not require a `ToolEnvironment` and does not invent a reward: + +```bash +miniverl import-verl \ + --profile verl-opd-v0.8-single-gpu-v1 \ + --config verl-opd.yaml \ + --set 'data.train_files=["data/train.parquet"]' \ + --out local-opd.yaml +``` + +`local-opd.yaml` is a canonical, round-trippable input to `miniverl run`. +`local-opd.import-report.json` records every source-field classification and +both source/output digests. PG OPD, task rewards, KL penalties, `n>1`, +multi-teacher and distributed semantics remain hard errors. + +## Legacy environment-profile import `import-verl` accepts the documented, resolved field subset—not arbitrary Hydra/OmegaConf or verl YAML. With only a source profile, it writes @@ -156,7 +177,7 @@ miniverl import-verl resolved-verl.yaml \ Every runnable output passes `RunConfig` validation before publication. -## Export a portable bundle +## Export a pure-OPD portable bundle ```bash miniverl export-verl --run runs/ \ @@ -166,30 +187,34 @@ miniverl export-verl --run runs/ \ miniverl bridge doctor exports/ --require-verl ``` -The bundle contains: +For a compatible `verl-opd-v0.8-single-gpu-v1` run, the bundle contains: ```text -model/ adapter_config.json, adapter_model.safetensors, tokenizer metadata, - base-model.json (identity only; base snapshot is not bundled) -data/ train.parquet, val.parquet -recipe/ verl-overrides.yaml, launch.template.sh, REQUIRED_VERL.txt -reward/ reward_or_verifier_scaffold.py (fails closed) -provenance/ source manifest/result, compatibility-report.json, SHA256SUMS +model/ student PEFT adapter, tokenizer metadata, base-model identity +teacher/ teacher identity and adapter/materialization requirements +data/ original train/validation Parquet bytes, without row reordering +recipe/ verl-opd-overrides.yaml, launch.template.sh, REQUIRED_VERL.txt +provenance/ source config, compiled plan, source manifest, + compatibility-report.json, SHA256SUMS README.md ``` -Available source-run response length and learning rate are preserved in the -override file. The miniVERL total-token bound, cycle schedule and environment -identity are preserved in `source_run_values`; they are not relabelled as -equivalent verl intent. Any prompt limit or schedule value inserted for the PPO -scaffold appears in `placeholder_defaults` with `source_run_intent: false`. - -`bridge doctor` verifies pins, standard adapter structure, tokenizer state, -Parquet schema, override structure, reward importability, privacy scopes and -hashes. An `ok` verdict means the artifact checks passed; it still returns -`launchable: false` while the fail-closed reward scaffold remains. The template -script also refuses to proceed without the immutable base snapshot and a -completed reward implementation. +The OPD override explicitly enables distillation, selects +`forward_kl_topk`, disables policy-gradient/task-reward/KL-reward paths and +preserves supported source data, optimizer, rollout and schedule values. Pure +OPD has no reward scaffold. A same-base teacher adapter is recorded but blocks +launch until it is explicitly merged/materialized as a teacher snapshot that +the pinned upstream can consume. + +`bridge doctor` verifies pins, adapter structure, tokenizer state, Parquet +schema, pure-OPD override structure, privacy scopes and hashes. An `ok` verdict +means those local artifact checks passed; it does not mean launchable or +distributed-tested. `launch.template.sh` refuses to proceed without both exact +base snapshots and never emits an unverified distributed launch command. + +Historical `single-gpu-online-distillation-v1` runs continue to export the +legacy PPO/reward scaffold for compatibility. That output remains explicitly +non-launchable and is not relabelled as OPD. ### Tokenizer verification levels diff --git a/release-state.yaml b/release-state.yaml index 138f20e..dc74fbc 100644 --- a/release-state.yaml +++ b/release-state.yaml @@ -17,13 +17,13 @@ # such distinction, which is how its tag shipped a docs selector still # advertising "Stable 0.6.1 / Development 0.6.2.dev0". schema_version: 1 -phase: development +phase: release stable: - version: "0.7.1" - tag: "v0.7.1" - release_commit: "830a4ca5d873bce4cdcc7c43a44d827b096e8c0c" - released_at: "2026-08-11" + version: "0.8.0" + tag: "v0.8.0" + release_commit: pending + released_at: "2026-08-12" development: - version: "0.8.0.dev0" + version: "0.8.0" diff --git a/scripts/publish_verl_opd_compatibility.py b/scripts/publish_verl_opd_compatibility.py new file mode 100644 index 0000000..82ef1e2 --- /dev/null +++ b/scripts/publish_verl_opd_compatibility.py @@ -0,0 +1,57 @@ +"""Publish the pinned OPD field matrix from the typed compiler.""" + +from __future__ import annotations + +import argparse +import hashlib +from collections import Counter +from pathlib import Path +from typing import Any + +from miniverl.bridge.opd_v08 import load_verl_opd_v08 +from miniverl.utils.runs import canonical_json + + +def build_matrix(source: Path) -> dict[str, Any]: + compiled = load_verl_opd_v08(source) + fields = [item.model_dump(mode="json") for item in compiled.compatibility] + return { + "schema_version": 1, + "profile": compiled.profile, + "upstream": compiled.upstream, + "source_fixture": source.as_posix(), + "source_sha256": hashlib.sha256(source.read_bytes()).hexdigest(), + "compiled_digest": compiled.compiled_digest, + "executable": compiled.executable, + "field_count": len(fields), + "classification_counts": dict( + sorted(Counter(item["classification"] for item in fields).items()) + ), + "fields": fields, + "scope": ( + "config conformance for one documented local pure-OPD profile; " + "not full verl compatibility or distributed execution" + ), + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument( + "--source", type=Path, default=Path("examples/verl-opd-v0.8-single-gpu.yaml") + ) + parser.add_argument( + "--out", type=Path, default=Path("docs/generated/verl-opd-v0.8-compatibility.json") + ) + parser.add_argument("--check", action="store_true") + args = parser.parse_args() + rendered = canonical_json(build_matrix(args.source)) + if args.check: + return 0 if args.out.read_text(encoding="utf-8") == rendered else 1 + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(rendered, encoding="utf-8", newline="\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/verify_verl_bridge_smoke.py b/scripts/verify_verl_bridge_smoke.py index a74f178..93b316e 100644 --- a/scripts/verify_verl_bridge_smoke.py +++ b/scripts/verify_verl_bridge_smoke.py @@ -7,7 +7,7 @@ import json from collections.abc import Mapping from pathlib import Path -from typing import Any +from typing import Any, cast from omegaconf import OmegaConf @@ -65,6 +65,7 @@ def verify_smoke(bundle: str | Path, *, out: str | Path) -> dict[str, Any]: official = OmegaConf.to_container(official_config, resolve=False) if not isinstance(official, Mapping): raise ConfigError("official generated PPO config is not a YAML mapping") + official = cast(Mapping[str, Any], official) missing = [field for field in _OFFICIAL_FIELDS if not _has_path(official, field)] if missing: raise ConfigError("official verl config is missing bridge fields: " + ", ".join(missing)) @@ -73,6 +74,7 @@ def verify_smoke(bundle: str | Path, *, out: str | Path) -> dict[str, Any]: exported = OmegaConf.to_container(exported_omegaconf, resolve=False) if not isinstance(exported, Mapping): raise ConfigError("exported verl overrides are not an OmegaConf mapping") + exported = cast(Mapping[str, Any], exported) missing_export_fields = [ field for field in _OFFICIAL_EXPORT_FIELDS if not _has_path(official, field) ] diff --git a/src/miniverl/__init__.py b/src/miniverl/__init__.py index a8e2c20..5e4d376 100644 --- a/src/miniverl/__init__.py +++ b/src/miniverl/__init__.py @@ -14,6 +14,6 @@ from __future__ import annotations -__version__ = "0.8.0.dev0" +__version__ = "0.8.0" __all__ = ["__version__"] diff --git a/src/miniverl/bridge/doctor.py b/src/miniverl/bridge/doctor.py index 89bab97..a6960de 100644 --- a/src/miniverl/bridge/doctor.py +++ b/src/miniverl/bridge/doctor.py @@ -314,8 +314,11 @@ def _check_parquet(root: Path) -> dict[str, Any]: required = {"data_source", "prompt", "ability", "reward_model", "extra_info"} schemas: dict[str, list[str]] = {} rows: dict[str, int] = {} - for split in ("train", "val"): - path = root / "data" / f"{split}.parquet" + files = sorted((root / "data").glob("*.parquet")) + if not files: + return {"status": "fail", "detail": "no Parquet data file is present"} + for path in files: + split = path.stem try: handle = pq.ParquetFile(path) except Exception as exc: @@ -341,6 +344,68 @@ def _check_parquet(root: Path) -> dict[str, Any]: def _check_config(root: Path) -> dict[str, Any]: + opd_path = root / "recipe" / "verl-opd-overrides.yaml" + if opd_path.is_file(): + try: + payload = yaml.safe_load(opd_path.read_text(encoding="utf-8")) + adapter = json.loads( + (root / "model" / "adapter_config.json").read_text(encoding="utf-8") + ) + base = json.loads((root / "model" / "base-model.json").read_text(encoding="utf-8")) + teacher_identity = json.loads( + (root / "teacher" / "teacher-model.json").read_text(encoding="utf-8") + ) + except (OSError, json.JSONDecodeError, yaml.YAMLError) as exc: + return {"status": "fail", "detail": str(exc)} + required_roots = {"data", "actor_rollout_ref", "algorithm", "distillation", "trainer"} + actual = set(payload) if isinstance(payload, dict) else set() + opd_problems: list[str] = [] + if not required_roots.issubset(actual): + opd_problems.append("missing required pure-OPD root") + try: + model = payload["actor_rollout_ref"]["model"] + expected = { + "path": "model/base", + "lora_adapter_path": "model", + "lora_rank": adapter["r"], + "lora_alpha": adapter["lora_alpha"], + "target_modules": adapter["target_modules"], + } + for field, value in expected.items(): + if model.get(field) != value: + opd_problems.append(f"actor_rollout_ref.model.{field}") + loss = payload["distillation"]["distillation_loss"] + semantic_expected = { + "loss_mode": "forward_kl_topk", + "use_task_rewards": False, + "use_policy_gradient": False, + } + for field, value in semantic_expected.items(): + if loss.get(field) != value: + opd_problems.append(f"distillation.distillation_loss.{field}") + if payload["distillation"].get("enabled") is not True: + opd_problems.append("distillation.enabled") + if payload["actor_rollout_ref"]["actor"].get("use_kl_loss") is not False: + opd_problems.append("actor_rollout_ref.actor.use_kl_loss") + if payload["algorithm"].get("use_kl_in_reward") is not False: + opd_problems.append("algorithm.use_kl_in_reward") + teacher = payload["distillation"]["teacher_models"]["teacher_model"] + if teacher.get("model_path") != "teacher/base": + opd_problems.append("distillation.teacher_models.teacher_model.model_path") + if not teacher_identity.get("model_id") or not teacher_identity.get("revision"): + opd_problems.append("teacher/teacher-model.json identity") + if base.get("model_id") != adapter["base_model_name_or_path"]: + opd_problems.append("base-model.json model_id") + if base.get("revision") != adapter["revision"]: + opd_problems.append("base-model.json revision") + except (KeyError, TypeError): + opd_problems.append("invalid pure-OPD handoff structure") + return { + "status": "ok" if not opd_problems else "fail", + "profile": "verl-opd-v0.8-single-gpu-v1", + "roots": sorted(actual), + "model_handoff_problems": opd_problems, + } try: payload = yaml.safe_load( (root / "recipe" / "verl-overrides.yaml").read_text(encoding="utf-8") @@ -388,6 +453,14 @@ def _check_reward(root: Path, *, trust_and_import: bool = False) -> dict[str, An The bundle is untrusted input. Inspection parses the scaffold and never imports it, so a bundle cannot act merely by being diagnosed. """ + if (root / "recipe" / "verl-opd-overrides.yaml").is_file(): + return { + "status": "ok", + "verification_level": "not_applicable_pure_opd", + "implementation_complete": False, + "code_executed": False, + "detail": "task rewards are disabled by the pure-OPD profile", + } path = root / "reward" / "reward_or_verifier_scaffold.py" return inspect_reward_scaffold(path, trust_and_import=trust_and_import) @@ -888,6 +961,13 @@ def _installed_verl() -> dict[str, Any]: #: Facts a bundle can only *assert*. Nothing in a doctor run recomputes them: #: they describe events that happened elsewhere, earlier, on other hardware. _DECLARED_ONLY_CLAIMS = ( + "artifact_complete", + "config_semantics_supported", + "student_artifact_loadable", + "teacher_artifact_loadable", + "dataset_loadable", + "upstream_parse_passed", + "upstream_tiny_smoke_passed", "upstream_config_parse_passed", "model_data_load_smoke_passed", "distributed_execution_tested", @@ -995,7 +1075,13 @@ def _recompute_upstream_smoke( if not generated.is_file(): return {"status": "failed", "reason": "installed verl omits the generated PPO config"} official = OmegaConf.load(generated) - exported = OmegaConf.load(root / "recipe" / "verl-overrides.yaml") + recipe = root / "recipe" + override_path = ( + recipe / "verl-opd-overrides.yaml" + if (recipe / "verl-opd-overrides.yaml").is_file() + else recipe / "verl-overrides.yaml" + ) + exported = OmegaConf.load(override_path) OmegaConf.set_struct(official, True) OmegaConf.merge(official, exported) except Exception as exc: @@ -1146,6 +1232,7 @@ def inspect_bridge_bundle( "artifact_hashes": hashes, "privacy": privacy, "local_smoke_status": local_smoke, + "artifact_complete": not artifact_failed, "artifact_bundle_complete": not artifact_failed, "bundle_declared_claims": declared, "locally_recomputed_checks": recomputed, @@ -1154,10 +1241,18 @@ def inspect_bridge_bundle( # Every flag below reflects what *this* process recomputed. A bundle # cannot raise any of them by describing itself favourably. "upstream_config_parse_passed": upstream["status"] == "passed", - "model_data_load_smoke_passed": upstream["status"] == "passed" + "upstream_parse_passed": upstream["status"] == "passed", + "upstream_tiny_smoke_passed": False, + "model_data_load_smoke_passed": config.get("profile") != "verl-opd-v0.8-single-gpu-v1" + and upstream["status"] == "passed" and not artifact_failed and require_verl, + "config_semantics_supported": config.get("status") == "ok", + "student_artifact_loadable": model.get("status") == "ok", + "teacher_artifact_loadable": False, + "dataset_loadable": parquet.get("status") == "ok", "reward_implementation_complete": bool(reward.get("implementation_complete", False)), + "reward_required": config.get("profile") != "verl-opd-v0.8-single-gpu-v1", "launchable": False, "distributed_execution_tested": False, "algorithm_semantic_parity": False, diff --git a/src/miniverl/bridge/export.py b/src/miniverl/bridge/export.py index 1cf1db1..d6f2862 100644 --- a/src/miniverl/bridge/export.py +++ b/src/miniverl/bridge/export.py @@ -2,6 +2,7 @@ from __future__ import annotations +import copy import hashlib import json import math @@ -62,7 +63,12 @@ def _sha256(path: Path) -> str: def _model_source(run: Path) -> Path: - candidates = (run / "model", run / "exported-adapter", run / "adapter") + candidates = ( + run / "final-peft-adapter", + run / "model", + run / "exported-adapter", + run / "adapter", + ) for candidate in candidates: if all((candidate / name).is_file() for name in _MODEL_REQUIRED): return candidate @@ -270,7 +276,7 @@ def _verl_overrides( } ) - overrides = { + overrides: dict[str, Any] = { "data": { "train_files": ["data/train.parquet"], "val_files": ["data/val.parquet"], @@ -408,6 +414,260 @@ def _write_hashes(root: Path) -> None: write_text(checksum, "\n".join(lines) + "\n") +_OPD_PROFILE = "verl-opd-v0.8-single-gpu-v1" + + +def _opd_run_contract(run: Path) -> tuple[dict[str, Any], dict[str, Any]] | None: + """Return the executable OPD source/profile pair, or select the legacy exporter.""" + report_path = run / "verl-compatibility-report.json" + if not report_path.is_file(): + return None + try: + report = read_json(report_path) + source_path = run / "verl-source-config.json" + source = read_json(source_path) if source_path.is_file() else report.get("source") + except (OSError, json.JSONDecodeError) as exc: + raise ConfigError(f"cannot read source-run OPD compatibility artifacts: {exc}") from exc + if report.get("profile") != _OPD_PROFILE: + return None + if report.get("executable") is not True or not isinstance(source, dict): + raise ConfigError("source run does not carry an executable verl OPD compatibility plan") + required = { + "distillation.distillation_loss.loss_mode": "forward_kl_topk", + "distillation.distillation_loss.use_task_rewards": False, + "distillation.distillation_loss.use_policy_gradient": False, + "actor_rollout_ref.actor.use_kl_loss": False, + "algorithm.use_kl_in_reward": False, + } + for field, expected in required.items(): + if _get(source, field) != expected: + raise ConfigError( + f"OPD export refuses unsupported source semantic {field}={_get(source, field)!r}" + ) + return source, report + + +def _resolve_source_file(run: Path, raw: str) -> Path: + candidate = Path(raw) + for path in (candidate, run / candidate, run.parent / candidate): + if path.is_file(): + return path + raise ConfigError(f"source-run Parquet file is unavailable: {raw}") + + +def _copy_opd_data( + run: Path, source: dict[str, Any], destination: Path +) -> tuple[dict[str, list[str]], dict[str, Any]]: + destination.mkdir() + exported: dict[str, list[str]] = {"train": [], "val": []} + evidence: dict[str, Any] = {} + for split, field in (("train", "data.train_files"), ("val", "data.val_files")): + values = _get(source, field) + if not isinstance(values, list) or not all(isinstance(item, str) for item in values): + raise ConfigError(f"OPD source field {field} must be a list of Parquet paths") + if split == "train" and not values: + raise ConfigError("OPD export requires at least one training Parquet file") + records: list[dict[str, Any]] = [] + for index, raw in enumerate(values): + source_path = _resolve_source_file(run, raw) + name = f"{split}.parquet" if len(values) == 1 else f"{split}-{index:03d}.parquet" + target = destination / name + shutil.copy2(source_path, target) + digest = _sha256(target) + exported[split].append(f"data/{name}") + records.append( + { + "source_index": index, + "bundle_path": f"data/{name}", + "sha256": digest, + "bytes": target.stat().st_size, + } + ) + evidence[split] = records + return exported, evidence + + +def _opd_overrides( + source: dict[str, Any], adapter: dict[str, Any], data_paths: dict[str, list[str]] +) -> dict[str, Any]: + """Preserve the supported source profile while rebasing portable paths.""" + overrides: dict[str, Any] = { + root: copy.deepcopy(source[root]) + for root in ("data", "actor_rollout_ref", "algorithm", "distillation", "trainer") + if root in source + } + data = overrides["data"] + data["train_files"] = data_paths["train"] + data["val_files"] = data_paths["val"] + model = overrides["actor_rollout_ref"]["model"] + model["path"] = "model/base" + model["lora_adapter_path"] = "model" + model["lora_rank"] = adapter["rank"] + model["lora_alpha"] = adapter["alpha"] + model["target_modules"] = adapter["target_modules"] + teacher = overrides["distillation"]["teacher_models"]["teacher_model"] + teacher["model_path"] = "teacher/base" + # The compiler accepts this resource declaration so it can reject or + # reinterpret distributed intent, but the pinned FSDP-generated config has + # no such teacher inference key. It therefore belongs in provenance, not + # in an override file that promises to parse upstream. + teacher.get("inference", {}).pop("pipeline_model_parallel_size", None) + return overrides + + +def _opd_launch_script(adapter: dict[str, Any], teacher: dict[str, Any]) -> str: + return f"""#!/usr/bin/env bash +set -euo pipefail + +# Template only: no distributed verl execution was tested by miniVERL. +BUNDLE_ROOT="$(cd "$(dirname "${{BASH_SOURCE[0]}}")/.." && pwd)" +if [[ ! -f "$BUNDLE_ROOT/model/base/config.json" ]]; then + echo "materialize student base {adapter["base_model"]}@{adapter["revision"]}" >&2 + exit 2 +fi +if [[ ! -f "$BUNDLE_ROOT/teacher/base/config.json" ]]; then + echo "materialize teacher {teacher["model_id"]}@{teacher["revision"]}" >&2 + exit 2 +fi +if [[ "{str(teacher["upstream_materialization_required"]).lower()}" == "true" ]]; then + echo "merge/materialize the recorded teacher adapter into an immutable teacher snapshot" >&2 + exit 2 +fi +echo "All artifact prerequisites are present; review the pinned OPD overrides before launch." >&2 +echo "No distributed launch command is emitted because distributed execution was not tested." >&2 +exit 2 +""" + + +def _opd_bundle_readme() -> str: + return f"""# miniVERL pure-OPD scale-out bundle + +This checksummed bundle preserves a local `{_OPD_PROFILE}` run as standard +PEFT, tokenizer and Parquet artifacts plus pure GKD `forward_kl_topk` +overrides for [`verl {VERL_TAG}`]({VERL_REPOSITORY}/tree/{VERL_TAG}) at +`{VERL_COMMIT}`. It contains no reward scaffold because task rewards are +disabled by contract. + +`recipe/launch.template.sh` remains fail-closed until the exact student and +teacher snapshots are materialized. The bundle has not run distributed verl; +it does not claim full verl compatibility or algorithmic parity beyond the +documented loss/config conformance checks. +""" + + +def _export_opd_bundle( + run: Path, + *, + manifest_path: Path, + model_source: Path, + adapter: dict[str, Any], + source: dict[str, Any], + compatibility: dict[str, Any], + destination: Path, +) -> dict[str, Any]: + student_id = _get(source, "actor_rollout_ref.model.path") + student_revision = _get(source, "miniverl.student_revision") + if student_id != adapter["base_model"] or student_revision != adapter["revision"]: + raise ConfigError("standard student adapter identity differs from the compiled OPD source") + teacher_id = _get(source, "distillation.teacher_models.teacher_model.model_path") + teacher_revision = _get(source, "miniverl.teacher_revision") + if not isinstance(teacher_id, str) or not isinstance(teacher_revision, str): + raise ConfigError("compiled OPD source has no pinned teacher identity") + teacher_adapter_path = _get(source, "miniverl.teacher_adapter.path") + teacher_adapter_revision = _get(source, "miniverl.teacher_adapter.revision") + teacher = { + "model_id": teacher_id, + "revision": teacher_revision, + "materialized_path": "teacher/base", + "status": "identity only; exact snapshot is not bundled", + "adapter": {"path": teacher_adapter_path, "revision": teacher_adapter_revision}, + "upstream_materialization_required": teacher_adapter_path is not None, + } + blockers = [ + "student base snapshot is not bundled", + "teacher base snapshot is not bundled", + "distributed verl execution was not tested", + ] + if teacher_adapter_path is not None: + blockers.append( + "teacher adapter requires an explicit merge/materialization step for upstream verl" + ) + temporary = destination.parent / f".{destination.name}.{uuid.uuid4().hex}.tmp" + temporary.mkdir() + try: + _copy_model(model_source, temporary / "model") + write_json( + temporary / "model/base-model.json", + { + "materialized_path": "model/base", + "model_id": adapter["base_model"], + "revision": adapter["revision"], + "status": "identity only; exact snapshot is not bundled", + }, + ) + teacher_dir = temporary / "teacher" + teacher_dir.mkdir() + write_json(teacher_dir / "teacher-model.json", teacher) + data_paths, data_evidence = _copy_opd_data(run, source, temporary / "data") + overrides = _opd_overrides(source, adapter, data_paths) + recipe = temporary / "recipe" + recipe.mkdir() + write_text( + recipe / "verl-opd-overrides.yaml", + yaml.safe_dump(overrides, sort_keys=False, allow_unicode=True, width=100), + ) + write_text(recipe / "launch.template.sh", _opd_launch_script(adapter, teacher)) + write_text(recipe / "REQUIRED_VERL.txt", required_verl_text()) + provenance = temporary / "provenance" + provenance.mkdir() + write_json( + provenance / "miniverl-manifest.json", portable_payload(read_json(manifest_path)) + ) + write_json(provenance / "source-config.json", portable_payload(source)) + plan_path = run / "local-execution-plan.json" + write_json( + provenance / "compiled-plan.json", + portable_payload(read_json(plan_path)) if plan_path.is_file() else compatibility, + ) + report: dict[str, Any] = { + "schema_version": 2, + "profile": _OPD_PROFILE, + "target_verl": { + "repository": VERL_REPOSITORY, + "tag": VERL_TAG, + "commit": VERL_COMMIT, + }, + "miniverl_version": __version__, + "target_semantics": "pure GKD forward_kl_topk OPD", + "artifact_complete": True, + "artifact_bundle_complete": True, + "config_semantics_supported": True, + "student_artifact_loadable": True, + "teacher_artifact_loadable": False, + "dataset_loadable": True, + "upstream_parse_passed": False, + "upstream_config_parse_passed": False, + "upstream_tiny_smoke_passed": False, + "model_data_load_smoke_passed": False, + "launchable": False, + "distributed_execution_tested": False, + "algorithm_semantic_parity": False, + "reward_required": False, + "launch_blockers": blockers, + "data_round_trip": data_evidence, + "unsupported_semantics": list(_UNSUPPORTED), + "distributed_execution_status": "not tested", + } + write_json(provenance / "compatibility-report.json", report) + write_text(temporary / "README.md", _opd_bundle_readme()) + _write_hashes(temporary) + temporary.replace(destination) + except BaseException: + shutil.rmtree(temporary, ignore_errors=True) + raise + return report + + def export_verl_bundle( run: str | Path, *, @@ -422,6 +682,25 @@ def export_verl_bundle( raise ConfigError(f"miniVERL run is missing manifest.json: {run_path}") model_source = _model_source(run_path) adapter = _adapter_contract(model_source) + opd_contract = _opd_run_contract(run_path) + if opd_contract is not None: + destination = Path(out) + if destination.exists(): + raise ConfigError( + f"export destination already exists: {destination}", + hint="choose a new directory so an earlier verified bundle cannot be mixed in", + ) + destination.parent.mkdir(parents=True, exist_ok=True) + source, compatibility = opd_contract + return _export_opd_bundle( + run_path, + manifest_path=manifest_path, + model_source=model_source, + adapter=adapter, + source=source, + compatibility=compatibility, + destination=destination, + ) source_config, source_config_file = _source_config(run_path) source_run_values = _source_run_values(source_config) overrides, placeholder_defaults = _verl_overrides(adapter, source_config) diff --git a/src/miniverl/bridge/opd_v08.py b/src/miniverl/bridge/opd_v08.py index 747ab54..86f7529 100644 --- a/src/miniverl/bridge/opd_v08.py +++ b/src/miniverl/bridge/opd_v08.py @@ -30,6 +30,7 @@ "load_verl_opd_v08", "load_verl_opd_v08_source", "parse_overrides", + "publish_imported_verl_opd_v08", ] VERL_OPD_V08_PROFILE = "verl-opd-v0.8-single-gpu-v1" @@ -880,3 +881,67 @@ def load_verl_opd_v08_source( overrides=overrides, require_executable=require_executable, ) + + +def publish_imported_verl_opd_v08( + source: str | Path, + *, + out: str | Path, + overrides: Sequence[str] = (), + overwrite: bool = False, +) -> dict[str, Any]: + """Transactionally publish one canonical, round-trippable OPD profile family.""" + from miniverl.bridge.publish import ( + OutputTransaction, + import_output_targets, + reject_source_output_alias, + ) + + source_path = Path(source) + targets = import_output_targets(out) + reject_source_output_alias({"source config": source_path}, targets) + compiled = load_verl_opd_v08(source_path, overrides=overrides) + rendered = yaml.safe_dump( + compiled.source.model_dump(mode="python"), + sort_keys=False, + allow_unicode=True, + width=100, + ).encode("utf-8") + # Prove the exact bytes about to be published remain executable without + # relying on the first in-memory model instance. + reparsed = yaml.safe_load(rendered) + validated = compile_verl_opd_v08(reparsed) + report = { + "schema_version": 1, + "status": "accepted", + "profile": VERL_OPD_V08_PROFILE, + "target_verl": {"tag": VERL_TAG, "commit": VERL_COMMIT}, + "source_config_sha256": hashlib.sha256(source_path.read_bytes()).hexdigest(), + "generated_profile_sha256": hashlib.sha256(rendered).hexdigest(), + "generated_profile_validated": validated.executable, + "environment_required": False, + "compiled_digest": compiled.compiled_digest, + "round_trip_compiled_digest": validated.compiled_digest, + "field_classification": [item.model_dump(mode="json") for item in compiled.compatibility], + "generated_path": targets["recipe"].name, + "report_path": targets["report"].name, + "claim": ( + "Runnable only through the documented pure-OPD single-GPU profile; " + "not arbitrary verl YAML or distributed execution." + ), + } + transaction = OutputTransaction( + targets=targets, + stem=targets["recipe"].stem, + lock_root=targets["recipe"].parent, + overwrite=overwrite, + ) + transaction.begin() + try: + transaction.write_bytes("recipe", rendered) + transaction.write_json("report", report) + transaction.discard("template") + transaction.commit() + finally: + transaction.close() + return report diff --git a/src/miniverl/cli.py b/src/miniverl/cli.py index 0dc7ad4..35f1f6e 100644 --- a/src/miniverl/cli.py +++ b/src/miniverl/cli.py @@ -1399,10 +1399,14 @@ def eval_command( @app.command("import-verl") def import_verl_command( - source: Path = typer.Argument(..., help="Pinned verl YAML configuration."), + source: Optional[Path] = typer.Argument(None, help="Pinned verl YAML configuration."), + config: Optional[Path] = typer.Option( + None, "--config", help="Resolved verl YAML configuration (v2 spelling)." + ), profile: str = typer.Option(..., "--profile", help="Documented bridge profile."), - target_verl: str = typer.Option(..., "--target-verl", help="Pinned verl tag or commit."), + target_verl: str = typer.Option("v0.8.0", "--target-verl", help="Pinned verl tag or commit."), out: Path = typer.Option(..., "--out", help="New miniVERL recipe path."), + overrides: list[str] = typer.Option([], "--set", help="Repeatable dotted key=value override."), environment: Optional[str] = typer.Option( None, "--environment", help="Explicit registered miniVERL environment." ), @@ -1427,20 +1431,44 @@ def import_verl_command( ) -> None: """Import the documented whitelist, never generic verl YAML.""" try: - from miniverl.bridge.config import import_verl_config - - report = import_verl_config( - source, - profile=profile, - target_verl=target_verl, - out=out, - environment=environment, - teacher_model=teacher_model, - teacher_adapter=teacher_adapter, - loss_profile=loss_profile, - schedule_mapping=schedule_mapping, - overwrite=overwrite, - ) + if source is not None and config is not None: + raise ConfigError("pass the verl YAML once, as a positional path or --config") + selected_source = config or source + if selected_source is None: + raise ConfigError("a resolved verl YAML path is required", hint="pass --config FILE") + from miniverl.bridge.opd_v08 import VERL_OPD_V08_PROFILE + + if profile == VERL_OPD_V08_PROFILE: + from miniverl.bridge.contract import validate_target_verl + from miniverl.bridge.opd_v08 import publish_imported_verl_opd_v08 + + validate_target_verl(target_verl) + report = publish_imported_verl_opd_v08( + selected_source, + out=out, + overrides=overrides, + overwrite=overwrite, + ) + else: + if overrides: + raise ConfigError( + "--set is supported by the verl OPD v2 profile only", + hint=f"use --profile {VERL_OPD_V08_PROFILE}", + ) + from miniverl.bridge.config import import_verl_config + + report = import_verl_config( + selected_source, + profile=profile, + target_verl=target_verl, + out=out, + environment=environment, + teacher_model=teacher_model, + teacher_adapter=teacher_adapter, + loss_profile=loss_profile, + schedule_mapping=schedule_mapping, + overwrite=overwrite, + ) except MiniVerlError as exc: _fail(exc) return @@ -1530,7 +1558,7 @@ def export_verl_command( out: Path = typer.Option(..., "--out", help="New scale-out bundle directory."), as_json: bool = typer.Option(False, "--json", help="Emit machine-readable JSON."), ) -> None: - """Export a self-checking miniVERL-defined Level-3 artifact bundle.""" + """Export a self-checking OPD bundle or the versioned legacy bridge profile.""" try: from miniverl.bridge.export import export_verl_bundle @@ -1594,9 +1622,10 @@ def bridge_doctor_command( ), as_json: bool = typer.Option(False, "--json", help="Emit machine-readable JSON."), ) -> None: - """Verify pins, standard artifacts, schema, scaffold, hashes and smoke status. + """Verify pins, standard artifacts, schema, semantics, hashes and smoke status. - Reward code is statically inspected and never executed by default. + Legacy reward code is statically inspected and never executed by default; + pure OPD bundles have no reward code. """ from miniverl.bridge.doctor import inspect_bridge_bundle diff --git a/tests/cli/test_verl_bridge_cli.py b/tests/cli/test_verl_bridge_cli.py index 979a96c..4abc8f4 100644 --- a/tests/cli/test_verl_bridge_cli.py +++ b/tests/cli/test_verl_bridge_cli.py @@ -7,6 +7,7 @@ from typer.testing import CliRunner from miniverl.bridge.contract import BRIDGE_PROFILE, VERL_TAG +from miniverl.bridge.opd_v08 import VERL_OPD_V08_PROFILE from miniverl.cli import app @@ -241,6 +242,39 @@ def test_import_verl_cli_explicit_contract_writes_a_valid_recipe(tmp_path: Path) assert out.is_file() +def test_import_verl_v2_writes_a_round_trippable_prompt_opd_profile(tmp_path: Path) -> None: + from miniverl.bridge.opd_v08 import load_verl_opd_v08 + + source = Path("examples/verl-opd-v0.8-single-gpu.yaml") + out = tmp_path / "local-opd.yaml" + result = CliRunner().invoke( + app, + [ + "import-verl", + "--config", + str(source), + "--profile", + VERL_OPD_V08_PROFILE, + "--set", + 'data.train_files=["train.parquet"]', + "--out", + str(out), + "--json", + ], + ) + + assert result.exit_code == 0, result.output + payload = json.loads(result.stdout) + assert payload["status"] == "accepted" + assert payload["generated_profile_validated"] is True + assert payload["environment_required"] is False + report = json.loads((tmp_path / "local-opd.import-report.json").read_text(encoding="utf-8")) + assert report["compiled_digest"] == payload["compiled_digest"] + round_trip = load_verl_opd_v08(out) + assert round_trip.executable is True + assert round_trip.source.data.train_files == ["train.parquet"] + + def test_benchmark_export_community_exact_command_needs_no_training_stack( tmp_path: Path, ) -> None: diff --git a/tests/unit/test_verl_opd_compatibility_artifact.py b/tests/unit/test_verl_opd_compatibility_artifact.py new file mode 100644 index 0000000..4a8fed0 --- /dev/null +++ b/tests/unit/test_verl_opd_compatibility_artifact.py @@ -0,0 +1,21 @@ +from __future__ import annotations + +import json +from pathlib import Path + + +def test_published_opd_compatibility_matrix_is_compiler_bound() -> None: + from miniverl.utils.runs import canonical_json + from scripts.publish_verl_opd_compatibility import build_matrix + + source = Path("examples/verl-opd-v0.8-single-gpu.yaml") + artifact = Path("docs/generated/verl-opd-v0.8-compatibility.json") + expected = canonical_json(build_matrix(source)) + + assert artifact.read_text(encoding="utf-8") == expected + payload = json.loads(expected) + assert payload["profile"] == "verl-opd-v0.8-single-gpu-v1" + assert payload["field_count"] >= 70 + assert payload["executable"] is True + assert payload["upstream"]["commit"] == "7aed6b230776f963fa09509c10d9c3a767d1102c" + assert "distributed execution" in payload["scope"] diff --git a/tests/unit/test_verl_opd_export.py b/tests/unit/test_verl_opd_export.py new file mode 100644 index 0000000..d3f4b58 --- /dev/null +++ b/tests/unit/test_verl_opd_export.py @@ -0,0 +1,248 @@ +from __future__ import annotations + +import hashlib +import importlib.metadata +import json +import struct +from pathlib import Path + +import pyarrow as pa +import pyarrow.parquet as pq +import pytest +import yaml + + +def _safetensors_bytes() -> bytes: + header = json.dumps( + {"weight": {"dtype": "F32", "shape": [1], "data_offsets": [0, 4]}}, + separators=(",", ":"), + ).encode() + header += b" " * ((8 - len(header) % 8) % 8) + return struct.pack(" tuple[Path, Path, Path]: + run = tmp_path / "run" + model = run / "final-peft-adapter" + model.mkdir(parents=True) + (run / "manifest.json").write_text(json.dumps({"status": "complete"}), encoding="utf-8") + (model / "adapter_config.json").write_text( + json.dumps( + { + "peft_type": "LORA", + "base_model_name_or_path": "Qwen/Qwen3-0.6B", + "revision": "c1899de289a04d12100db370d81485cdf75e47ca", + "target_modules": ["q_proj", "v_proj"], + "r": 8, + "lora_alpha": 16, + } + ), + encoding="utf-8", + ) + (model / "adapter_model.safetensors").write_bytes(_safetensors_bytes()) + (model / "tokenizer_config.json").write_text( + json.dumps({"tokenizer_class": "Qwen2Tokenizer"}), encoding="utf-8" + ) + rows = [ + { + "data_source": "unit", + "prompt": [{"role": "user", "content": f"prompt-{index}"}], + "ability": "chat", + "reward_model": None, + "extra_info": {"row_identity": index}, + } + for index in range(3) + ] + train = tmp_path / "source-train.parquet" + val = tmp_path / "source-val.parquet" + pq.write_table(pa.Table.from_pylist(rows), train) + pq.write_table(pa.Table.from_pylist(rows[:1]), val) + source = { + "data": { + "train_files": [str(train)], + "val_files": [str(val)], + "prompt_key": "prompt", + "train_batch_size": 2, + "max_prompt_length": 128, + "max_response_length": 16, + "filter_overlong_prompts": True, + "truncation": "error", + "shuffle": False, + "seed": 7, + }, + "actor_rollout_ref": { + "model": { + "path": "Qwen/Qwen3-0.6B", + "enable_gradient_checkpointing": True, + "lora_rank": 8, + "lora_alpha": 16, + "target_modules": ["q_proj", "v_proj"], + }, + "actor": { + "optim": {"lr": 1e-5, "weight_decay": 0.0, "lr_warmup_steps": 0}, + "loss_agg_mode": "token-mean", + "use_kl_loss": False, + "ppo_mini_batch_size": 2, + }, + "rollout": { + "name": "hf", + "n": 1, + "temperature": 0.0, + "top_p": 1.0, + "tensor_model_parallel_size": 1, + }, + }, + "algorithm": {"use_kl_in_reward": False}, + "distillation": { + "enabled": True, + "n_gpus_per_node": 1, + "nnodes": 1, + "teacher_models": { + "teacher_model": { + "model_path": "Qwen/Qwen3-1.7B", + "num_replicas": 1, + "inference": { + "name": "hf", + "dtype": "bfloat16", + "tensor_model_parallel_size": 1, + "data_parallel_size": 1, + "pipeline_model_parallel_size": 1, + }, + } + }, + "distillation_loss": { + "loss_mode": "forward_kl_topk", + "topk": 32, + "use_task_rewards": False, + "distillation_loss_coef": 1.0, + "log_prob_min_clamp": -10.0, + "use_policy_gradient": False, + }, + }, + "trainer": { + "project_name": "mini-verl", + "experiment_name": "opd-export", + "save_freq": 1, + "test_freq": 0, + "total_training_steps": 1, + "n_gpus_per_node": 1, + "nnodes": 1, + }, + "miniverl": { + "student_revision": "c1899de289a04d12100db370d81485cdf75e47ca", + "teacher_revision": "70d244cc86ccca08cf5af4e1e306ecf908b1ad5e", + "runtime": {"mode": "dual_model"}, + "teacher_adapter": { + "path": "teacher-adapter" if teacher_adapter else None, + "revision": "a" * 40 if teacher_adapter else None, + }, + }, + } + (run / "verl-source-config.json").write_text(json.dumps(source), encoding="utf-8") + (run / "verl-compatibility-report.json").write_text( + json.dumps( + { + "profile": "verl-opd-v0.8-single-gpu-v1", + "executable": True, + "compiled_digest": "b" * 64, + "source": source, + } + ), + encoding="utf-8", + ) + (run / "local-execution-plan.json").write_text( + json.dumps({"profile": "verl-opd-v0.8-single-gpu-v1", "compiled_digest": "b" * 64}), + encoding="utf-8", + ) + return run, train, val + + +def test_pure_opd_export_is_reward_free_and_preserves_data_bytes(tmp_path: Path) -> None: + from miniverl.bridge.contract import VERL_TAG + from miniverl.bridge.doctor import inspect_bridge_bundle + from miniverl.bridge.export import export_verl_bundle + + run, train, val = _opd_run(tmp_path) + out = tmp_path / "bundle" + report = export_verl_bundle(run, target_verl=VERL_TAG, out=out) + + assert (out / "recipe/verl-opd-overrides.yaml").is_file() + assert not (out / "recipe/verl-overrides.yaml").exists() + assert not (out / "reward").exists() + assert (out / "teacher/teacher-model.json").is_file() + assert (out / "provenance/source-config.json").is_file() + assert (out / "provenance/compiled-plan.json").is_file() + assert (out / "data/train.parquet").read_bytes() == train.read_bytes() + assert (out / "data/val.parquet").read_bytes() == val.read_bytes() + + overrides = yaml.safe_load((out / "recipe/verl-opd-overrides.yaml").read_text()) + assert overrides["distillation"]["enabled"] is True + loss = overrides["distillation"]["distillation_loss"] + assert loss == { + "loss_mode": "forward_kl_topk", + "topk": 32, + "use_task_rewards": False, + "distillation_loss_coef": 1.0, + "log_prob_min_clamp": -10.0, + "use_policy_gradient": False, + } + assert overrides["actor_rollout_ref"]["actor"]["use_kl_loss"] is False + assert overrides["algorithm"]["use_kl_in_reward"] is False + assert report["artifact_complete"] is True + assert report["config_semantics_supported"] is True + assert report["student_artifact_loadable"] is True + assert report["teacher_artifact_loadable"] is False + assert report["dataset_loadable"] is True + assert report["upstream_parse_passed"] is False + assert report["upstream_tiny_smoke_passed"] is False + assert report["launchable"] is False + assert report["distributed_execution_tested"] is False + assert report["target_semantics"] == "pure GKD forward_kl_topk OPD" + assert ( + report["data_round_trip"]["train"][0]["sha256"] + == hashlib.sha256(train.read_bytes()).hexdigest() + ) + + diagnosis = inspect_bridge_bundle(out) + assert diagnosis["verdict"] == "ok" + assert diagnosis["config_profile"]["profile"] == "verl-opd-v0.8-single-gpu-v1" + assert diagnosis["reward_verification_level"] == "not_applicable_pure_opd" + assert diagnosis["launchable"] is False + + +def test_teacher_adapter_requires_upstream_materialization(tmp_path: Path) -> None: + from miniverl.bridge.contract import VERL_TAG + from miniverl.bridge.export import export_verl_bundle + + run, _, _ = _opd_run(tmp_path, teacher_adapter=True) + report = export_verl_bundle(run, target_verl=VERL_TAG, out=tmp_path / "bundle") + + assert report["teacher_artifact_loadable"] is False + assert report["launchable"] is False + assert "teacher adapter" in " ".join(report["launch_blockers"]).lower() + identity = json.loads( + (tmp_path / "bundle/teacher/teacher-model.json").read_text(encoding="utf-8") + ) + assert identity["adapter"]["path"] == "teacher-adapter" + assert identity["upstream_materialization_required"] is True + + +@pytest.mark.verl_conformance +def test_pure_opd_overrides_parse_under_the_pinned_upstream_config(tmp_path: Path) -> None: + try: + importlib.metadata.distribution("verl") + except importlib.metadata.PackageNotFoundError: + pytest.skip("official verl v0.8.0 is not installed") + from miniverl.bridge.contract import VERL_TAG + from miniverl.bridge.doctor import inspect_bridge_bundle + from miniverl.bridge.export import export_verl_bundle + + run, _, _ = _opd_run(tmp_path) + out = tmp_path / "bundle" + export_verl_bundle(run, target_verl=VERL_TAG, out=out) + diagnosis = inspect_bridge_bundle(out, require_verl=True) + + assert diagnosis["upstream_parse_passed"] is True, diagnosis["upstream_config_parse_recheck"] + assert diagnosis["config_semantics_supported"] is True + assert diagnosis["model_data_load_smoke_passed"] is False + assert diagnosis["distributed_execution_tested"] is False From 6147d16f7623e7a99ba50902edf62ee80ba9af46 Mon Sep 17 00:00:00 2001 From: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com> Date: Wed, 12 Aug 2026 01:31:24 -0700 Subject: [PATCH 2/2] Record v0.8.0 release candidate validation --- docs/generated/quality.json | 34 +++++++++++++++++----------------- docs/release-checklist.md | 26 +++++++++++++++++++++++--- 2 files changed, 40 insertions(+), 20 deletions(-) diff --git a/docs/generated/quality.json b/docs/generated/quality.json index bd2c2e0..28dd7fd 100644 --- a/docs/generated/quality.json +++ b/docs/generated/quality.json @@ -1,22 +1,22 @@ { "schema_version": 2, - "release": "0.7.1", - "status": "released", - "quality_floor": "2,000+ tests and 85%+ branch coverage at v0.7.1", + "release": "0.8.0", + "status": "candidate", + "quality_floor": "2,000+ tests and 85%+ branch coverage at v0.8.0", "local_validation": { "scope": "the maintainer's workstation, where the GPU and Windows-specific paths actually run", - "commit": "5142ed681a9c538b3175faf64561968a43cab547", - "commit_relationship": "implementation commit plus following validation-record-only updates; the final pull-request head is validated by CI", - "measured_at": "2026-08-11T03:05:00-07:00", + "commit": "4fe229ebd87d2c6ea2e6007033f4fe0cb7877c98", + "commit_relationship": "exact implementation commit plus following validation-record-only updates; the final pull-request head is validated by CI", + "measured_at": "2026-08-12T01:18:27-07:00", "platform": "Windows 11 Pro 10.0.22631", "python": "CPython 3.12", "coverage_mode": "branch", "cpu_non_gpu_non_network": { - "passed": 2110, - "skipped": 6, + "passed": 2178, + "skipped": 9, "deselected": 21, - "branch_coverage_percent": 86.19, - "skip_reason": "symlink creation requires privileges on Windows; hard-link and case aliases cover the same guard" + "branch_coverage_percent": 85.3, + "skip_reason": "six platform/privilege skips plus three pinned-verl conformance skips while the official package was intentionally absent from the general environment; the conformance tests were run separately against the pin" }, "gpu": { "passed": 8, @@ -28,15 +28,15 @@ }, "release_validation": { "scope": "the exact published commit, validated by CI rather than locally", - "commit": "830a4ca5d873bce4cdcc7c43a44d827b096e8c0c", + "commit": "pending", "workflows": { - "ci": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31566406949", - "build": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31566406885", - "docs": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31566406859", - "pinned_verl_bridge": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31566406861", - "release": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31566663507" + "ci": null, + "build": null, + "docs": null, + "pinned_verl_bridge": null, + "release": null }, - "conclusion": "success", + "conclusion": "pending", "gpu_coverage": "none; no GPU runner is configured for this repository, so the GPU counts above exist only from the local measurement" } } diff --git a/docs/release-checklist.md b/docs/release-checklist.md index 7a729e7..71d5c9d 100644 --- a/docs/release-checklist.md +++ b/docs/release-checklist.md @@ -6,9 +6,24 @@ after the exact release commit and its remote checks are green. ## v0.8.0 single-GPU verl OPD pivot -- [ ] Implement and validate the documented verl v0.8 single-GPU OPD subset. -- [ ] Preserve every frozen benchmark and keep unsupported distributed semantics fail-closed. -- [ ] Complete the v0.8.0 release gates and public-distribution verification. +- [x] Implement and validate the documented `verl-opd-v0.8-single-gpu-v1` + subset against official verl `v0.8.0` commit `7aed6b23`: typed config, + Parquet prompts, current-policy rollout/teacher/update, token-mean + `forward_kl_topk`, plan/run, PEFT output and OPD import/export. +- [x] Preserve every frozen benchmark and keep policy-gradient, task-reward, + multi-teacher, multimodal and distributed semantics fail-closed. The + calculator benchmark remains SHA-256 `53fc1d4d5b7adee09618d77ad62d4086ba56b78569832d6fc7c3bcd5c2695bbc`. +- [x] Complete the v0.8.0 candidate gates; public-distribution verification is + a post-tag invariant recorded under **After the tag**. + +Candidate `4fe229ebd87d2c6ea2e6007033f4fe0cb7877c98` plus validation-record-only +changes passed Ruff, format, mypy, actionlint, strict MkDocs, Markdown/text and +generated-artifact checks; 2,178 CPU tests at 85.30% branch coverage, 8 GPU +tests, 15 network tests and 3 pinned-verl conformance tests. Four Playwright +viewports checked 36 rendered SVG instances and the screenshots were manually +reviewed. The wheel/sdist passed Twine, clean core and `[train]` installs, +pip-only OPD sample/plan/dry-run, and an extracted-sdist test/rebuild whose 142 +package-file inventory matched the repository wheel. ## v0.7.1 Product correction @@ -102,6 +117,11 @@ unauthorized after checkpoint-selection failure**. ## After the tag +- [ ] Verify the exact v0.8.0 merge-commit release workflow, OIDC publication, + PyPI attestations, public clean install and GitHub Release. +- [ ] Record identical public wheel/sdist hashes and advance main to + `0.8.1.dev0` in a separate state-sync PR. + - [x] v0.7.1 release run [`31566663507`](https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31566663507) completed OIDC Trusted Publishing, attestation verification, exact clean