Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/verl-bridge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
changed=$(git diff --name-only "origin/$BASE_REF"...HEAD)
echo "changed files:"
echo "$changed"
if echo "$changed" | grep -Eq '^(\.github/workflows/verl-bridge\.yml|pyproject\.toml|scripts/[^/]*verl_bridge[^/]*|src/miniverl/bridge/.*|src/miniverl/losses/verl_topk\.py|tests/.*verl_bridge.*|tests/conformance/test_verl_v08_loss\.py)$'; then
if echo "$changed" | grep -Eq '^(\.github/workflows/verl-bridge\.yml|pyproject\.toml|scripts/[^/]*verl_bridge[^/]*|src/miniverl/bridge/.*|src/miniverl/losses/verl_topk\.py|tests/.*verl_(bridge|opd).*|tests/conformance/test_verl_v08_loss\.py)$'; then
echo "bridge=true" >> "$GITHUB_OUTPUT"
else
echo "bridge=false" >> "$GITHUB_OUTPUT"
Expand Down Expand Up @@ -75,6 +75,10 @@ jobs:
"git+https://github.com/verl-project/verl.git@7aed6b230776f963fa09509c10d9c3a767d1102c"
- name: Compare forward_kl_topk values, diagnostics and gradients
run: python -m pytest -q tests/conformance/test_verl_v08_loss.py -m verl_conformance
- name: Materialize and recheck a tiny launchable pure-OPD bundle
run: >-
python -m pytest -q tests/conformance/test_verl_v08_materialize.py
-m verl_conformance
- name: Generate and export the standards smoke bundle
run: |
python scripts/prepare_verl_bridge_smoke.py --out _verl-smoke-source
Expand Down
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,20 @@ All notable changes to miniVERL are recorded here. The format follows
tiny rollout/teacher-score/selected-position-backward phases, exact cache
identity, zero optimizer updates and post-release CUDA-memory verification.

### Transactional scale-out materialization

- Added `miniverl bridge materialize` for exact local or downloaded student and
teacher snapshots. It rejects moving revisions and unsafe trees, validates
model/tokenizer/PEFT/Parquet/top-k inputs under the pinned verl commit, hashes
every copied or merged file, and publishes through a rollback-safe staged
directory replacement.
- New pure-OPD exports correctly keep both base-model loadability flags false.
`launch.sh` replaces the fail-closed template only after the pinned upstream
config merge and bounded sequential model-load smoke pass; distributed
execution remains explicitly untested.
- Teacher-adapter merging requires an explicit flag, never mutates the source
base, and records base, adapter, software, output and licensing provenance.

### Verl config UX

- Added safe trailing Hydra-style overrides and repeatable plain/JSON
Expand Down
11 changes: 11 additions & 0 deletions PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,17 @@ Probe caches bind hardware/software, model, tokenizer, quantization, token and
plan identities; cached and fresh publication of the same measurement produces
byte-identical immutable plans.

The scale-out slice adds transactional `bridge materialize`. A fresh export
continues to carry identity-only bases and remains non-launchable. Materialize
accepts exact regular-file snapshots or resolves the recorded 40-character
commits through download/cache mode, validates shard closure, tokenizer and
model loads, student PEFT, Parquet, top-k and the exact installed verl config,
then publishes `launch.sh` and full file hashes through a staged directory
swap. Teacher-adapter merging is an explicit, non-mutating operation with base,
adapter, software and output provenance. `launchable: true` means only that the
pinned upstream entry point has complete local inputs; distributed execution
remains `false` and algorithm parity remains unclaimed.

## v0.8.1 product surface

The landing pages now lead with the documented one-GPU verl-style OPD journey,
Expand Down
9 changes: 5 additions & 4 deletions PYPI.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,15 +189,16 @@ checkpoints, measurements and a PEFT adapter. Inspect before moving it:
miniverl inspect runs/my-opd/trajectories.jsonl
miniverl cache stats runs/my-opd/teacher-cache
miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --json
```

The v0.8.1 export preserves student/teacher identities, Parquet bytes and pure
OPD overrides, but reports `launchable: false` until exact base snapshots are
materialized. Artifact completeness, upstream parse/load smoke, launchability,
algorithm semantics and distributed execution are separate statuses. A
successful bridge check never means that a distributed verl job ran. Review
the [bridge contract](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-bridge.md) and [compatibility policy](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/compatibility.md).
materialized and validated against the installed pinned verl commit. Only then
does `bridge materialize` publish a checksummed `launch.sh`; distributed
execution remains untested. Review the [materialization contract](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/scaleout-materialization.md),
[bridge contract](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-bridge.md) and [compatibility policy](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/compatibility.md).

The intended operating loop is **plan → inspect → run → inspect → export**.
`plan --out` byte-binds the YAML, ordered overrides and scanned Parquet inputs
Expand Down
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,15 +189,16 @@ checkpoints, measurements and a PEFT adapter. Inspect before moving it:
miniverl inspect runs/my-opd/trajectories.jsonl
miniverl cache stats runs/my-opd/teacher-cache
miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --json
```

The v0.8.1 export preserves student/teacher identities, Parquet bytes and pure
OPD overrides, but reports `launchable: false` until exact base snapshots are
materialized. Artifact completeness, upstream parse/load smoke, launchability,
algorithm semantics and distributed execution are separate statuses. A
successful bridge check never means that a distributed verl job ran. Review
the [bridge contract](docs/verl-bridge.md) and [compatibility policy](docs/compatibility.md).
materialized and validated against the installed pinned verl commit. Only then
does `bridge materialize` publish a checksummed `launch.sh`; distributed
execution remains untested. Review the [materialization contract](docs/scaleout-materialization.md),
[bridge contract](docs/verl-bridge.md) and [compatibility policy](docs/compatibility.md).

The intended operating loop is **plan → inspect → run → inspect → export**.
`plan --out` byte-binds the YAML, ordered overrides and scanned Parquet inputs
Expand Down
10 changes: 6 additions & 4 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,13 +158,15 @@ prompt 的 role/content 保持结构化,data source、ability 与 extra metada
miniverl inspect runs/my-opd/trajectories.jsonl
miniverl cache stats runs/my-opd/teacher-cache
miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --json
```

v0.8.1 export 保留 student/teacher 身份、Parquet 原始字节与纯 OPD override;精确 base
snapshot materialize 之前仍报告 `launchable: false`。产物完整性、上游 parse/load smoke、
launchability、算法语义与 distributed execution 分开报告。bridge doctor 通过并不表示运行过
分布式 verl。详见[桥接契约](docs/verl-bridge.md)与[兼容性政策](docs/compatibility.md)。
v0.8.1 export 保留 student/teacher 身份、Parquet 原始字节与纯 OPD override;在精确 base
snapshot 被物化并通过已安装的固定 verl commit 验证前,仍报告 `launchable: false`。只有此后
`bridge materialize` 才会发布带校验和的 `launch.sh`;分布式执行仍是未测试。详见
[物化契约](docs/scaleout-materialization.md)、[桥接契约](docs/verl-bridge.md)与
[兼容性政策](docs/compatibility.md)。

建议操作闭环是 **plan → inspect → run → inspect → export**。`plan --out` 把 YAML、按顺序
应用的 override、扫描后的 Parquet 与精确 native config 绑定;`run --plan` 在加载权重前拒绝
Expand Down
8 changes: 5 additions & 3 deletions docs/for-verl-users.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,11 +99,13 @@ identities never collapse in provenance.

```bash
miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --json
```

The v0.8.1 bundle carries PEFT, Parquet, config and provenance artifacts, but is
reported as `launchable: false` until exact base snapshots are materialized.
reported as `launchable: false` until exact base snapshots are materialized and
the pinned upstream checks pass. Read the [materialization workflow](scaleout-materialization.md).
Upstream parse/load smoke, artifact completeness, launchability and distributed
execution are separate statuses. miniVERL never reports a distributed job as
tested.
Expand All @@ -126,8 +128,8 @@ tested.
for the shared token space; a legacy behavioral fingerprint is not proof.
- **CUDA out of memory:** reduce context, response length or physical batches;
keep logical update semantics unchanged. See [single-GPU planning](single-gpu-guide.md).
- **Bundle not launchable:** this is expected before exact model snapshots are
present. Read the [bridge boundary](verl-bridge.md).
- **Bundle not launchable:** materialize exact snapshots and install the pinned
verl commit; inspect the reported blocker. Read [scale-out materialization](scaleout-materialization.md).

Next: follow the [OPD quickstart](opd-quickstart.md), inspect the
[compatibility policy](compatibility.md), or review [all limitations](limitations.md).
3 changes: 2 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ FSDP, Megatron, PPO, GRPO and distributed launch are outside the runtime.
The executable profile pins verl `v0.8.0` at `7aed6b23`: one actor, one teacher,
`n=1`, GKD `forward_kl_topk`, token-mean and no reward/KL penalty. Unsupported
algorithm or distributed semantics fail closed. Exports remain unlaunchable
until exact base snapshots are materialized.
until exact base snapshots are materialized and checked under the pinned verl
commit. See [scale-out materialization](scaleout-materialization.md).

## Measured runtime evidence

Expand Down
8 changes: 5 additions & 3 deletions docs/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -452,9 +452,11 @@ job. It does not convert optimizer state, distributed RNG, native sharded
checkpoints, Ray runtime state or teacher caches into PPO reference caches.
Unknown and distributed-only config fields fail by default. The
miniVERL-defined label therefore means a validated scale-out bundle, not
runtime parity or generic verl YAML support. Current bundles are not
launchable: the base snapshot is absent, reward logic fails closed and user
mappings remain placeholders. See the [exact contract and evidence](verl-bridge.md).
runtime parity or generic verl YAML support. Legacy PPO/reward-scaffold bundles
remain non-launchable. A pure-OPD bundle can become `launchable: true` only
after exact snapshots, the fixed upstream config merge and bounded local model
loads pass; that status still does not say a distributed invocation succeeded.
See the [exact contract](verl-bridge.md) and [materialization workflow](scaleout-materialization.md).

`models.teacher.mode: privileged_context` works only with environments that
implement `privileged_context()`. All three built-in environments do;
Expand Down
5 changes: 4 additions & 1 deletion docs/opd-quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ miniverl import-verl --profile verl-opd-v0.8-single-gpu-v1 \
miniverl run --profile verl-opd-v0.8-single-gpu-v1 \
--config local-opd.yaml --output runs
miniverl export-verl --run runs/<run-id> --target-verl v0.8.0 --out scaleout
miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --require-verl
```

Expand All @@ -66,4 +67,6 @@ Validation data is exported only when the source declared it; an empty
The bundle stays `launchable: false` until the exact student and teacher base
snapshots are materialized. A local teacher adapter adds an explicit merge
requirement because the pinned upstream profile does not consume that adapter
path directly. The bridge never claims a distributed verl job ran.
path directly. Materialization requires the exact installed verl pin and emits
`launch.sh` only after model/tokenizer/data/config checks pass. The bridge never
claims a distributed verl job ran. See the [materialization contract](scaleout-materialization.md).
106 changes: 106 additions & 0 deletions docs/scaleout-materialization.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Materialize a pinned verl handoff

`export-verl` intentionally starts with identity-only student and teacher base
models. Its PEFT adapter, Parquet data, pure-OPD override and provenance are
complete, but `launchable` remains `false` until the exact model snapshots and
the pinned upstream validation are present.

## Offline-first workflow

The simplest offline path copies the two exact commits from the Hugging Face
cache into a regular-file staging tree:

```bash
miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
miniverl bridge materialize scaleout --download --offline
```

Hugging Face cache snapshots commonly contain symlinks. The materializer will
not follow them from an explicitly supplied directory; download mode resolves
cached bytes into regular staging files without network access.

For a separately copied regular-file snapshot, pass both directories. Each
path must either end in `snapshots/<40-character-commit>` or carry a
`miniverl-snapshot.json` binding the recorded model id, revision and file
hashes:

```bash
miniverl bridge materialize scaleout \
--student-snapshot /snapshots/c1899de289a04d12100db370d81485cdf75e47ca \
--teacher-snapshot /snapshots/70d244cc86ccca08cf5af4e1e306ecf908b1ad5e \
--offline
```

Remove `--offline` only when downloading the two exact recorded commits is
intended. A branch name such as `main` or a tag is never accepted as the bundle
identity.

The command also requires official verl `v0.8.0` installed from pinned commit
`7aed6b230776f963fa09509c10d9c3a767d1102c`. It merges the exported override
into that exact configuration, validates Parquet and PEFT payloads, loads both
local model/tokenizer snapshots sequentially, runs a tiny CPU forward for each
role and verifies the top-k/tokenizer contract. It launches no Ray worker and
performs no distributed training.

## What is published

Materialization stages a complete copy beside the bundle. Snapshot files must
be regular files; model shard indexes must be closed; configs, tokenizer
vocabularies and safetensors structure must validate. Every copied or merged
file is SHA-256 bound in
`provenance/materialization-manifest.json`, then the bundle-wide
`provenance/SHA256SUMS` is regenerated. Only after all checks pass is the old
directory replaced.

Before success:

```text
recipe/launch.template.sh
launchable: false
```

After success:

```text
recipe/verl-opd-resolved.yaml
recipe/launch.sh
launchable: true
distributed_execution_tested: false
```

Here `launchable: true` means the pinned config and local artifacts are
complete enough to invoke the documented upstream entry point. It is not a
claim that the invocation, a distributed job or miniVERL-to-verl end-to-end
algorithm parity was tested. Recheck the bytes and installed pin in the
current process before use:

```bash
miniverl bridge doctor scaleout \
--require-verl --require-tokenizer-load --require-adapter-payload
```

## Teacher adapters

If the local run used a teacher adapter that pinned verl cannot consume in the
recorded role, export includes its standard PEFT payload when available and
materialization remains blocked. Merging requires explicit consent:

```bash
miniverl bridge materialize scaleout --download \
--merge-teacher-adapter
```

The original base is never modified. The merged output is written to a new
teacher snapshot, then reloaded and hashed. Provenance records the base and
adapter identities and hashes, merge software versions, output hashes, and
copied license/notice files. A missing adapter payload or identity mismatch
fails before publication.

## Failure and recovery

Copy, merge, validation and provenance writes happen in a sibling staging
directory. An ordinary exception leaves the existing bundle unchanged and
removes staging files. Final publication uses a same-filesystem directory
rename with in-process rollback. As with any multi-rename update, a power loss
at the final swap is not a cross-platform filesystem transaction; preserve the
source run and exported bundle until the materialized copy is verified.
7 changes: 7 additions & 0 deletions docs/verl-bridge-launch.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# From one GPU to a verified scale-out handoff

!!! note "Historical v0.6 article"

This page preserves the original bridge launch announcement. The current
pure-OPD profile can now publish `launch.sh` after the exact checks in
[scale-out materialization](scaleout-materialization.md); the legacy
PPO/reward scaffold described below remains non-launchable.

miniVERL v0.6 adds a narrow, tested bridge to official verl `v0.8.0`. The goal
is not to imitate a distributed runtime on a laptop. It is to make the boundary
between local scientific diagnosis and later scale-out explicit, standard and
Expand Down
16 changes: 12 additions & 4 deletions docs/verl-bridge.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@ claimed.

| State | Current value | Meaning |
| --- | --- | --- |
| `artifact_complete` | `true` | Required PEFT, Parquet, config, identity and provenance files are present and hashed. |
| `artifact_complete` | `false` in a new bundle | Exact student and teacher base snapshots are still required. |
| `config_semantics_supported` | `true` for a compatible OPD run | The source is the bounded pure-GKD profile, not a PPO reinterpretation. |
| `student_artifact_loadable` | separate check | Standard PEFT structure and payload are checked independently. |
| `student_artifact_loadable` | `false` in a new bundle | PEFT is present, but the exact base snapshot is not yet bundled. |
| `teacher_artifact_loadable` | `false` in a new bundle | Teacher identity is preserved; the exact snapshot is not bundled. |
| `dataset_loadable` | separate check | Every exported Parquet footer and required column is checked. |
| `upstream_parse_passed` | `false` in a new bundle | Set only when doctor recomputes a merge under the exact installed pin. |
| `upstream_tiny_smoke_passed` | `false` | No model execution occurs during export or doctor. |
| `upstream_tiny_smoke_passed` | `false` in a new bundle | Materialization performs the bounded local model/tokenizer smoke. |
| `launchable` | `false` | Exact student/teacher snapshots are not materialized in the bundle. |
| `distributed_execution_tested` | `false` | No distributed job ran. |
| `algorithm_semantic_parity` | `false` | Conformance is scoped to documented config/loss behavior, not an end-to-end distributed algorithm. |
Expand Down Expand Up @@ -184,6 +184,7 @@ miniverl export-verl --run runs/<run-id> \
--target-verl v0.8.0 \
--out exports/<bundle>

miniverl bridge materialize exports/<bundle> --download --offline
miniverl bridge doctor exports/<bundle> --require-verl
```

Expand All @@ -206,11 +207,18 @@ OPD has no reward scaffold. A same-base teacher adapter is recorded but blocks
launch until it is explicitly merged/materialized as a teacher snapshot that
the pinned upstream can consume.

`bridge materialize` resolves the two immutable model commits, copies only a
preflighted regular-file tree, validates model/tokenizer/PEFT/data inputs under
the exact installed verl pin and transactionally publishes a checksummed
`launch.sh`. See [materialization](scaleout-materialization.md).

`bridge doctor` verifies pins, adapter structure, tokenizer state, Parquet
schema, pure-OPD override structure, privacy scopes and hashes. An `ok` verdict
means those local artifact checks passed; it does not mean launchable or
distributed-tested. `launch.template.sh` refuses to proceed without both exact
base snapshots and never emits an unverified distributed launch command.
base snapshots and never emits an unverified distributed launch command. On a
materialized bundle, `doctor --require-verl` recomputes launchability in the
current process; it does not merely trust the bundle's compatibility report.

Historical `single-gpu-online-distillation-v1` runs continue to export the
legacy PPO/reward scaffold for compatibility. That output remains explicitly
Expand Down
Loading