Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,22 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

### Measured developer workload

- Published a checksummed RTX 4080 systems workload over 32 distinct consumed
prompts, 64-token responses and eight QLoRA updates. The data-bound figure
reports steady-state phase time, labelled throughput and 3.1914 GiB peak
reserved VRAM without a task-quality or method-comparison claim.
- Fixed Parquet resume to reconstruct the saved row/epoch cursor. A real
Qwen3 interruption after update four now reproduces uninterrupted
trajectories, adapter and optimizer tensors byte for byte; all training
state fields match apart from the intentionally run-specific resolved-config
digest.
- Qualified the Apache-2.0 SmolLM2-360M/1.7B pair with a pinned one-update
compatibility smoke covering tokenizer identity, rollout, teacher scoring,
actor update and PEFT reload. This is not a second full recipe or quality
benchmark.

### Immutable plan/run workflow

- Added deterministic `plan --out plan.json` artifacts that bind the source
Expand Down
6 changes: 6 additions & 0 deletions PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,12 @@ adapter, software and output provenance. `launchable: true` means only that the
pinned upstream entry point has complete local inputs; distributed execution
remains `false` and algorithm parity remains unclaimed.

A second-family compatibility smoke used pinned Apache-2.0 SmolLM2-360M and
SmolLM2-1.7B snapshots. Structural tokenizer identity matched; one strict
forward-top-k rollout, teacher-score, update and PEFT reload completed on the
RTX 4080 at 1.416 GiB peak reserved VRAM. This is a tiny compatibility check,
not a second supported recipe or task-quality result.

## v0.8.1 product surface

The landing pages now lead with the documented one-GPU verl-style OPD journey,
Expand Down
17 changes: 10 additions & 7 deletions PYPI.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,17 +144,20 @@ arbitrary launch script.

## Measured RTX 4080 path

The packaged Qwen3-0.6B/1.7B smoke completed two 16-token rollouts and one OPD
update on one RTX 4080 with **3.1758 GiB peak reserved VRAM**. The first update
completed in **12.0224 seconds**, the final adapter was exported as standard
PEFT/safetensors, and a clean reload passed. The recipe records immutable model
revisions, phase timings, cache identity, checkpoint bytes and adapter hashes.
The Qwen3-0.6B/1.7B developer workload consumed **32 distinct prompts**, each
with a 64-token response bound, and completed **8 current-policy updates** at
**3.1914 GiB peak reserved VRAM**. Median steady-state rollout, teacher-scoring
and update times were 9.7200, 0.4864 and 2.3260 seconds. A matched 4-update
interruption resumed to the same byte-identical trajectories, adapter and
optimizer tensors. See the [data-bound figure and full record](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-opd-reference-workload.md);
the original one-update [pip smoke](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/opd-quickstart.md) remains preserved.
A separate pinned SmolLM2-360M/1.7B compatibility smoke completed one full
rollout/scoring/update cycle; it is not a second measured recipe.

This is deliberately a runtime and artifact proof. It is not a throughput
benchmark, an alignment-quality endpoint, or evidence that OPD beats SFT, DPO
or KD. Other NVIDIA GPUs use the same device-name-agnostic CUDA path, but model
fit depends on VRAM, context length, quantization and installed kernels. Read
the [exact smoke record and limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/opd-quickstart.md).
fit depends on VRAM, context length, quantization and installed kernels.

### Choose a path by hardware, not GPU branding

Expand Down
17 changes: 10 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,17 +144,20 @@ arbitrary launch script.

## Measured RTX 4080 path

The packaged Qwen3-0.6B/1.7B smoke completed two 16-token rollouts and one OPD
update on one RTX 4080 with **3.1758 GiB peak reserved VRAM**. The first update
completed in **12.0224 seconds**, the final adapter was exported as standard
PEFT/safetensors, and a clean reload passed. The recipe records immutable model
revisions, phase timings, cache identity, checkpoint bytes and adapter hashes.
The Qwen3-0.6B/1.7B developer workload consumed **32 distinct prompts**, each
with a 64-token response bound, and completed **8 current-policy updates** at
**3.1914 GiB peak reserved VRAM**. Median steady-state rollout, teacher-scoring
and update times were 9.7200, 0.4864 and 2.3260 seconds. A matched 4-update
interruption resumed to the same byte-identical trajectories, adapter and
optimizer tensors. See the [data-bound figure and full record](docs/verl-opd-reference-workload.md);
the original one-update [pip smoke](docs/opd-quickstart.md) remains preserved.
A separate pinned SmolLM2-360M/1.7B compatibility smoke completed one full
rollout/scoring/update cycle; it is not a second measured recipe.

This is deliberately a runtime and artifact proof. It is not a throughput
benchmark, an alignment-quality endpoint, or evidence that OPD beats SFT, DPO
or KD. Other NVIDIA GPUs use the same device-name-agnostic CUDA path, but model
fit depends on VRAM, context length, quantization and installed kernels. Read
the [exact smoke record and limitations](docs/opd-quickstart.md).
fit depends on VRAM, context length, quantization and installed kernels.

### Choose a path by hardware, not GPU branding

Expand Down
14 changes: 9 additions & 5 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,14 +120,18 @@ Megatron、多 GPU 与多节点均不支持。已知不支持值会得到机器

## RTX 4080 实测路径

打包的 Qwen3-0.6B/1.7B smoke 在一张 RTX 4080 上完成两条 16-token rollout 与一次
OPD update,**peak reserved VRAM 为 3.1758 GiB**,首次 update 在 **12.0224 秒**完成;
标准 PEFT/safetensors adapter 导出与干净重载均通过。recipe 同时记录不可变模型 revision、
阶段计时、cache 身份、checkpoint 字节数与 adapter 哈希。
Qwen3-0.6B/1.7B developer workload 实际消费 **32 个不同 prompt**,response 上限为
64 token,完成 **8 次 current-policy update**,**peak reserved VRAM 为 3.1914 GiB**。
稳态 rollout、teacher scoring 与 update 的中位耗时分别为 9.7200、0.4864 与 2.3260 秒。
匹配的运行在第 4 次 update 后中断并恢复;最终 trajectory、adapter 与 optimizer tensor
均字节一致。详见[数据绑定图与完整记录](docs/verl-opd-reference-workload.md);原始一次更新的
[pip smoke](docs/opd-quickstart.md)仍完整保留。
另一个固定版本的 SmolLM2-360M/1.7B 兼容性 smoke 完成了一次完整的
rollout/scoring/update 循环;它不是第二套完整实测 recipe。

这只证明一个运行时与产物路径,不是吞吐 benchmark、对齐质量 endpoint,也不证明 OPD
优于 SFT、DPO 或 KD。其他 NVIDIA GPU 使用相同的 device-name-agnostic CUDA 路径,但
能否装下仍取决于显存、上下文、量化与 kernel。见[精确实测记录](docs/opd-quickstart.md)。
能否装下仍取决于显存、上下文、量化与 kernel。

### 按硬件条件选择路径,而不是按显卡名称

Expand Down
83 changes: 83 additions & 0 deletions benchmarks/results/rtx4080-smollm2-opd-family-smoke-v1.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
{
"schema_version": 1,
"kind": "single_gpu_opd_model_family_compatibility_smoke",
"status": "measured",
"measured_at": "2026-08-13T07:48:14Z",
"source_commit": "2819f7290ffb0c14a0ff5d13509a0fec29c8100d",
"miniverl_version": "0.9.0.dev0",
"profile": "verl-opd-v0.8-single-gpu-v1",
"models": {
"student": {
"id": "HuggingFaceTB/SmolLM2-360M-Instruct",
"revision": "a10cc1512eabd3dde888204e902eca88bddb4951",
"license": "apache-2.0",
"quantization": "nf4",
"adapter": "lora-r8-alpha16"
},
"teacher": {
"id": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
"revision": "31b70e2e869a7173562077fd711b654946d38674",
"license": "apache-2.0",
"quantization": "nf4"
}
},
"tokenizer_identity": {
"tokenizer_class": "GPT2Tokenizer",
"length": 49152,
"structural_digest_v2": "4bf67fa8fdbf333c2f79c00bdc43909ed2ffb4bd7ed1f44a11d65f171ea9d1c5",
"behavioral_fingerprint_v1": "5772ce854a388a7613d1284a34fa0360f9c603256796cb1f72e25713ab82f7b5",
"student_teacher_match": true
},
"workload": {
"distinct_prompts": 1,
"response_limit": 16,
"logical_batch": 1,
"optimizer_updates": 1,
"top_k": 32,
"loss": "forward_kl_topk",
"runtime_strategy": "dual_model_resident"
},
"measurements": {
"construction_seconds": 8.5539,
"time_to_first_rollout_seconds": 10.084,
"time_to_first_teacher_targets_seconds": 10.1989,
"time_to_first_update_seconds": 10.7449,
"rollout_tokens_per_second": 10.46,
"teacher_scored_positions_per_second": 139.24,
"update_positions_per_second": 35.5365,
"selected_positions": 16,
"peak_allocated_gib": 1.384,
"peak_reserved_gib": 1.416,
"oom_retries": 0,
"checkpoint_bytes": 19798324,
"run_disk_bytes": 26507663
},
"artifacts": {
"adapter_sha256": "1eda290c7e19b55d9d497ec014711fa451c67595525ad5aa304167fd1be189a6",
"peft_adapter_model_sha256": "90bf81a33c6c23acb87c93a1df53caeb5e9b91aa72e5678edfe9832b60716aec",
"peft_load_verified": true
},
"runtime": {
"gpu": "NVIDIA GeForce RTX 4080",
"gpu_count": 1,
"vram_gib": 15.992,
"torch": "2.13.0+cu130",
"cuda_runtime": "13.0",
"transformers": "5.14.1",
"peft": "0.18.0",
"bitsandbytes": "0.50.0",
"distributed_execution_tested": false
},
"scope": {
"config_compiled": true,
"tokenizer_identity_verified": true,
"student_loaded": true,
"teacher_loaded": true,
"rollout_completed": true,
"teacher_scoring_completed": true,
"optimizer_update_completed": true,
"task_quality_evaluated": false,
"alignment_quality_evaluated": false,
"full_recipe_supported": false
}
}
122 changes: 122 additions & 0 deletions benchmarks/results/rtx4080-verl-opd-developer-v1.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
{
"artifacts": {
"checkpoint_hashes": {
"adapter.safetensors": "60e2eced54a3a13d5d66c765b1a9f45d26b2f16bf982fa0011f9316b0cc0433f",
"optimizer.safetensors": "0bb7dde497204151b952d1e5866059822334f71150bfe4a5ce46e253cc84f494",
"state.json": "e92abdf28f18f07c2ebb7c49a712b806c9462ad422bfeb99fb4cce037a248962"
},
"standard_peft_adapter_sha256": "085b9c1241acaf0a2ab2c6c9081484b77eafbd329d92c7a6181ddfa3709bfe0c",
"standard_peft_load_verified": true,
"trajectory_sha256": "f0d7e256e977508d698c4db506f1c37f42188b91b27c372bce1dad8bcf6d54d0",
"workload_output_bytes": 130233537
},
"hardware": {
"bitsandbytes": "0.50.0",
"cuda_runtime": "13.0",
"driver": "596.49",
"gpu": "NVIDIA GeForce RTX 4080",
"gpu_count": 1,
"peft": "0.18.0",
"torch": "2.13.0+cu130",
"transformers": "5.14.1",
"vram_gib": 15.992
},
"kind": "single_gpu_opd_developer_workload",
"measured_at": "2026-08-13T07:31:03Z",
"measurements": {
"adapter_bytes": 9213148,
"batch_downshifts": {
"rollout_oom": 0,
"update_chunk_oom": 0
},
"cache_bytes": 90727,
"checkpoint_bytes": 27649609,
"cold_startup_seconds": 9.0802,
"distinct_prompts_consumed": 32,
"observed_rollout_physical_batch_sizes": [
4
],
"peak_allocated_gib": 2.3033,
"peak_reserved_gib": 3.1914,
"prompts_consumed": 32,
"steady_state_median_seconds": {
"actor_update": 2.326,
"rollout": 9.72,
"teacher_scoring": 0.4864
},
"steady_state_median_throughput": {
"rollout_tokens_per_second": 26.3374,
"teacher_scored_positions_per_second": 526.32,
"update_positions_per_second": 110.0602
},
"time_to_first_rollout_seconds": 19.0015,
"time_to_first_teacher_targets_seconds": 19.5584,
"time_to_first_update_seconds": 22.0412,
"total_run_bytes": 65065194
},
"miniverl_version": "0.9.0.dev0",
"models": {
"student": {
"adapter": "lora-r8-alpha16",
"id": "Qwen/Qwen3-0.6B",
"quantization": "nf4",
"revision": "c1899de289a04d12100db370d81485cdf75e47ca"
},
"teacher": {
"id": "Qwen/Qwen3-1.7B",
"quantization": "nf4",
"revision": "70d244cc86ccca08cf5af4e1e306ecf908b1ad5e"
}
},
"profile": "verl-opd-v0.8-single-gpu-v1",
"recipe": {
"compiled_plan_sha256": "9b2b0b522585e5a88603f83f3ce714080a1560d92940d9d5dd661fe586dc1a5b",
"dataset_rows": 64,
"distinct_dataset_prompts": 64,
"distinct_prompts_consumed": 32,
"input_parquet_sha256": "baa99ba8c59d4c30b2e3081346860e7e8b2665dd70fdbcd7952935545a0cd770",
"logical_batch": 4,
"optimizer_updates": 8,
"prompt_limit": 128,
"prompts_consumed": 32,
"response_limit": 64,
"rollout_physical_batch": 4,
"runtime_strategy": "dual_model_resident",
"teacher_score_batch": 4,
"top_k": 32,
"update_physical_batch": 1
},
"resource_contract": {
"gpu_hour_limit": 4.0,
"peak_reserved_limit_gib": 14.5,
"peak_reserved_within_limit": true,
"total_gpu_workload_seconds": 235.49
},
"resume": {
"adapter_and_optimizer_byte_identical": true,
"excluded_run_identity_field": "resolved_config_digest",
"global_optimizer_steps": 8,
"interrupt_after_optimizer_updates": 4,
"interrupt_checkpoint_bytes": 27649609,
"resume_load_seconds": 8.2339,
"status": "exact_match",
"task_cursor": 32,
"training_state_fields_identical": true,
"trajectories_byte_identical": true
},
"schema_version": 1,
"scientific_scope": {
"alignment_quality_evaluated": false,
"opd_beats_sft_dpo_or_kd_claimed": false,
"runtime_correctness_only": true,
"task_quality_evaluated": false
},
"source_commit": "2819f7290ffb0c14a0ff5d13509a0fec29c8100d",
"status": "measured",
"verl": {
"commit": "7aed6b230776f963fa09509c10d9c3a767d1102c",
"distributed_execution_tested": false,
"tag": "v0.8.0"
},
"workload_driver_sha256": "00356532e8d1ea6a488340bffc42acf59300e9c130d483866645233e5cc823af"
}
4 changes: 4 additions & 0 deletions docs/opd-quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ for run artifacts. The plan command itself is CPU-only and weight-free; use

## Measured reference

The v0.9 developer workload extends this install smoke to 32 distinct consumed
prompts, 64 response tokens and eight optimizer updates. See its
[phase-time, throughput, VRAM and resume evidence](verl-opd-reference-workload.md).

| GPU | student / teacher | strategy | limits / top-k | peak reserved | first update | status |
| --- | --- | --- | --- | ---: | ---: | --- |
| RTX 4080 16 GiB | Qwen3-0.6B / Qwen3-1.7B, both NF4 | dual resident | 128 + 16 tokens / 32 | 3.176 GiB | 12.02 s | measured |
Expand Down
58 changes: 58 additions & 0 deletions docs/verl-opd-reference-workload-mobile.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading