Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Current maintainer handoff for **miniVERL** (`mini-verl` package, `miniverl`
CLI). `release-state.yaml` is the canonical version source; this page indexes
current product and evidence state rather than repeating release history.

Last updated: 2026-08-13.
Last updated: 2026-08-14.

Canonical release state: stable `v0.9.1` (`6c0f3d818c10419e0bfba81f3ad1c5adf24eaf09`), development `0.10.0.dev0`.

Expand All @@ -31,13 +31,12 @@ Executable compatibility claims are mutation-tested and recorded in
unknown-size quantized roles require proof instead of receiving an executable
plan.

Development `0.10.0.dev0` now has a closed typed profile registry and
torch-free compatibility introspection. New profile-scoped plans, caches,
checkpoints and exports bind an independent identity; the only registered
executable profile remains the measured direct-GKD path until PG-k1 conformance
and runtime work is complete.
Development `0.10.0.dev0` has a closed typed profile registry and torch-free
compatibility introspection. Profile-scoped plans, caches, checkpoints and
exports bind an independent identity. The direct-GKD and sampled-k1 vanilla
policy-loss profiles both have pinned conformance and measured RTX 4080 paths.

Arbitrary verl YAML, policy-gradient OPD, rewards, PPO/GRPO, Ray, FSDP,
Arbitrary verl YAML, other policy-gradient modes, rewards, PPO/GRPO, Ray, FSDP,
Megatron, multi-GPU and distributed execution remain unsupported. The legacy
environment/PPO artifact bridge is migration-only.

Expand All @@ -46,7 +45,9 @@ environment/PPO artifact bridge is migration-only.
| Evidence | Status |
| --- | --- |
| Qwen3-0.6B/1.7B developer workload | 32 prompts, 8 current-policy updates, 3.1914 GiB peak reserved on one RTX 4080; matched interruption/resume was byte-identical |
| SmolLM2-360M/1.7B | one pinned compatibility smoke; not a second measured recipe |
| Qwen3 sampled-k1 PG | 32 prompts, 8 updates, exact interruption/resume, 3.1914 GiB peak reserved; no quality comparison |
| SmolLM2-360M/1.7B direct GKD | 32 prompts, 8 updates, 1.4961 GiB peak reserved; exact resume, PEFT reload and materialized export passed |
| Ubuntu 26.04 WSL2 | plan, bounded probe, rollout, teacher scoring, one update and PEFT reload measured on the same RTX 4080 |
| external alignment v1 | preregistered early stop: 0 selected checkpoints, teachers, continuation arms or final-test accesses |
| distributed verl execution | not tested |

Expand Down
6 changes: 4 additions & 2 deletions PYPI.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,8 +163,10 @@ and update times were 9.7200, 0.4864 and 2.3260 seconds. A matched 4-update
interruption resumed to the same byte-identical trajectories, adapter and
optimizer tensors. See the [data-bound figure and full record](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-opd-reference-workload.md);
the original one-update [pip smoke](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/opd-quickstart.md) remains preserved.
A separate pinned SmolLM2-360M/1.7B compatibility smoke completed one full
rollout/scoring/update cycle; it is not a second measured recipe.
A separate pinned SmolLM2-360M/1.7B recipe consumed 32 distinct prompts across
8 updates at 1.4961 GiB peak reserved VRAM. Interruption/resume was
byte-identical, PEFT reload passed, and the exact-snapshot scale-out bundle
materialized successfully. See the [full SmolLM2 systems record](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/smollm2-opd-workload.md).

This is deliberately a runtime and artifact proof. It is not a throughput
benchmark, an alignment-quality endpoint, or evidence that OPD beats SFT, DPO
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,8 +163,10 @@ and update times were 9.7200, 0.4864 and 2.3260 seconds. A matched 4-update
interruption resumed to the same byte-identical trajectories, adapter and
optimizer tensors. See the [data-bound figure and full record](docs/verl-opd-reference-workload.md);
the original one-update [pip smoke](docs/opd-quickstart.md) remains preserved.
A separate pinned SmolLM2-360M/1.7B compatibility smoke completed one full
rollout/scoring/update cycle; it is not a second measured recipe.
A separate pinned SmolLM2-360M/1.7B recipe consumed 32 distinct prompts across
8 updates at 1.4961 GiB peak reserved VRAM. Interruption/resume was
byte-identical, PEFT reload passed, and the exact-snapshot scale-out bundle
materialized successfully. See the [full SmolLM2 systems record](docs/smollm2-opd-workload.md).

This is deliberately a runtime and artifact proof. It is not a throughput
benchmark, an alignment-quality endpoint, or evidence that OPD beats SFT, DPO
Expand Down
5 changes: 3 additions & 2 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,9 @@ Qwen3-0.6B/1.7B developer workload 实际消费 **32 个不同 prompt**,respon
匹配的运行在第 4 次 update 后中断并恢复;最终 trajectory、adapter 与 optimizer tensor
均字节一致。详见[数据绑定图与完整记录](docs/verl-opd-reference-workload.md);原始一次更新的
[pip smoke](docs/opd-quickstart.md)仍完整保留。
另一个固定版本的 SmolLM2-360M/1.7B 兼容性 smoke 完成了一次完整的
rollout/scoring/update 循环;它不是第二套完整实测 recipe。
另一套固定版本的 SmolLM2-360M/1.7B recipe 消费 32 个不同 prompt 并完成 8 次
update,peak reserved VRAM 为 1.4961 GiB;中断恢复保持字节一致,PEFT 重载与精确
snapshot 的 scale-out 物化均通过。详见[完整 SmolLM2 系统记录](docs/smollm2-opd-workload.md)。

这只证明一个运行时与产物路径,不是吞吐 benchmark、对齐质量 endpoint,也不证明 OPD
优于 SFT、DPO 或 KD。其他 NVIDIA GPU 使用相同的 device-name-agnostic CUDA 路径,但
Expand Down
180 changes: 180 additions & 0 deletions benchmarks/results/rtx4080-smollm2-opd-developer-v1.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
{
"artifacts": {
"checkpoint_hashes": {
"adapter.safetensors": "7d00e5e99ac19ae8c35055e6dee527b737c3af3504b846feef60a0e60d21856d",
"optimizer.safetensors": "1b3473085336cca8b41458c300df368979cb625520462a9152a4d49031b6dd12",
"state.json": "a8f459ce233bd59c011ed31f9b9b308d10572865a2227e0f6e77c04d2de56307"
},
"standard_peft_adapter_sha256": "6132c9aa94b828deed70d03108805beece3392e50409395423a8ae092aac3e4f",
"standard_peft_load_verified": true,
"total_workload_bytes": 4214630242,
"trajectory_sha256": "f8137936991a17d1531352534eb6e5dd4428e2495a0f5de0b9c68fc21d63bb43"
},
"hardware": {
"bitsandbytes": "0.50.0",
"cuda_runtime": "13.0",
"driver": "596.49",
"gpu": "NVIDIA GeForce RTX 4080",
"gpu_count": 1,
"peft": "0.18.0",
"torch": "2.13.0+cu130",
"transformers": "5.14.1",
"vram_gib": 15.992
},
"kind": "single_gpu_smollm2_direct_gkd_developer_workload",
"measured_at": "2026-08-14T07:46:06Z",
"measurements": {
"adapter_bytes": 6825692,
"batch_downshifts": {
"rollout_oom": 0,
"update_chunk_oom": 0
},
"cache_bytes": 81819,
"checkpoint_bytes": 9912101,
"cold_startup_seconds": 8.4132,
"distinct_prompts_consumed": 32,
"observed_rollout_physical_batch_sizes": [
4
],
"peak_allocated_gib": 1.4254,
"peak_reserved_gib": 1.4961,
"prompts_consumed": 32,
"steady_state_median_seconds": {
"actor_update": 1.127,
"rollout": 16.3067,
"teacher_scoring": 0.1941
},
"steady_state_median_throughput": {
"rollout_tokens_per_second": 13.7173,
"teacher_scored_positions_per_second": 1190.38,
"update_positions_per_second": 186.9733
},
"time_to_first_rollout_seconds": 21.6392,
"time_to_first_teacher_targets_seconds": 21.8723,
"time_to_first_update_seconds": 23.1082,
"total_run_bytes": 27178611
},
"miniverl_version": "0.10.0.dev0",
"models": {
"student": {
"adapter": null,
"attn_implementation": "sdpa",
"dtype": "auto",
"gradient_checkpointing": true,
"lora": {
"alpha": 16,
"bias": "none",
"dropout": 0.0,
"enabled": true,
"r": 8,
"target_modules": [
"q_proj",
"v_proj"
]
},
"model_id": "HuggingFaceTB/SmolLM2-360M-Instruct",
"prepare_kbit_training": true,
"quantization": "nf4",
"revision": "a10cc1512eabd3dde888204e902eca88bddb4951",
"tokenizer_id": null,
"tokenizer_revision": null,
"toy": {
"hidden_size": 64,
"intermediate_size": 128,
"max_position_embeddings": 1024,
"num_heads": 4,
"num_layers": 2
},
"trust_remote_code": false
},
"teacher": {
"adapter": null,
"attn_implementation": "sdpa",
"dtype": "bfloat16",
"mode": "standard",
"model_id": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
"quantization": "nf4",
"revision": "31b70e2e869a7173562077fd711b654946d38674",
"tokenizer_id": null,
"tokenizer_revision": null,
"toy": {
"hidden_size": 64,
"intermediate_size": 128,
"max_position_embeddings": 1024,
"num_heads": 4,
"num_layers": 2
},
"toy_pretrain_lr": 0.003,
"toy_pretrain_steps": 0,
"toy_teacher_seed": 99,
"trust_remote_code": false
}
},
"profile": "verl-opd-v0.8-single-gpu-v1",
"profile_identity": {
"digest": "cc30bae03b65fe1bb915990fbd0359bbff436332b1b44c4887bd77ae8e0e4c35",
"export_version": "verl-opd-export-v1",
"field_rule_digest": "8479344d1d033974b2bbc3a8212c0a01bcc54afbe697ded1987794b7a458ee97",
"loss_conformance_version": "forward-kl-topk-verl-v0.8-v1",
"native_compiler_version": "direct-gkd-native-v1",
"profile_name": "verl-opd-v0.8-single-gpu-v1",
"profile_schema_version": 1,
"upstream_commit": "7aed6b230776f963fa09509c10d9c3a767d1102c",
"upstream_repository": "https://github.com/verl-project/verl",
"upstream_tag": "v0.8.0"
},
"recipe": {
"compiled_plan_sha256": "d3ff5ab77f2c57d89a2fb52a04a842cbc06b1c2ca5350a82313aadb6131d6067",
"input_parquet_sha256": "baa99ba8c59d4c30b2e3081346860e7e8b2665dd70fdbcd7952935545a0cd770",
"interrupt_after_updates": 4,
"logical_batch": 4,
"optimizer_updates": 8,
"prompt_limit": 128,
"prompts_available": 64,
"prompts_consumed": 32,
"response_limit": 64,
"teacher_target": "top_k_ids_log_probs_and_mass",
"top_k": 32
},
"resource_contract": {
"gpu_hour_limit": 3.0,
"peak_reserved_limit_gib": 14.5,
"peak_reserved_within_limit": true,
"total_gpu_workload_seconds": 301.27
},
"resume": {
"adapter_and_optimizer_byte_identical": true,
"excluded_run_identity_field": "resolved_config_digest",
"global_optimizer_steps": 8,
"interrupt_checkpoint_bytes": 9912101,
"resume_load_seconds": 3.1039,
"status": "exact_match",
"task_cursor": 32,
"training_state_fields_identical": true,
"trajectories_byte_identical": true
},
"scaleout": {
"artifact_bundle_complete": true,
"bundle_bytes": 4160177169,
"distributed_execution_tested": false,
"launchable": true,
"model_data_load_smoke_passed": true,
"upstream_config_parse_passed": true
},
"schema_version": 1,
"scientific_scope": {
"algorithm_comparison": false,
"alignment_quality_evaluated": false,
"runtime_correctness_only": true,
"task_quality_evaluated": false
},
"source_commit": "6cf11cfacb455c872559dc1ca48d928594556b15",
"status": "maintainer_measured",
"verl": {
"commit": "7aed6b230776f963fa09509c10d9c3a767d1102c",
"distributed_execution_tested": false,
"repository": "https://github.com/verl-project/verl",
"tag": "v0.8.0"
},
"workload_driver_sha256": "f249766c774e1b44e09ff323d22f548aa5ca9bfa782f54bfbf9ca8782ce59fcd"
}
71 changes: 71 additions & 0 deletions docs/evidence/wsl2-rtx4080-smollm2-opd-v1.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
{
"schema_version": 1,
"kind": "wsl2_single_gpu_opd_smoke",
"status": "maintainer_measured",
"measured_at": "2026-08-14T07:49:21Z",
"source_commit": "6cf11cfacb455c872559dc1ca48d928594556b15",
"miniverl_version": "0.10.0.dev0",
"profile": "verl-opd-v0.8-single-gpu-v1",
"profile_identity_digest": "cc30bae03b65fe1bb915990fbd0359bbff436332b1b44c4887bd77ae8e0e4c35",
"platform": {
"distribution": "Ubuntu 26.04 LTS",
"kernel": "6.18.33.2-microsoft-standard-WSL2",
"python": "3.12.13",
"gpu": "NVIDIA GeForce RTX 4080",
"vram_gib": 15.992,
"driver": "596.49",
"torch": "2.13.0+cu130",
"cuda_runtime": "13.0",
"transformers": "5.15.0",
"bitsandbytes": "0.50.1",
"peft": "0.20.0"
},
"models": {
"student": {
"id": "HuggingFaceTB/SmolLM2-360M-Instruct",
"revision": "a10cc1512eabd3dde888204e902eca88bddb4951",
"quantization": "nf4"
},
"teacher": {
"id": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
"revision": "31b70e2e869a7173562077fd711b654946d38674",
"quantization": "nf4"
}
},
"probe": {
"status": "measured",
"parameter_updates": 0,
"duration_seconds": 38.648,
"peak_reserved_gib": 1.0449
},
"execution": {
"plan_digest": "a7e941448a73562235a5b05ad8294dcfff5c508c1424adfbddf908523bb9075e",
"prompts": 1,
"response_limit": 64,
"generated_tokens": 42,
"optimizer_updates": 1,
"time_to_first_update_seconds": 9.5106,
"rollout_tokens_per_second": 12.84,
"teacher_scored_positions_per_second": 313.15,
"update_positions_per_second": 75.1401,
"peak_allocated_gib": 1.3808,
"peak_reserved_gib": 1.4043,
"peft_export_and_reload": "passed"
},
"artifacts": {
"checkpoint_adapter_sha256": "0ad396b576ab89bba894407331bedff529272dbc80a10d5b73935f42f0d444de",
"checkpoint_optimizer_sha256": "b41f8da8946c3d6691bbf9266866a95a7fbd0e76a0d927a56f802749c4d4cd0e",
"trajectory_sha256": "0adbe928a45d5aba8c2c85234fcbba246b4c1bbd1c48397c4f6ea584d701d3e3",
"peft_adapter_sha256": "9429bc79db18ac773a65c7f6a776eef45f9275f7a7f5c783ccd10286d5127c55"
},
"scope": {
"plan_completed": true,
"probe_completed": true,
"rollout_completed": true,
"teacher_scoring_completed": true,
"optimizer_update_completed": true,
"peft_reload_completed": true,
"task_quality_evaluated": false,
"distributed_execution_tested": false
}
}
10 changes: 6 additions & 4 deletions docs/for-verl-users.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# For verl users

miniVERL is a local runtime for one documented subset of verl v0.8 OPD. It
miniVERL is a local runtime for two documented subsets of verl v0.8 OPD. It
keeps familiar field names and Parquet data, then compiles distributed resource
intent into sequential phases on one CUDA GPU. It is an independent project;
the mapping is explicit and does not imply endorsement or full compatibility.
Expand All @@ -19,16 +19,18 @@ by plans, caches, checkpoints and exports.

- A resolved YAML using the `verl-opd-v0.8-single-gpu-v1` field subset.
- Reward-free verl-style Parquet prompts with structured chat messages.
- One actor, one teacher, `n=1`, forward top-k GKD and token-mean aggregation.
- One actor, one teacher, `n=1`, token-mean aggregation, and either direct
forward-top-k GKD or the sampled-k1 vanilla policy-loss profile.
- Immutable Hugging Face revisions, PEFT adapters and tokenizer snapshots.
- Familiar fields such as `actor_rollout_ref.model.path`,
`distillation.teacher_models.teacher_model.model_path`, response bounds,
learning rate and LoRA configuration.

What is not reusable: arbitrary Hydra composition inside miniVERL, shell launch scripts,
resource pools, Ray actors, FSDP/Megatron checkpoints, PPO/GRPO, critics,
policy-gradient OPD, task-reward mixtures, multiple teachers and multimodal
workers. Unsupported semantics fail closed instead of falling back silently.
policy-gradient modes beyond the closed sampled-k1 profile, task-reward
mixtures, multiple teachers and multimodal workers. Unsupported semantics fail
closed instead of falling back silently.

## Command mapping

Expand Down
6 changes: 3 additions & 3 deletions docs/release-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@ after the exact release commit and its remote checks are green.
## v0.10.0 development

- [x] Begin from the verified v0.9.1 release and `0.10.0.dev0` canonical state.
- [ ] Add the versioned internal compatibility-profile registry and profile
- [x] Add the versioned internal compatibility-profile registry and profile
identity binding without an unrestricted plugin loader.
- [ ] Add the pinned verl v0.8 policy-gradient k1 profile with scalar, gradient,
- [x] Add the pinned verl v0.8 policy-gradient k1 profile with scalar, gradient,
freshness, resume and export conformance.
- [ ] Publish bounded RTX 4080 systems evidence for PG k1 and a full SmolLM2
- [x] Publish bounded RTX 4080 systems evidence for PG k1 and a full SmolLM2
direct-GKD developer recipe; do not add a task-quality benchmark.
- [ ] Add validated community hardware records and document the measured or
explicitly unmeasured Linux/WSL state.
Expand Down
Loading