Skip to content

Add typed verl v0.8 OPD config compiler #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DaoyuanLi2816 merged 1 commit into from
Aug 12, 2026
Merged

Add typed verl v0.8 OPD config compiler #58

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,16 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

### verl v0.8 OPD config compiler

- Added the typed `verl-opd-v0.8-single-gpu-v1` configuration profile and an
offline `bridge compile-opd` command. Resolved YAML and repeatable dotted
overrides compile into deterministic field-by-field compatibility reports.
- Unsupported policy-gradient OPD, task-reward mixtures, KL penalties,
multi-generation, multi-teacher and distributed dimensions fail closed.
Engine/resource fields are labelled as local reinterpretations rather than
upstream-exact behavior; the config-only command loads no model weights.

## [0.7.1] - 2026-08-11

### Product correction
Expand Down
13 changes: 13 additions & 0 deletions PROJECT_STATE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,19 @@ Canonical release state: stable `v0.7.1` (`830a4ca5d873bce4cdcc7c43a44d827b096e8
Every public version claim is generated from `release-state.yaml` and gated by
`python scripts/release_state.py --check`.

## v0.8.0 single-GPU verl OPD pivot — PR A

Development starts with the config-only profile
`verl-opd-v0.8-single-gpu-v1`, pinned to verl `v0.8.0` at
`7aed6b230776f963fa09509c10d9c3a767d1102c`. Typed models compile resolved YAML
and repeatable dotted overrides into a deterministic, checksummed field report.
The report distinguishes exact, semantically conformant, locally reinterpreted,
derived, informational-only and unsupported fields. PG OPD, task-reward
mixtures, KL penalties, multiple generations, multiple teachers and
distributed dimensions fail closed. `bridge compile-opd` is deliberately an
offline config compiler: it loads no weights and does not claim the prompt
runtime or verl engine equivalence implemented by later PRs.

## v0.7.1 Product correction — RELEASE CANDIDATE

Branch `v0.7.1-product-correction` starts from synchronized main
Expand Down
90 changes: 90 additions & 0 deletions examples/verl-opd-v0.8-single-gpu.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Resolved fixture for the pinned verl-opd-v0.8-single-gpu-v1 compiler.
# This file compiles offline. Runtime execution is added in later v0.8 PRs.
data:
train_files: [data/train.parquet]
val_files: [data/val.parquet]
prompt_key: prompt
train_batch_size: 8
max_prompt_length: 256
max_response_length: 64
filter_overlong_prompts: false
truncation: error
shuffle: true
seed: 17
actor_rollout_ref:
model:
path: Qwen/Qwen3-0.6B
enable_gradient_checkpointing: true
lora_rank: 16
lora_alpha: 32
target_modules: [q_proj, v_proj]
lora_adapter_path: null
actor:
optim:
lr: 1e-5
weight_decay: 0.01
lr_warmup_steps: 0
loss_agg_mode: token-mean
use_kl_loss: false
ppo_mini_batch_size: 8
ppo_max_token_len_per_gpu: 2048
use_dynamic_bsz: true
rollout:
name: vllm
n: 1
temperature: 1.0
top_p: 0.95
tensor_model_parallel_size: 1
gpu_memory_utilization: 0.5
max_model_len: 320
max_num_batched_tokens: 2048
max_num_seqs: 8
algorithm:
use_kl_in_reward: false
distillation:
enabled: true
teacher_key: data_source
n_gpus_per_node: 1
nnodes: 1
teacher_models:
teacher_model:
model_path: Qwen/Qwen3-1.7B
num_replicas: 1
inference:
name: vllm
dtype: bfloat16
tensor_model_parallel_size: 1
data_parallel_size: 1
pipeline_model_parallel_size: 1
gpu_memory_utilization: 0.5
max_model_len: 321
distillation_loss:
loss_mode: forward_kl_topk
topk: 32
use_task_rewards: false
distillation_loss_coef: 1.0
loss_max_clamp: null
log_prob_min_clamp: -10.0
use_policy_gradient: false
trainer:
project_name: mini-verl
experiment_name: opd-smoke
save_freq: 10
test_freq: 10
total_epochs: 1
total_training_steps: 2
n_gpus_per_node: 1
nnodes: 1
miniverl:
runtime:
mode: auto
memory:
vram_limit_gib: 16
headroom_gib: 1.5
batching:
rollout_batch_size: 2
teacher_score_batch_size: 2
update_trajectory_batch_size: 2
teacher_adapter:
path: null
revision: null
9 changes: 8 additions & 1 deletion src/miniverl/bridge/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,12 @@
VERL_REPOSITORY,
VERL_TAG,
)
from miniverl.bridge.opd_v08 import VERL_OPD_V08_PROFILE

__all__ = ["BRIDGE_PROFILE", "VERL_COMMIT", "VERL_REPOSITORY", "VERL_TAG"]
__all__ = [
"BRIDGE_PROFILE",
"VERL_COMMIT",
"VERL_OPD_V08_PROFILE",
"VERL_REPOSITORY",
"VERL_TAG",
]
Loading