DaoyuanLi2816 · DaoyuanLi2816 · Aug 14, 2026 · Aug 14, 2026 · Aug 14, 2026 · Aug 14, 2026
diff --git a/.github/workflows/verl-bridge.yml b/.github/workflows/verl-bridge.yml
@@ -47,7 +47,7 @@ jobs:
           changed=$(git diff --name-only "origin/$BASE_REF"...HEAD)
           echo "changed files:"
           echo "$changed"
-          if echo "$changed" | grep -Eq '^(\.github/workflows/verl-bridge\.yml|pyproject\.toml|scripts/[^/]*verl_bridge[^/]*|src/miniverl/bridge/.*|src/miniverl/losses/verl_topk\.py|tests/.*verl_(bridge|opd).*|tests/conformance/test_verl_v08_loss\.py)$'; then
+          if echo "$changed" | grep -Eq '^(\.github/workflows/verl-bridge\.yml|pyproject\.toml|scripts/[^/]*verl_bridge[^/]*|src/miniverl/bridge/.*|src/miniverl/losses/verl_(topk|pg)\.py|tests/.*verl_(bridge|opd|pg).*|tests/conformance/test_verl_v08_(loss|pg_k1)\.py)$'; then
             echo "bridge=true" >> "$GITHUB_OUTPUT"
           else
             echo "bridge=false" >> "$GITHUB_OUTPUT"
@@ -75,6 +75,8 @@ jobs:
           "git+https://github.com/verl-project/verl.git@7aed6b230776f963fa09509c10d9c3a767d1102c"
       - name: Compare forward_kl_topk values, diagnostics and gradients
         run: python -m pytest -q tests/conformance/test_verl_v08_loss.py -m verl_conformance
+      - name: Compare sampled k1 policy-loss values, gradients and optimizer step
+        run: python -m pytest -q tests/conformance/test_verl_v08_pg_k1.py -m verl_conformance
       - name: Materialize and recheck a tiny launchable pure-OPD bundle
         run: >-
           python -m pytest -q tests/conformance/test_verl_v08_materialize.py

diff --git a/PYPI.md b/PYPI.md
@@ -130,19 +130,24 @@ explain/check` to inspect the closed built-in registry and distinguish an
 accepted field from one with a demonstrated native effect. New plans, caches,
 checkpoints and exports bind the complete independent profile identity.
 
-`verl-opd-v0.8-single-gpu-v1` pins official verl `v0.8.0` at commit
-`7aed6b230776f963fa09509c10d9c3a767d1102c`. Its executable path is intentionally
-narrow:
+Both profiles pin official verl `v0.8.0` at commit
+`7aed6b230776f963fa09509c10d9c3a767d1102c`:
+
+| Profile | Objective | Teacher target | Status |
+| --- | --- | --- | --- |
+| direct GKD | `forward_kl_topk` | top-k IDs/log-probabilities | measured |
+| PG OPD | sampled `k1` + vanilla policy loss | sampled-token teacher log-probability | measured |
+
+Their shared executable boundary is intentionally narrow:
 
 - one trainable actor and one teacher;
 - one generated response per prompt (`n=1`);
-- reward-free generalized knowledge distillation;
-- `forward_kl_topk` with teacher top-k IDs/log-probabilities, mass/overlap
-  diagnostics and token-mean aggregation;
+- reward-free distillation with token-mean aggregation;
 - LoRA or QLoRA adapter updates on one CUDA GPU;
 - immutable model revisions and verl-style structured prompt Parquet.
 
-Policy-gradient OPD, task rewards, KL penalties, multi-teacher routing,
+The PG path is not PPO: it uses a detached distillation-derived advantage with
+the pinned vanilla policy-loss form. Task rewards, KL penalties, multi-teacher routing,
 multimodal inputs, PPO, GRPO, critics, Ray, FSDP, Megatron, multi-GPU and
 multi-node execution are unsupported. Known unsupported values receive a
 machine-readable classification; unknown fields and unresolved `${...}`

diff --git a/README.md b/README.md
@@ -130,19 +130,24 @@ explain/check` to inspect the closed built-in registry and distinguish an
 accepted field from one with a demonstrated native effect. New plans, caches,
 checkpoints and exports bind the complete independent profile identity.
 
-`verl-opd-v0.8-single-gpu-v1` pins official verl `v0.8.0` at commit
-`7aed6b230776f963fa09509c10d9c3a767d1102c`. Its executable path is intentionally
-narrow:
+Both profiles pin official verl `v0.8.0` at commit
+`7aed6b230776f963fa09509c10d9c3a767d1102c`:
+
+| Profile | Objective | Teacher target | Status |
+| --- | --- | --- | --- |
+| direct GKD | `forward_kl_topk` | top-k IDs/log-probabilities | measured |
+| PG OPD | sampled `k1` + vanilla policy loss | sampled-token teacher log-probability | measured |
+
+Their shared executable boundary is intentionally narrow:
 
 - one trainable actor and one teacher;
 - one generated response per prompt (`n=1`);
-- reward-free generalized knowledge distillation;
-- `forward_kl_topk` with teacher top-k IDs/log-probabilities, mass/overlap
-  diagnostics and token-mean aggregation;
+- reward-free distillation with token-mean aggregation;
 - LoRA or QLoRA adapter updates on one CUDA GPU;
 - immutable model revisions and verl-style structured prompt Parquet.
 
-Policy-gradient OPD, task rewards, KL penalties, multi-teacher routing,
+The PG path is not PPO: it uses a detached distillation-derived advantage with
+the pinned vanilla policy-loss form. Task rewards, KL penalties, multi-teacher routing,
 multimodal inputs, PPO, GRPO, critics, Ray, FSDP, Megatron, multi-GPU and
 multi-node execution are unsupported. Known unsupported values receive a
 machine-readable classification; unknown fields and unresolved `${...}`

diff --git a/README.zh-CN.md b/README.zh-CN.md
@@ -109,17 +109,24 @@ explain/check` 查看封闭的内置 registry，并区分“字段可接受”
 生效”。新生成的 plan、cache、checkpoint 与 export 都绑定完整且独立版本化的 profile
 identity。
 
-`verl-opd-v0.8-single-gpu-v1` 固定官方 verl `v0.8.0` 与 commit
-`7aed6b230776f963fa09509c10d9c3a767d1102c`。可执行边界有意保持狭窄：
+两个 profile 都固定官方 verl `v0.8.0` 与 commit
+`7aed6b230776f963fa09509c10d9c3a767d1102c`：
+
+| Profile | 目标 | Teacher target | 状态 |
+| --- | --- | --- | --- |
+| direct GKD | `forward_kl_topk` | top-k ID/log-probability | 已实测 |
+| PG OPD | sampled `k1` + vanilla policy loss | sampled-token teacher log-probability | 已实测 |
+
+两者的可执行边界有意保持狭窄：
 
 - 一个可训练 actor 与一个 teacher；
 - 每个 prompt 只生成一个 response（`n=1`）；
-- 无 reward 的 generalized knowledge distillation；
-- `forward_kl_topk` 使用 teacher top-k token ID/log-probability、top-k mass/overlap 诊断与 token-mean 聚合；
+- 无 reward 的 distillation 与 token-mean 聚合；
 - 一张 CUDA GPU 上的 LoRA/QLoRA adapter 更新；
 - 不可变模型 revision 与 verl 风格结构化 prompt Parquet。
 
-PG OPD、task reward、KL penalty、多 teacher、多模态、PPO、GRPO、critic、Ray、FSDP、
+PG 路径不是 PPO：它用 detach 的 distillation advantage 和固定 vanilla policy-loss 形式。
+Task reward、KL penalty、多 teacher、多模态、PPO、GRPO、critic、Ray、FSDP、
 Megatron、多 GPU 与多节点均不支持。已知不支持值会得到机器可读分类；未知字段和未解析
 的 `${...}` 会被拒绝。输入必须是 resolved profile 子集，而不是任意启动脚本。
 

diff --git a/docs/adr/0010-verl-v0.8-pg-k1-contract.md b/docs/adr/0010-verl-v0.8-pg-k1-contract.md
@@ -0,0 +1,35 @@
+# ADR 0010: pinned verl v0.8 sampled-k1 policy-gradient contract
+
+Status: accepted for `verl-opd-v0.8-single-gpu-pg-k1-v1`.
+
+## Decision
+
+miniVERL implements one closed, reward-free policy-gradient OPD profile pinned
+to verl `v0.8.0` commit `7aed6b230776f963fa09509c10d9c3a767d1102c`.
+The contract is the composition of upstream `kl_penalty_forward(..., "k1")`
+and `compute_policy_loss_vanilla`: rollout-time actor log probability minus
+teacher log probability forms the sampled k1 estimator, its detached negative
+is the token advantage, and the current/old actor probability ratio is clipped
+with the pinned vanilla settings before token-mean aggregation.
+
+The cache binds sampled token IDs, old actor and teacher log probabilities,
+response mask, actor policy version, teacher/tokenizer identity, profile
+identity and estimator implementation version. A target cannot cross policy
+versions. Each accepted group is generated by the current policy and produces
+exactly one optimizer update.
+
+## Boundary
+
+This is not PPO. It has no task reward, critic, externally supplied advantage,
+reference KL, replay, multiple generations, multi-teacher routing, multimodal
+input or distributed execution. Only `loss_mode: k1`,
+`use_policy_gradient: true`, `policy_loss_mode: vanilla`, `n: 1`, and the
+conformance-tested clipping values are accepted. Other upstream options fail
+closed instead of being approximated.
+
+## Verification
+
+The optional conformance suite loads the exact pinned upstream function bodies
+and compares estimator tensors, detached advantages, scalar loss, metrics,
+actor-logit gradients and a deterministic optimizer step. Runtime evidence is
+systems/conformance evidence only; it makes no task-quality or alignment claim.
diff --git a/docs/evidence/rtx4080-verl-pg-k1-v1.json b/docs/evidence/rtx4080-verl-pg-k1-v1.json
@@ -0,0 +1,171 @@
+{
+  "artifacts": {
+    "checkpoint_hashes": {
+      "adapter.safetensors": "315673b72ab3c871b6225e32981bce21877fca9d63d1c2e8e34d0b4b9295b8e6",
+      "optimizer.safetensors": "972cfa720e394da0435eeffde2a6602d764dfc774060b082e6dbb754e8842ffc",
+      "state.json": "e2b5135ba3bb78c09cdd1d946cd5b2a22456cbec4ff8f4b5bfe364da7e550e61"
+    },
+    "standard_peft_adapter_sha256": "769a2be71627d1e0d78e35aa7131d360d0750e71c8dce89c0b7c354c2f5dee01",
+    "total_workload_bytes": 130048818,
+    "trajectory_sha256": "765d76ed2f16da571d35cd71176f08ffe447b78e7fb94a13e73474ed3b7cb893"
+  },
+  "hardware": {
+    "bitsandbytes": "0.50.0",
+    "cuda_runtime": "13.0",
+    "driver": "596.49",
+    "gpu": "NVIDIA GeForce RTX 4080",
+    "gpu_count": 1,
+    "peft": "0.18.0",
+    "torch": "2.13.0+cu130",
+    "transformers": "5.14.1",
+    "vram_gib": 15.992
+  },
+  "kind": "single_gpu_verl_pg_k1_systems_workload",
+  "measured_at": "2026-08-14T06:51:51Z",
+  "measurements": {
+    "adapter_bytes": 9198518,
+    "batch_downshifts": {
+      "rollout_oom": 0,
+      "update_chunk_oom": 0
+    },
+    "cache_bytes": 26382,
+    "checkpoint_bytes": 27600769,
+    "cold_startup_seconds": 8.1302,
+    "distinct_prompts_consumed": 32,
+    "observed_rollout_physical_batch_sizes": [
+      4
+    ],
+    "peak_allocated_gib": 2.3032,
+    "peak_reserved_gib": 3.1914,
+    "prompts_consumed": 32,
+    "steady_state_median_seconds": {
+      "actor_update": 1.1579,
+      "rollout": 19.0062,
+      "teacher_scoring": 0.2864
+    },
+    "steady_state_median_throughput": {
+      "rollout_tokens_per_second": 13.4693,
+      "teacher_scored_positions_per_second": 893.86,
+      "update_positions_per_second": 221.0899
+    },
+    "time_to_first_rollout_seconds": 28.4604,
+    "time_to_first_teacher_targets_seconds": 28.7706,
+    "time_to_first_update_seconds": 30.0902,
+    "total_run_bytes": 64966843
+  },
+  "miniverl_version": "0.10.0.dev0",
+  "models": {
+    "student": {
+      "adapter": null,
+      "attn_implementation": "sdpa",
+      "dtype": "auto",
+      "gradient_checkpointing": true,
+      "lora": {
+        "alpha": 32,
+        "bias": "none",
+        "dropout": 0.0,
+        "enabled": true,
+        "r": 16,
+        "target_modules": [
+          "q_proj",
+          "v_proj"
+        ]
+      },
+      "model_id": "Qwen/Qwen3-0.6B",
+      "prepare_kbit_training": true,
+      "quantization": "nf4",
+      "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
+      "tokenizer_id": null,
+      "tokenizer_revision": null,
+      "toy": {
+        "hidden_size": 64,
+        "intermediate_size": 128,
+        "max_position_embeddings": 1024,
+        "num_heads": 4,
+        "num_layers": 2
+      },
+      "trust_remote_code": false
+    },
+    "teacher": {
+      "adapter": null,
+      "attn_implementation": "sdpa",
+      "dtype": "bfloat16",
+      "mode": "standard",
+      "model_id": "Qwen/Qwen3-1.7B",
+      "quantization": "nf4",
+      "revision": "70d244cc86ccca08cf5af4e1e306ecf908b1ad5e",
+      "tokenizer_id": null,
+      "tokenizer_revision": null,
+      "toy": {
+        "hidden_size": 64,
+        "intermediate_size": 128,
+        "max_position_embeddings": 1024,
+        "num_heads": 4,
+        "num_layers": 2
+      },
+      "toy_pretrain_lr": 0.003,
+      "toy_pretrain_steps": 0,
+      "toy_teacher_seed": 99,
+      "trust_remote_code": false
+    }
+  },
+  "profile": "verl-opd-v0.8-single-gpu-pg-k1-v1",
+  "profile_identity": {
+    "digest": "8bbff2027f44d9601d3ed3143621a60280a1bacafd1b08d2ad95bf4d9caac3ca",
+    "export_version": "verl-opd-pg-k1-export-v1",
+    "field_rule_digest": "d53a91b0094a12428006d973683ac85e91fc6b55f47dc33636ce1d2e661b7e57",
+    "loss_conformance_version": "pg-k1-verl-v0.8-v1",
+    "native_compiler_version": "pg-k1-native-v1",
+    "profile_name": "verl-opd-v0.8-single-gpu-pg-k1-v1",
+    "profile_schema_version": 1,
+    "upstream_commit": "7aed6b230776f963fa09509c10d9c3a767d1102c",
+    "upstream_repository": "https://github.com/verl-project/verl",
+    "upstream_tag": "v0.8.0"
+  },
+  "recipe": {
+    "compiled_plan_sha256": "47f7a2218c823abf8e021d6569805db7c4d2805f33493a920d627322533cea5e",
+    "estimator": "k1",
+    "input_parquet_sha256": "baa99ba8c59d4c30b2e3081346860e7e8b2665dd70fdbcd7952935545a0cd770",
+    "interrupt_after_updates": 4,
+    "logical_batch": 4,
+    "optimizer_updates": 8,
+    "policy_loss": "vanilla",
+    "prompt_limit": 128,
+    "prompts_available": 64,
+    "prompts_consumed": 32,
+    "response_limit": 64,
+    "teacher_target": "sampled_token_log_probability"
+  },
+  "resource_contract": {
+    "gpu_hour_limit": 3.0,
+    "peak_reserved_limit_gib": 14.5,
+    "peak_reserved_within_limit": true,
+    "total_gpu_workload_seconds": 359.7
+  },
+  "resume": {
+    "adapter_and_optimizer_byte_identical": true,
+    "excluded_run_identity_field": "resolved_config_digest",
+    "global_optimizer_steps": 8,
+    "interrupt_checkpoint_bytes": 27600769,
+    "resume_load_seconds": 5.5309,
+    "status": "exact_match",
+    "task_cursor": 32,
+    "training_state_fields_identical": true,
+    "trajectories_byte_identical": true
+  },
+  "schema_version": 1,
+  "scientific_scope": {
+    "algorithm_comparison": false,
+    "runtime_correctness_only": true,
+    "task_quality_evaluated": false
+  },
+  "source_commit": "5c2482c34244508c4581056e09f46f6eab0948fb",
+  "status": "measured",
+  "verl": {
+    "commit": "7aed6b230776f963fa09509c10d9c3a767d1102c",
+    "distributed_execution_tested": false,
+    "repository": "https://github.com/verl-project/verl",
+    "tag": "v0.8.0"
+  },
+  "workload_driver_sha256": "5b712a40ef2932800ce43072fa88f175a2454febe7a60b0642356bc5be4dfdd3"
+}
diff --git a/docs/for-verl-users.md b/docs/for-verl-users.md
@@ -94,10 +94,13 @@ bytes and schema are copied into export provenance.
 
 ## Actor, teacher and reference roles
 
-The actor is the trainable PEFT policy. The teacher produces top-k token IDs
-and log-probabilities on actor-generated tokens; top-k mass and overlap are
-diagnostics, not an explicit tail bucket. Pure GKD in this profile has no reference
-policy, critic, task reward or policy-gradient term. Local runtime strategies
+The actor is the trainable PEFT policy. In the direct-GKD profile the teacher
+produces top-k token IDs and log-probabilities on actor-generated tokens;
+top-k mass and overlap are diagnostics, not an explicit tail bucket. In the PG
+k1 profile the target is only the sampled token's teacher log-probability,
+bound to the rollout actor log-probability and policy version; current actor
+log-probability is recomputed at update time. Neither profile has a reference
+policy, critic or task reward. Local runtime strategies
 may keep quantized roles resident, swap movable unquantized roles, or share a
 compatible backbone, but role identities never collapse in provenance.
 
@@ -121,8 +124,9 @@ tested.
 - **Unknown field:** capture a resolved config and remove fields outside the
   documented profile; inspect-only compilation can still explain known
   unsupported values.
-- **Algorithm field rejected:** PG OPD, rewards, KL penalties, `n>1`, multiple
-  teachers and distributed counts are intentionally unsupported.
+- **Algorithm field rejected:** select the explicit PG-k1 profile for its
+  narrow sampled policy-loss path. Rewards, critics, external advantages, KL
+  penalties, `n>1`, multiple teachers and distributed counts remain unsupported.
 - **Interpolation rejected:** resolve Hydra/OmegaConf in your trusted verl
   environment first. miniVERL will not execute `${...}`.
 - **High-risk reinterpretation not accepted:** inspect `miniverl plan`, then

diff --git a/docs/profiles/index.md b/docs/profiles/index.md
@@ -39,6 +39,17 @@ scale-out reports carry the complete profile identity. Changing any version
 axis produces a new digest. Existing profile names cannot silently acquire new
 semantics; a later algorithm path is registered under a different name.
 
-Currently measured: `verl-opd-v0.8-single-gpu-v1`, the reward-free direct-GKD
-`forward_kl_topk` path. See [For verl users](../for-verl-users.md) for the
-algorithm and execution boundary.
+## Which profile should I use?
+
+| Profile | Objective | Teacher target | Trade-off | Status |
+| --- | --- | --- | --- | --- |
+| `verl-opd-v0.8-single-gpu-v1` | direct GKD `forward_kl_topk` | top-k token IDs and log-probabilities | fuller distributional signal; larger target artifact | measured |
+| `verl-opd-v0.8-single-gpu-pg-k1-v1` | sampled `k1` + vanilla policy loss | sampled-token teacher log-probability | sampled-token signal; smaller target artifact | measured |
+
+Both are reward-free, strict current-policy, one-actor/one-teacher profiles on
+one CUDA GPU. The PG profile is not PPO: it uses a policy-loss form with a
+detached distillation-derived advantage and has no task reward, critic,
+reference KL or replay. Neither profile is claimed to be more accurate or
+better aligned. See [For verl users](../for-verl-users.md), the
+[PG-k1 ADR](../adr/0010-verl-v0.8-pg-k1-contract.md), and the runtime evidence
+for the measured boundary.
diff --git a/docs/verl-opd-runtime.md b/docs/verl-opd-runtime.md
@@ -1,6 +1,6 @@
 # Current verl-style OPD runtime
 
-This is miniVERL's current executable path: one documented, resolved subset of
+This is miniVERL's current executable path: two documented, resolved subsets of
 official verl `v0.8.0`, pinned at `7aed6b23`, compiled into local phases on one
 NVIDIA CUDA GPU. It is not a distributed verl runtime and does not accept
 arbitrary Hydra YAML.
@@ -38,6 +38,12 @@ reinterpretations; the built-in profile has a value-bound approval manifest.
 reports top-k mass/overlap diagnostics. It does not create the explicit K+1
 tail bucket used by miniVERL's separate native `bucketed_topk_tail` objective.
 
+The second profile, `verl-opd-v0.8-single-gpu-pg-k1-v1`, records the sampled
+token, rollout-time actor log-probability and teacher log-probability, then
+recomputes the current actor log-probability for pinned `k1` + vanilla
+policy-loss semantics. It exports no top-k requirement. See [Which profile
+should I use?](profiles/index.md) for the neutral choice boundary.
+
 ## Placement is fail-closed
 
 - NF4/int8 roles use resident local phases; bitsandbytes parameters cannot swap.