- [2025-06] PowerOPD paper released on arXiv: arXiv:2606.17199.
On-policy distillation (OPD) trains a student model to imitate a teacher on the student's own generated trajectories. While widely adopted for its efficiency, the standard log-ratio reward log p_T(o_t) − log p_S(o_t) is unbounded by construction. We identify three resulting training pathologies:
- (a) High variance / heavy negative tail — reward values plummet to nearly −50, allowing single rare tokens to dominate gradient updates.
- (b) Early-position extremes — massive reward magnitudes concentrate at early rollout positions, destabilizing the prefix distribution and causing cascading errors.
- (c) Persistent extremes throughout training — these extreme values never decay, continuously injecting instability across the entire optimization process.
Standard post-hoc fixes (clipping, tanh compression, z-score normalization) operate only after the distortion occurs and do not resolve the root cause. PowerOPD addresses this at the reward level by replacing the log-ratio with a natively bounded power-transformed reward:
The power transformation has three key properties:
- Bounded: rewards lie in
[−1, 1]by construction, eliminating instability from extreme log-ratios. - Sign-consistent: the reward is positive when the teacher assigns higher probability than the student, and negative otherwise — same direction as log-ratio OPD.
- Recovers log-ratio in the limit: as α → 0,
r_t^(α)converges (after rescaling) to the standard log-ratio reward.
Compared to vanilla OPD (log-ratio reward), PowerOPD achieves:
| Metric | Improvement |
|---|---|
| Pass@8 (6 math benchmarks) | +5.71 |
| Avg@8 (6 math benchmarks) | +6.37 |
| Wall-clock training time | −59.2% |
| Peak GPU memory | −23.1% |
git clone https://github.com/EIT-NLP/PowerOPD
cd PowerOPD
pip install -r requirements.txtThe training code uses a local TRL checkout included in trl/. Set up PYTHONPATH automatically via the launch scripts.
Download your student and teacher models and place them (or symlink them) under models/:
models/
Qwen3-1.7B-Base/
Qwen3-4B/
The scripts default to ${PROJECT_ROOT}/models/Qwen3-1.7B-Base and ${PROJECT_ROOT}/models/Qwen3-4B. Override with environment variables:
export STUDENT_MODEL=/path/to/student
export TEACHER_MODEL=/path/to/teacherPowerOPD (recommended):
bash scripts/run_poweropd.shKey hyperparameters (all overridable via environment variables):
| Variable | Default | Description |
|---|---|---|
POWER_REWARD_ALPHA |
5 |
Power exponent α (larger = stronger compression) |
TOKEN_LOSS_NORMALIZATION |
window_token_mean |
Loss normalization across tokens |
TERMINAL_STOP_TARGET |
mask |
How to handle the final stop token |
MAX_STEPS |
1500 |
Total training steps |
LEARNING_RATE |
5e-7 |
Learning rate |
GRADIENT_ACCUMULATION_STEPS |
32 |
Gradient accumulation steps |
EVAL_STEPS |
30 |
Evaluation frequency |
Example with custom settings:
POWER_REWARD_ALPHA=1.0 \
STUDENT_MODEL=/path/to/student \
TEACHER_MODEL=/path/to/teacher \
bash scripts/run_poweropd.shTeacher quantization (reduces GPU memory for large teachers):
TEACHER_QUANT_MODE=4bit bash scripts/run_poweropd_teacher_quant.shStandard log-ratio OPD and other reward variants are provided for comparison:
bash scripts/run_log_ratio.sh # vanilla OPD (log-ratio reward)
bash scripts/run_log_clip.sh # log-ratio with clipping
bash scripts/run_log_tanh.sh # log-ratio with tanh normalizationThe scripts report to SwanLab by default. Set your API key:
export SWANLAB_API_KEY=your_api_keyTo disable tracking:
REPORT_TO=none bash scripts/run_poweropd.shTraining and evaluation data are included under data/:
data/train/deepscaler_conversation.jsonl— DeepScaler math reasoning training setdata/eval/math_test.jsonl— MATH-500 evaluation set
For questions about the paper or codebase, open an issue or reach the authors directly:
- Anhao Zhao: anhao.zhao@connect.polyu.hk
- Xiaoyu Shen: xyshen@eitech.edu.cn
@article{zhao2025poweropd,
title={PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation},
author={Zhao, Anhao and Tong, Junlong and Fan, Yingqi and Nie, Ping and Li, Wenjie and Shen, Xiaoyu},
journal={arXiv preprint arXiv:2606.17199},
year={2025}
}