Skip to content
EIT-NLPPublic

About

No description, website, or topics provided.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Latest commit

 

History

6 Commits

Folders and files

Repository files navigation

PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation

arXiv GitHub


🎉 News


📖 Overview

On-policy distillation (OPD) trains a student model to imitate a teacher on the student's own generated trajectories. While widely adopted for its efficiency, the standard log-ratio reward log p_T(o_t) − log p_S(o_t) is unbounded by construction. We identify three resulting training pathologies:

  • (a) High variance / heavy negative tail — reward values plummet to nearly −50, allowing single rare tokens to dominate gradient updates.
  • (b) Early-position extremes — massive reward magnitudes concentrate at early rollout positions, destabilizing the prefix distribution and causing cascading errors.
  • (c) Persistent extremes throughout training — these extreme values never decay, continuously injecting instability across the entire optimization process.

Standard post-hoc fixes (clipping, tanh compression, z-score normalization) operate only after the distortion occurs and do not resolve the root cause. PowerOPD addresses this at the reward level by replacing the log-ratio with a natively bounded power-transformed reward:

$$r_t^{(\alpha)} = p_T(o_t)^\alpha - p_S(o_t)^\alpha$$

The power transformation has three key properties:

  • Bounded: rewards lie in [−1, 1] by construction, eliminating instability from extreme log-ratios.
  • Sign-consistent: the reward is positive when the teacher assigns higher probability than the student, and negative otherwise — same direction as log-ratio OPD.
  • Recovers log-ratio in the limit: as α → 0, r_t^(α) converges (after rescaling) to the standard log-ratio reward.

Compared to vanilla OPD (log-ratio reward), PowerOPD achieves:

Metric Improvement
Pass@8 (6 math benchmarks) +5.71
Avg@8 (6 math benchmarks) +6.37
Wall-clock training time −59.2%
Peak GPU memory −23.1%

✨ Getting Started

Environment

git clone https://github.com/EIT-NLP/PowerOPD
cd PowerOPD
pip install -r requirements.txt

The training code uses a local TRL checkout included in trl/. Set up PYTHONPATH automatically via the launch scripts.

Model Setup

Download your student and teacher models and place them (or symlink them) under models/:

models/
  Qwen3-1.7B-Base/
  Qwen3-4B/

The scripts default to ${PROJECT_ROOT}/models/Qwen3-1.7B-Base and ${PROJECT_ROOT}/models/Qwen3-4B. Override with environment variables:

export STUDENT_MODEL=/path/to/student
export TEACHER_MODEL=/path/to/teacher

Training

PowerOPD (recommended):

bash scripts/run_poweropd.sh

Key hyperparameters (all overridable via environment variables):

Variable Default Description
POWER_REWARD_ALPHA 5 Power exponent α (larger = stronger compression)
TOKEN_LOSS_NORMALIZATION window_token_mean Loss normalization across tokens
TERMINAL_STOP_TARGET mask How to handle the final stop token
MAX_STEPS 1500 Total training steps
LEARNING_RATE 5e-7 Learning rate
GRADIENT_ACCUMULATION_STEPS 32 Gradient accumulation steps
EVAL_STEPS 30 Evaluation frequency

Example with custom settings:

POWER_REWARD_ALPHA=1.0 \
STUDENT_MODEL=/path/to/student \
TEACHER_MODEL=/path/to/teacher \
bash scripts/run_poweropd.sh

Teacher quantization (reduces GPU memory for large teachers):

TEACHER_QUANT_MODE=4bit bash scripts/run_poweropd_teacher_quant.sh

Baseline Comparisons

Standard log-ratio OPD and other reward variants are provided for comparison:

bash scripts/run_log_ratio.sh   # vanilla OPD (log-ratio reward)
bash scripts/run_log_clip.sh    # log-ratio with clipping
bash scripts/run_log_tanh.sh    # log-ratio with tanh normalization

Experiment Tracking

The scripts report to SwanLab by default. Set your API key:

export SWANLAB_API_KEY=your_api_key

To disable tracking:

REPORT_TO=none bash scripts/run_poweropd.sh

📄 Data

Training and evaluation data are included under data/:

  • data/train/deepscaler_conversation.jsonl — DeepScaler math reasoning training set
  • data/eval/math_test.jsonl — MATH-500 evaluation set

📨 Contact

For questions about the paper or codebase, open an issue or reach the authors directly:


🎈 Citation

@article{zhao2025poweropd,
  title={PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation},
  author={Zhao, Anhao and Tong, Junlong and Fan, Yingqi and Nie, Ping and Li, Wenjie and Shen, Xiaoyu},
  journal={arXiv preprint arXiv:2606.17199},
  year={2025}
}

⭐ Star History

Star History Chart

About

No description, website, or topics provided.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages