Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han
NAVER AI Lab

On-policy distillation is an alternative post-training method that provides token-level supervision from a teacher model. In this work, we introduce the delta signal, defined as the difference between a reasoning-tuned teacher and its base model before reasoning tuning. This signal captures the changes induced by reasoning tuning and provides a more direct objective for transferring reasoning capabilities. Based on this idea, we propose On-Policy Delta Distillation (OPD²). Across mathematics, science, and code-reasoning benchmarks, OPD² consistently outperforms conventional on-policy distillation methods on multiple Qwen3 model sizes, in both thinking and non-thinking settings, as well as on Gemma-4-E4B.

• arXiv: https://arxiv.org/abs/2607.15161

The code and training recipes will be released soon.

@article{heo2026opd2,
  title   = {On-Policy Delta Distillation},
  author  = {Heo, Byeongho and Hwang, Jaehui and Yun, Sangdoo and Han, Dongyoon},
  journal = {arXiv preprint arXiv:2607.15161},
  year    = {2026}
}