Code for ACL2023 paper "Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation". [paper]

• python version == 3.7.0
• fairseq version == 0.12.2
• pytorch version == 1.13.1
• sacrebleu version == 1.5.1
• admin-torch version == 0.1.0

bash train_ende_big_teacher.sh

bash train_ende_vanilla_word_kd.sh

bash train_ende_word_kd_wo_corr.sh

bash train_ende_word_kd_wo_top1.sh

bash train_ende_student_baseline.sh

bash train_ende_word_kd_topk_info.sh

bash train_ende_tie_kd.sh

We use model averaging trick for the evaluation of all the student models (average last 5 checkpoints):

bash average_ckpts.sh

Then we use fairseq-interactive to generate translations with the averaged model:

bash interactive_ende.sh

We use multi-bleu.perl in mosesdecoder to calculate the tokenized BLEU scores of the translations. Besides, we also use COMET (Unbabel/wmt20-comet-da) for a more convincing evaluation.

Please cite this paper if you use this repo.

@article{zhang2023towards,
  title={Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation},
  author={Zhang, Songming and Liang, Yunlong and Wang, Shuaibo and Han, Wenjuan and Liu, Jian and Xu, Jinan and Chen, Yufeng},
  journal={arXiv preprint arXiv:2305.08096},
  year={2023}
}