Exploring how benign finetuning affects safety refusal and compliance rates in language models

This repository contains the complete code, data, and results for the paper "The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning".

We investigate how benign multilingual supervised fine-tuning (MLSFT) of language models affects their compliance and refusal rates, both in English and the target languages. We perform finetuning using a 1000 datapoint multilingual (8 languages) synthetic version of the Dolly dataset, with manual quality verification for each target language. We then finetune various language models and evaluate changes in safety compliance using Sorry-Bench, both in the English and (via NLLB translation) in the target translated language. Our findings reveal that finetuning has heterogeneous impacts on compliance rates, with effects varying significantly across different models and languages.

For installation, setup, orchestration, and reproduction details, see docs/README.md.

• Benign finetuning produces heterogeneous impacts on safety complaince rates across models and languages.
• Effects on safety are measured using both English and translated evaluations.
• A comprehensive, reproducible multlilingual benchmark with both finetuning data and an evaluation pipeline, verfied manually across 8 non-english languages.

All datasets, fine-tuned models, and evaluation outputs are available on Hugging Face:

• MLSFT Datasets — Dataset collection containing SynthDolly multilingual finetuning data and multilingual Sorry-Bench evaluation datasets for 8 non-english languages: Chinese, Danish, Greek, Hindi, Irish, Portuguese, Spanish, and Tagalog.
• MLSFT Large Models (Epoch 1) — SynthDolly-finetuned larger LLM checkpoints after 1 epoch.
• MLSFT Small Models (Epoch 1) — SynthDolly-finetuned smaller LLM checkpoints after 1 epoch.
• MLSFT Small Models (Epoch 3) — SynthDolly-finetuned smaller LLM checkpoints after 3 epochs.
• MLSFT Small Models (Epoch 5) — SynthDolly-finetuned smaller LLM checkpoints after 5 epochs.
• MLSFT Small Models (Epoch 8) — SynthDolly-finetuned smaller LLM checkpoints after 8 epochs.

Paper results are available in data/results/:

• compliance_rate_stats.csv — Compliance rate statistics across models and languages
• judgement_stats_summary.csv — Detailed safety judgment summaries

These are generated from individual eval runs in data/outputs/ and are automatically refreshed at the end of each orchestration run. To manually refresh the results:

python src/summarize_judgement_stats.py

If you use this code or datasets in your research, please cite:

@InProceedings{hawkins2026mlsft,
  title={Heterogeneous Impacts of Multilingual Safety Finetuning},
  author={Will Hawkins and Kai Rawal and Jonathan Rystrom and Stratis Tsirtsis and Zihao Fu and Greta Warren and Eoin D. Delaney and Ryan Brown and Sandra Wachter and Brent Mittelstadt and Chris Russell},
  booktitle={Proceedings of the 43rd International Conference on Machine Learning},
  year={2026},
  series={Proceedings of Machine Learning Research},
  publisher={PMLR}
}

For questions, feedback, or collaboration inquiries, the best way to reach the authors is by opening a GitHub issue in this repository.

This project is licensed under the MIT License — see LICENSE for details.