M-IFEval: Multilingual Instruction Following Evaluation

This repository contains source code and data for M-IFEval: Multilingual Instruction-Following Evaluation

Large language models (LLMs) are increasingly used in real-world applications that require understanding and executing user instructions accurately. Evaluating instruction following is crucial to ensure these models perform reliably across different tasks and languages.

Building upon the Instruction Following Evaluation for Large Language Models benchmark, which first underscored the importance of instruction-following evaluation, we introduce M-IFEval, a benchmark designed to extend this evaluation to multiple languages.

M-IFEval currently supports French, Japanese, and Spanish, incorporating both general and language-specific instructions to provide a more comprehensive assessment of multilingual instruction adherence.

• 🏆 Leader board
• ⚙️ How to run
  • 🌐 With Colab
  • 🖥️ Locally
    • Setup Instructions
    • Evaluate YOUR model
• Contributions 🤝
• 📚 References
• 📜 License

The table below presents the average scores across all instructions for each language, sorted in ascending order by the mean scores across the languages supported by the M-IFEval benchmark (i.e., Spanish, French, and Japanese).

(*) The values reported for English (EN) correspond to the evaluation on the original IFEval dataset.

We provide a Jupyter Notebook designed to run seamlessly in Google Colab. This notebook guides you through:

• Generating responses from models supported by Hugging Face (HF), OpenAI, and Anthropic.
• Evaluating a given model based on its generated responses.
• Displaying basic visualizations of evaluation results.

Click the badge to open the notebook in Colab:

If you prefer to run the evaluation directly on your machine, whether for computational reasons or other preferences, follow these steps:

First, clone the repository:

git clone -b main https://github.com/lightblue-tech/M-IFEval.git

Move into the cloned directory:

cd M-IFEval 

Ensure all required Python packages are installed in your working environment:

pip3 install --user -r requirements.txt

Download Spacy models:

python -m spacy download es_core_news_sm --quiet
python -m spacy download xx_sent_ud_sm --quiet

To evaluate your model, you need to create a JSONL file containing the instruction prompts and their corresponding responses.

You can either manually create a file with the following structure:

{"prompt": "Write a 300+ word summary ...", "response": "PUT YOUR MODEL RESPONSE HERE"}
{"prompt": "I am planning a trip to ...", "response": "PUT YOUR MODEL RESPONSE HERE"}

Or, you can use get_responses.py to automatically generate the JSONL file in the required format:

• For Hugging Face, OpenAI, or Anthropic models:

  To add support for a new model, update the SUPPORTED_MODELS dictionary in get_responses.py by specifying the model name and its corresponding inference method:

  SUPPORTED_MODELS = {
      "openai_model_name_or_version": "openai",
      "anthropic_model_name_or_version": "anthropic",
      "hf_model_name_or_path": "vllm"
  }

  Replace "openai_model_name_or_version", "anthropic_model_name_or_version", or "hf_model_name_or_path" with the actual model identifiers.

  Note: The dictionary values ("openai", "anthropic", "vllm") must remain unchanged to ensure the correct inference method is used.

  Once the model is added, generate responses by running:

  python3 -m get_responses.py --model_name {model_name}

  Replace {model_name} with the exact model identifier you added.

export OPENAI_API_KEY="your_openai_api_key_here"
export ANTHROPIC_API_KEY="your_anthropic_api_key_here"

export HUGGINGFACE_TOKEN="your_huggingface_token_here"
huggingface-cli login --token $HUGGINGFACE_TOKEN

• For (yet) unsupported providers requiring API keys:

  Implement a subclass of ResponseGenerator in get_responses.py to handle API calls for your provider.

Once you have the JSONL file ready, run the evaluation script:

python3 -m evaluation_main \
  --input_data=./data/{lang}_input_data.jsonl \
  --input_response_data=./data/input_response_data_model_name.jsonl \
  --output_dir=./evaluation/

• Replace {lang} with the language tag corresponding to the language you wish to evaluate (e.g., en for English, fr for French).
• Update input_response_data with the path to your model's response JSONL file.

This command will generate evaluation results in the specified output directory.

We welcome contributions to:

• Support the evaluation of other models
• Add models to the leaderboard
• Expand the benchmark to additional languages
• Improve evaluation quality (data and instructions) for currently supported languages

If you have any feedback, concerns, or inquiries about the benchmark, feel free to open an issue.

If you use our work, please consider citing our preprint:

@article{Dussolle2025MIFEval,
  title={M-IFEval: Multilingual Instruction-Following Evaluation},
  author={Antoine Dussolle and Andrea Cardena Díaz and Shota Sato and Peter Devine},
  year={2025},
  journal={arXiv preprint},
  volume={arXiv:2502.04688},
  eprint={2502.04688},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2502.04688}
}

As well, as the original IFEval paper's preprint:

@article{zhou2023instruction,
  title={Instruction-Following Evaluation for Large Language Models},
  author={Zhou, Jeffrey and Lu, Tianjian and Mishra, Swaroop and Brahma, Siddhartha and Basu, Sujoy and Luan, Yi and Zhou, Denny and Hou, Le},
  journal={arXiv preprint arXiv:2311.07911},
  year={2023}
}

This project is licensed under the Apache License, Version 2.0. See the LICENSE file for details.

This project includes code from Instruction Following Evaluation for Large Language Models licensed under Apache 2.0.