This repository provides an overview of all components used for the creation of BLOOMZ & mT0 and xP3 introduced in the paper Crosslingual Generalization through Multitask Finetuning.

• Data
• Models
• Create xP3
• Train models
  • BLOOMZ
  • mT0
• Evaluate models
  • Rank Evaluation
  • Generation Evaluation
• Plots & Tables
  • Plots
  • Tables
• Citation

We have processed & uploaded xP3. If you want to recreate it, follow these steps:

1. Get promptsource: For xP3mt git clone -b xp3mt https://github.com/Muennighoff/promptsource.git, for xP3 git clone -b tr13 https://github.com/Muennighoff/promptsource.git & install cd promptsource; pip install -e .
2. Get packages pip install -q datasets iso-639
3. Get the creation script & edit it if necessary:

• For xP3mt, set USE_ENGLISH_PROMPTS = False in the beginning
• For xP3, set USE_ENGLISH_PROMPTS = True in the beginning

4. Run the script, such as via python prepare_xp3.py or a SLURM script

For the new extension of xP3, xP3x, the process is largely the same except:

1. Install the xp3x branch instead i.e. pip install git+https://github.com/Muennighoff/promptsource.git@xp3x
2. The creation script is in this repository & named create_xp3x.py.

xP3x is a superset of xP3, so unless you want to reproduce the paper, we recommend always using xP3x (or xP3mt if you want machine-translated prompts).

1. Download the pretrained model checkpoint, which is of shape PP=12, TP=4, DP=4. If you'd like to reshape the model you will also need to download the universal checkpoint. If you want to continue finetuning, you should use our finetuned checkpoint, which is of shape PP=72, TP=1, DP=4.
2. Setup the training code: git clone -b t0loading https://github.com/bigscience-workshop/Megatron-DeepSpeed & follow its setup guide to create an environment with necessary packages.
3. Download the Megatron-DeepSpeed processed xP3megds or repreprocess it for Megatron-DeepSpeed yourself by downloading xP3, removing the merged_{lang}.jsonl files & preprocess it using the script here.
4. Setup & run the training script: We use SLURM scripts available at bigscience-workshop/bigscience/train/tr13-mtf and referred to as xp3capmixnewcodelonglossseq. E.g. this is the script launched to train bloomz. Important parts of the script to modify are:

• #SBATCH variables, such as nodes, gpus, time, etc. - Our SLURM guide is here
• source $six_ALL_CCFRWORK/start-tr13f-6B3-ml-t0 to point to your own conda environment setup via Megatron-DeepSpeed
• PATH environment variables, notably
  • TRAIN_DATA_PATH & VALID_DATA_PATH, which point to files pointing to your processed training and validation data. We provide our files in this repository (xp3capmixnewcodelong_train.txt & xp3capmixnewcodelong_validation.txt), but you will likely want to change the paths inside. The percentages per language are based on how much each language makes up in xP3 with code being slightly upsampled.
• PP_SIZE=72, TP_SIZE=1 & BATCH SIZE & co specifying the layout. This will depend on the hardware available to you. If you change, you may have to reshape the model. For reshaping you need to use the universal checkpoint and use the --universal flag in the script. We recommend saving a new checkpoint right after & then continuing training without --universal, which will be faster.
• If you want to restart from a saved checkpoint (e.g. after training a few steps like above), make sure to remove the --no-load-optim & --reset-progress flags
• After training, you can convert the checkpoint to transformers format using the script here

Helpful resources:

• Blog post
• BLOOM community tab, such as here

Follow the finetuning instructions here making sure to use pretrained mT5 models & the xP3 dataset.

Helpful resources:

• T5X paper

Evaluation results are all available in this repository: https://huggingface.co/datasets/bigscience/evaluation-results under the respective models. Below we explain how to run evaluation.

We evaluate the models on Rank Evaluation on XCOPA, XNLI, XStoryCloze & XWinograd:

1. Get promptsource fork: git clone -b xp3mt https://github.com/Muennighoff/promptsource.git & cd promptsource; pip install -e .
2. Get t-zero fork: git clone -b muennighoff/upgrdps https://github.com/Muennighoff/t-zero.git & cd t-zero; pip install -e .
3. Download model & run evaluation script, for example for bloomz.

We evaluate generation on translation & summarization during training for validation:

1. Get promptsource fork: git clone -b xp3mt https://github.com/Muennighoff/promptsource & cd promptsource; pip install -e .
2. Get bigscience-workshop/lm-evaluation-harness: git clone https://github.com/bigscience-workshop/lm-evaluation-harness. The script for the 7.1B model, for example, is here.

We also evaluate code generation on HumanEval:

1. Get code evaluation code git clone https://github.com/loubnabnl/bloom-code-evaluation & go through its setup.
2. Set prepend_eos to False in code_eval.py at complete_code(model, tokenizer, prompt, num_completions=1, prepend_eos=True, **gen_kwargs) i.e. complete_code(model, tokenizer, prompt, num_completions=1, prepend_eos=False, **gen_kwargs).
3. Download model & run evaluation script swapping out MODEL_CKPT for your path, for example for bloomz use this.

• Figure 1: plotstables/xp3_taxonomy.drawio & plotstables/xp3_taxonomy.pdf
• Figure 2: plotstables/xp3_languages.ipynb & colab
• Figure 3: plotstables/xp3_variants.pdf & drawings
• Figure 4: plotstables/xp3_generalization_bar.pdf & colab
• Figure 5: plotstables/lang_generalization & colab
• Figure 6: plotstables/scale.pdf & colab
• Figure 7: plotstables/validation.pdf & colab
• Figure 8: plotstables/pretraining_sizes.pdf & colab
• Figure 9: plotstables/english_task_generalization.pdf & colab
• Figure 10: plotstables/task_generalization.pdf & colab
• Figure 11: plotstables/roots_xp3_languages.pdf & colab requiring some of the files in plotstables/contamination
• Figure 12: plotstables/examples/bloom_code_example.py & plotstables/examples/bloom_code_light.pdf & plotstables/examples/bloomz_code_light.pdf; The raw code files can be found here & here
• Figure 13 - Figure 16: plotstables/examples/*.pdf & plotstables/examples/generations.drawio

• Table 1: Colab & Colab for complex version
• Table 2: Adapted from the Codex paper
• Table 3: Manual
• Table 4: plotstables/compute_codegen_len.ipynb for generations & plotstables/countcode.py for xP3
• Table 5: Manual
• Table 6: Manual
• Table 7: plotstables/levenshtein.py
• Table 8: Same as Table 1 with languages swapped from L1 to L2
• Table 9: Colab
• Table 10: Colab
• Prompt Appendix: https://github.com/albanie/prompt_formatting_in_latex

@article{muennighoff2022crosslingual,
  title={Crosslingual generalization through multitask finetuning},
  author={Muennighoff, Niklas and Wang, Thomas and Sutawika, Lintang and Roberts, Adam and Biderman, Stella and Scao, Teven Le and Bari, M Saiful and Shen, Sheng and Yong, Zheng-Xin and Schoelkopf, Hailey and others},
  journal={arXiv preprint arXiv:2211.01786},
  year={2022}
}