D2Controller

Code and data for "Dynamic Demonstrations Controller for In-Context Learning".

• Increasing the number of demonstrations does not necessarily improve the model performance
• The optimal k-shot setting differs depending on specific datasets and models

Based on the above analyses, we propose a method named D2Controller:

• D2Controller not only boosts ICL performance but also saves time and space during inference of the LLMs
• We apply D2Controller to eight different sizes of LLMs and realize an average of 5.4% relative improvement across ten datasets. We also extend D2Controller to previous ICL models and yield competitive results

Create a new virtual environment with Python==3.9.16

conda create --name d2controller python=3.9.16

Install requirement.txt

pip install -r requirement.txt

Please download LLMs (except GPT-3) from the HuggingFace. The LLMs we use in our paper include gpt2-medium, gpt2-large, gpt2-xl, Cerebras-GPT-2.7B, Cerebras-GPT-6.7B, opt-13b and opt-30b. Put the model files under llm/ directory. For example

llm/
  |--gpt2-medium/
      |--config.json
      |--merges.txt
      |--pytorch_model.bin
      |--tokenizer.json
      |--vocab.json
  |--gpt2-large/
      |--...
      ...
  ...

run the bash script do_preprocess.sh to transform original dataset files

bash scripts/do_preprocess.sh

To obtain selected $k$-shot settings, run the script run_selectk.sh

bash scripts/run_selectk.sh

For OPT-30B model, we use the script run_selectk_multi.sh to obtain results

bash scripts/run_selectk_multi.sh

Notice that you should allocate names and directories for datasets and models in the script.

To obtain ICL results based on the select $k$-shot setting, run the script run_icl.sh

bash scripts/run_icl.sh

To obtain Oracle $k$-shot setting on validation set, run the script run_oracle_k.sh

bash scripts/run_oracle_k.sh

We will release the code and scripts for GPT-3 later.

Main Results

D2Controller is beneficial to other ICL models

D2Controller can achieve competitive results on a small number of in-context example groups

Details

IICScore is effective in selecting evaluation examples

Dynamically selecting k performs better than using the maximum k

If you find this helpful, please cite our paper.

@misc{zhao2023dynamic,
      title={Dynamic Demonstrations Controller for In-Context Learning}, 
      author={Fei Zhao and Taotian Pang and Zhen Wu and Zheng Ma and Shujian Huang and Xinyu Dai},
      year={2023},
      eprint={2310.00385},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}