This repository contains the code for the paper:
Submodular Evaluation Subset Selection in Automatic Prompt Optimization
Jinming Nian, Zhiyuan Peng, Hongwei Shang, Dae Hoon Park, Yi Fang
arXiv:2601.03493
- Python >= 3.10
- CUDA-compatible GPU (for vLLM inference)
# Clone the repository
git clone https://github.com/jmnian/SESS.git
cd SESS
# Install dependencies using uv (recommended)
pip install uv
uv sync
# Or using pip
pip install -e .The datasets are not included in the repository. Use the provided scripts to download them:
# Set your HuggingFace token (required for GPQA - it's a gated dataset)
export HF_TOKEN="your_huggingface_token"
# Download GPQA dataset
python scripts/download_gpqa_dataset.py
# Download MATH dataset
python scripts/download_math_dataset.pyGSM8K is downloaded automatically when first used.
The main entry point is run_opro_parallel.py:
python run_opro_parallel.py \
--dataset="gsm8k" \
--subset_select_method="confidence_weighted_representative" \
--subset_portion=3.5 \
--num_search_steps=100 \
--scorer_model="Qwen/Qwen2.5-7B-Instruct"| Method | --subset_select_method |
Description |
|---|---|---|
| Random | random |
Uniformly random sampling (baseline) |
| Representative | representative |
Facility location for diversity |
| Least Confident | least_confident |
Select samples with lowest model confidence |
| Verbal Least Confident | verbal_least_confident |
Select based on verbalized confidence |
| Confidence-Weighted Representative | confidence_weighted_representative |
Submodular selection weighted by confidence |
# GSM8K with confidence-weighted representative selection (3.5% of training data)
python run_opro_parallel.py \
--dataset="gsm8k" \
--subset_select_method="confidence_weighted_representative" \
--subset_portion=3.5 \
--confidence_weight=0.5 \
--alpha=0.7 \
--num_search_steps=100
# MATH with representative selection
python run_opro_parallel.py \
--dataset="math" \
--subset_select_method="representative" \
--subset_portion=3.5 \
--num_search_steps=100
# GPQA-Diamond with least confident selection
python run_opro_parallel.py \
--dataset="gpqa" \
--task="diamond" \
--subset_select_method="least_confident" \
--subset_portion=10 \
--num_search_steps=100Or use the experiment runner script:
bash run_experiment.bashExperiment results are stored in outputs/optimization-results/. Each experiment directory contains:
configs_dict.json: Experiment configurationtest_evaluation_results.json: Test set evaluation results
If you find this code useful, please cite our paper:
@article{nian2026submodular,
title={Submodular Evaluation Subset Selection in Automatic Prompt Optimization},
author={Nian, Jinming and Peng, Zhiyuan and Shang, Hongwei and Park, Dae Hoon and Fang, Yi},
journal={arXiv preprint arXiv:2601.03493},
year={2026}
}This codebase builds upon OPRO (Large Language Models as Optimizers) by Yang et al. We thank the authors for releasing their code.
This project is licensed under the Apache 2.0 License - see the LICENSE file for details.