CI-REPAIR-BENCH

CI-REPAIR-BENCH is a benchmark for evaluating automated repair of failing CI builds under real GitHub Actions workflows. It collects real CI failure instances, applies candidate patches, and re-runs the original CI pipeline to verify correctness. A repair is successful only if the full CI workflow transitions from failure to pass.

• Python 3.9 or later
• GitHub account
• Hugging Face account
• Access to the benchmark GitHub organization

You must provide both of the following:

1. .env
   Stores secrets (tokens, API keys)

2. config.yaml
   Stores benchmark runtime configuration (paths, usernames, language)

A template config is available at:

config.example.yaml

Create a file named .env in the repository root and add the following values:

# GitHub
GITHUB_USERNAME=your_github_username
GITHUB_TOKEN=your_github_personal_access_token

# Hugging Face
HF_TOKEN=your_huggingface_token

# OpenAI (required for deepeval)
OPENAI_API_KEY=your_openai_api_key

Copy the example configuration file:

cp config.example.yaml config.yaml

• username_gh should match GITHUB_USERNAME
• Environment Setup Use separate virtual environments for:
  1. Benchmark execution
  2. Baseline experiments

To run the benchmark using repositories own GitHub account, fork everything first:

python setup_github/bulk_fork_repositories.py

Commands to set up and run benchmark:

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

To run the benchmark using repositories own GitHub account, fork everything first:

python setup/bulk_fork_repositories.py

• Run the Benchmark

python run_benchmark.py

Results are written to out_folder:

jobs_ids.jsonl
Job identifiers sent to GitHub

jobs_results.jsonl
Results for each job

jobs_awaiting.jsonl
Jobs still running (normally empty)

jobs_invalid.jsonl
Invalid jobs (normally empty)

Sometimes GitHub Actions runs slowly, and not all jobs finish in the initial time window. For this reason, the outcome can be rechecked for the pushed commits and update results without pushing again.

recheck_waiting_jobs.py

• To run the baseline project:

cd baselines
python -m venv venv
source venv/bin/activate
pip install -r requirements-dev.txt
python main.py