Human Abstraction and Reasoning Corpus (H-ARC)

This repository contains scripts and additional information about H-ARC to accompany our paper published in the Scientific Data journal.

The data can be downloaded from our OSF repository.

Participant responses, natural language descriptions, errors and state space graphs can all be explored visually on our project webpage.

H-ARC consists of action by action traces of humans solving ARC tasks from the both the training and evaluation sets using an interface and setup similar to François Chollet's initial proposal. The experiment platform can be viewed at this link and the underlying code is available in an accompanying Github repo.

The original ARC dataset can be found here.

@article{legrisComprehensiveBehavioralDataset2025,
  title = {A {{Comprehensive Behavioral Dataset}} for the {{Abstraction}} and {{Reasoning Corpus}}},
  author = {LeGris, Solim and Vong, Wai Keen and Lake, Brenden M. and Gureckis, Todd M.},
  year = {2025},
  month = aug,
  journal = {Scientific Data},
  volume = {12},
  number = {1},
  pages = {1380},
  issn = {2052-4463},
  doi = {10.1038/s41597-025-05687-1},
  abstract = {The Abstraction and Reasoning Corpus (ARC) is a visual program synthesis benchmark designed to test out-of-distribution generalization in machines. Comparing AI algorithms to human performance is essential to measure progress on these problems. In this paper, we present H-ARC (Human-ARC): a novel large-scale dataset containing solution attempts from over 1700 humans on ARC problems. The dataset spans the full set of 400 training and 400 evaluation tasks from the original ARC benchmark, and it is the largest human evaluation to date. By publishing the dataset, we contribute human responses to each problem, step-by-step behavioral action traces from the ARC user-interface, and natural-language solution descriptions of the inferred program/rule. We believe this dataset will be of value to researchers, both in cognitive science and AI, since it offers the potential to facilitate the discovery of underlying mechanisms supporting abstraction and reasoning in people. The insights to be gained from these data not only have value for cognitive science, but could in turn inform the design of more efficient, human-like AI algorithms.}
}

1. Ensure you have Python 3.10 or later installed on your system.

2. Clone this repository to your local machine:

   gh repo clone le-gris/h-arc
   cd h-arc

3. Create a virtual environment:

   python -m venv .venv

4. Activate the virtual environment:

   • On Windows:

     venv\Scripts\activate

   • On macOS and Linux:

     source .venv/bin/activate

5. Install the required packages using pip and the requirements.txt file:

   pip install -r requirements.txt

The H-ARC dataset can be downloaded as a zip archive from our OSF repository. To extract it:

1. Navigate to the project root directory if you're not already there and move the zip archive there. Make sure it is named osfstorage-archive.zip.

2. Use the following command to extract the dataset:

   • On Windows:

     tar -xf data/osfstorage-archive.zip

   • On macOS and Linux:

     unzip data/osfstorage-archive.zip

After extraction, you should see several CSV files in the data and survey folders.

The H-ARC dataset consists of several CSV files containing different aspects of human performance on ARC tasks.

All files are in CSV format. In the data folder, there are the following files:

• data.csv: All collected data from complete / incomplete participant data
• incorrect_submissions.csv: All unique errors on each task and their counts from complete/incomplete participant data
• summary_data.csv: Attempt by attempt summary data for complete/incomplete participant data
• readme.md: data directory description

In the survey folder, there are the following files:

• feedback.csv: Participant feedback
• demographics.csv: Demographic information
• withdraw.csv: Withdrawal information
• readme.md: survey directory description

For more detailed information about the dataset, see each of the readme files.

To analyze performance using a Bayesian Item Response Theory model:

1. Generate the model:

   python src/generate_IRT_model_results.py --n_samples 10000 --n_burn 2000 --seed 4 --impute

   Remove the --impute flag to exclude missing data from the analysis.

2. Generate plots and statistics:

   python src/IRT_model_plots_analysis.py --model_path models/bayes_IRT_model_burn2000_N10000_imputed_4.pkl --verbose

This will create trace plots, parameter visualizations, and detailed statistics in the figures/ and results/ directories.

For more details, see the model readme.md file.

We include in this repository the main Jupyter notebooks used to compute reported results from our paper.

This notebook looks at some aspects of the ARC dataset structure.

This notebook computes basic performance metrics on the H-ARC dataset, including overall solve rates, action counts, and time-related statistics for both training and evaluation tasks.

This notebook looks at some basic demographics data from our pool of participants.

This notebook contains miscellaneous analyses, including participant counts for different experimental conditions and various data processing steps.

This notebook analyzes error patterns in participant responses, including copy errors and other common mistake types across both training and evaluation tasks.

This notebook examines learning effects across tasks using mixed-effects logistic regression models. It analyzes how task success rates change as participants progress through the experiment.

This notebook focuses on analyzing incomplete task attempts, comparing performance metrics between participants who completed all tasks and those who didn't, and examining factors that might contribute to task incompletion.

This notebook compares the performance of human participants with that of algorithmic solutions to evaluation set ARC tasks. It analyzes success rates, error patterns, and solution strategies between humans and AI systems.

This notebook shows how to use the data to extract action traces for further analysis.

Follow these steps to process a Kaggle submission file. This will faciliate downstream human-machine comparisons. Here we use the "Claude-3.5 (Baseline)" approach from the ARC Prize leaderboard as an example.

1. Create the necessary directories:

   mkdir -p data/kaggle_solutions/claude3_5-langchain

2. Visit the following webpage: Claude 3.5 Langchain ARC Submission

3. Download the submission.json file from the webpage into the data/kaggle_solutions/claude3_5-langchain directory.

4. Run the kaggle_submision_to_csv.py script with the appropriate submission ID:

   python src/kaggle_submision_to_csv.py --submission_id claude3_5-langchain

This will process the JSON file and create a CSV file in the same directory with a similar format to our human data.

This dataset is licensed under the CC0 1.0 Universal and can be used for any purposes.