Around the World in Eighty Ratings?

Arkadiy Saakyan (Columbia University) · Charvi Rastogi (Google DeepMind) · Lora Aroyo (Google DeepMind)

Proceedings of the 43rd International Conference on Machine Learning (ICML 2026)

Code to reproduce the paper. We study whether geo-cultural values (operationalized via the Inglehart–Welzel Cultural Map) predict AI safety ratings beyond standard demographic factors (age, gender, ethnicity), and show that roughly 10% of items in current safety datasets are culturally sensitive: likely to be mislabeled as safe if raters from a particular cultural quadrant are absent.

The repo is organized into four main components, each with clear entry points:

culture-safety/
├── preprocessing/          # Per-dataset preprocessing notebooks
├── datasets/               # download_links.txt for raw data files
├── linear-models/          # Mixed-effects models (R)
├── cultural-sensitivity/   # Cultural Sensitivity Index (CSI) computation (Python)
├── classifier/             # LLM classifiers: quadrant prediction + sensitive-item triage
└── figure/                 # Figure generation

The paper analyzes 8 safety datasets with geo-cultural rater information:

Raw data download links are in datasets/download_links.txt. CulturalFrames was obtained from the authors. WVS data was obtained from the WVS and from the authors.

The World Values Survey (WVS) data used to derive Inglehart–Welzel scores is at datasets/wvs_trad_surv_latest.csv.

conda create -n cult python=3.11
conda activate cult
pip install pandas numpy scipy matplotlib seaborn transformers torch accelerate

For the linear models, install R dependencies:

install.packages(c("dplyr", "car", "lme4", "lmerTest", "broom",
                   "tidyverse", "broom.mixed", "ggplot2",
                   "emmeans", "MuMIn", "texreg", "ordinal"))

preprocessing/ contains one Jupyter notebook per dataset that loads the raw files, joins rater geo-cultural attributes, maps countries to Inglehart–Welzel zones and quadrants, and writes a CSV to datasets/.

preprocessing/
├── WVS_preprocessing.ipynb          # Derives wvs_trad_surv_latest.csv
├── DIVE Preprocessing.ipynb
├── CulturalFrames preprocessing.ipynb
├── PRISM preprocessing.ipynb
├── DICES990 preprocessing.ipynb
├── NLPositionality preprocessing.ipynb
├── D3 preprocessing.ipynb
├── CreHate preprocessing.ipynb
├── Severity preprocessing.ipynb
└── overall_stats.ipynb              # Dataset-level descriptive statistics

Run notebooks in order: WVS_preprocessing.ipynb first (produces the shared WVS reference file), then any dataset notebook.

linear-models/ contains R scripts that fit the full hierarchy of mixed-effects models reported in Tables 2–3, plus likelihood-ratio tests and AIC comparisons.

linear-models/
├── 00_setup.R            # Shared library loading
├── 01_dive.R … 08_severity.R   # One script per dataset
└── run_all.R             # Runs all eight scripts sequentially

cd linear-models
Rscript run_all.R

Outputs: LaTeX tables in tex/ (cleaned versions in processed_tex/), comparison-metric JSON logs in logs/. See linear-models/README.md for the full model inventory and how to regenerate tables.

We also include some experimentation with ordinal regression models (clmm) in linear-models/ordinal_models/, but it is not necessary for reproducing the main results.

cultural-sensitivity/ computes the Culturally Sensitive Item (CSI) rate: the fraction of items that exactly one cultural quadrant is likely to rate as unsafe (flagging a potential false-negative if that quadrant is absent).

cultural-sensitivity/
├── simulation.py                  # Core: Beta-posterior S_iq scores
├── run_<dataset>.py               # Per-dataset entry points
├── run_all.py                     # Run all datasets + print LaTeX overview table
├── threshold_sensitivity.py       # Sweep vote_unsafe_thresh × S_iq threshold grid
└── visualize_threshold_sensitivity.py   # Heatmaps of the sweep

cd cultural-sensitivity

# Run all datasets and print the overview table (Table 5)
python run_all.py

# Threshold sensitivity sweep (Appendix)
python threshold_sensitivity.py
python visualize_threshold_sensitivity.py

Results are written to datasets/simulation-results/. See cultural-sensitivity/README.md for parameter details.

Two classification tasks, each in its own subdirectory under classifier/:

Can LLMs predict the safety rating of each cultural quadrant for a given item?

classifier/quadrant-level-prediction/
├── fine-tuned/
│   ├── deberta_cultsafe_classifier_train.py   # DeBERTa-v3-large training
│   ├── gemma_cultsafe_classifier_train.py     # Gemma-3-4B training
│   ├── cultsafe_train_lib.py                  # Shared training logic
│   ├── cultsafe_metrics.py                    # Custom metric computation (quadrant-level F1)
│   └── cultsafe_trainer.py                    # Masked multi-label BCE loss
└── api-based/
    ├── generate.py       # Prompt Gemini-3 Flash / GPT-5 Nano (0-shot)
    └── prompt.txt        # Inglehart–Welzel quadrant definitions + JSON format

# Fine-tune DeBERTa (10 seeds × 5 epochs)
python classifier/quadrant-level-prediction/fine-tuned/deberta_cultsafe_classifier_train.py \
    --epochs 5 --seeds 10

# Fine-tune Gemma-3-4B
python classifier/quadrant-level-prediction/fine-tuned/gemma_cultsafe_classifier_train.py \
    --epochs 5 --seeds 10 --cuda-visible-devices 0,1,2,3

Can LLMs identify which items need culturally diverse annotation?

classifier/sensitive-item-prediction/
├── train.py              # Unified entry point (DeBERTa or Gemma, two tasks)
├── cultsafe_train_lib.py
└── cultsafe_train_runtime.py

# Safe-vs-sensitive classification with DeBERTa
python classifier/sensitive-item-prediction/train.py \
    --model deberta --dataset all --task safe-vs-sensitive --epochs 5

# Safe-vs-unsafe baseline
python classifier/sensitive-item-prediction/train.py \
    --model deberta --dataset all --task safe-vs-unsafe --epochs 5

Bootstrap confidence intervals for both tasks are computed via classifier/bootstrap.py.

@misc{saakyan2026quantifyingsaliencegeoculturalvalues,
      title={Quantifying the Salience of Geo-Cultural Values for Pluralistic Safety Alignment}, 
      author={Arkadiy Saakyan and Charvi Rastogi and Lora Aroyo},
      year={2026},
      eprint={2606.00369},
      archivePrefix={arXiv},
      primaryClass={cs.CY},
      url={https://arxiv.org/abs/2606.00369}, 
}

Contact me at a.saakyan@cs.columbia.edu if there are any questions!

I have used Claude Code to reformat some of the original code and generate README files. It is possible some hallucinations might have slipped through, though I verified the outputs.