AI-TRAITS

Repository for the AI-TRAITS dataset, including data generation, pre-processing, feature extraction, post-processing, and data analysis. This repository supports our research paper "A Multilingual, Large-Scale Study of the Interplay between LLM Safeguards, Personalisation, and Disinformation".

This repository contains the complete pipeline for generating and analysing the AI-TRAITS dataset, which studies how Large Language Models (LLMs) personalise content based on target personas. The pipeline includes:

• Data Generation: Creating target-agnostic and persona-targeted documents using multiple LLMs
• Pre-processing: Cleaning and preparing generated texts
• Feature Extraction: Computing metrics, detecting entities, and analyzing model behavior
• Post-processing: Classifying persuasion techniques and personalisation levels
• Dataset Construction: Assembling the final dataset for analysis

• Elasticsearch database (required for storing generated documents)
• Python 3.11
• CUDA-capable GPU (for local model inference)
• API keys for cloud-based models (OpenAI, Anthropic, X.AI)

conda create -n ai_traits python=3.11
conda activate ai_traits
conda install pip
pip install --upgrade pip

The util folder contains a library of Python classes to help construct documents in the correct JSON format and send them to Elasticsearch for indexing.

cd util
pip install --editable .

The utility package has optional dependencies for different LLMs and features. Install the appropriate option(s) for the models and features you want to use:

# For cloud-based models (OpenAI, Anthropic, X.AI)
pip install --editable .[openai,anthropic,xai]

# For local GPU models (using vLLM)
pip install --editable .[vllm]

# For automated metrics (readability, BERT score, ROUGE, etc.)
pip install --editable .[automated_metrics]

# For entity recognition (spaCy with GPU support)
# Choose based on your CUDA version:
pip install --editable .[ner-cuda11x]  # For CUDA 11.x
# OR
pip install --editable .[ner-cuda12x]  # For CUDA 12.x

# Install all dependencies
pip install --editable .[openai,xai,anthropic,vllm,automated_metrics,ner-cuda11x]

Note: For entity recognition, you must install the appropriate CUDA version. If you're unsure, check your CUDA version with nvidia-smi or nvcc --version.

After installing the entity recognition dependencies, download the required spaCy models:

python -m spacy download en_core_web_trf
python -m spacy download pt_core_news_lg
python -m spacy download ru_core_news_lg

Create a .env file in the root directory with the following variables:

ELASTIC_API_KEY="your-elasticsearch-api-key"
OPENAI_API_KEY="your-openai-api-key"          # For ChatGPT models
XAI_API_KEY="your-xai-api-key"                # For Grok models
ANTHROPIC_API_KEY="your-anthropic-api-key"    # For Claude models
RESEARCHER_INITIALS="your-initials"          # Used for document IDs
RESEARCHER_NAME="Your Name"                   # Used for document metadata

Edit the makefile and set your Elasticsearch index name:

INDEX = your-index-name

You can also customise:

• CUDA_DEVICE: GPU device number (default: 0)
• MODEL: Model to process or all for all models (default: all)

Create a CSV file named prompts.csv in the root directory with your base prompts. The file must contain a column named prompt:

prompt,setting
"Your first prompt text here",setting1
"Your second prompt text here",setting2

The makefile provides shortcuts to run the different stages of the pipeline. The entire pipeline should be executed in the following order:

1. target_agnostic - Generate target-agnostic documents (no persona targeting)
2. targeted - Generate persona-targeted documents
3. clean - Clean indexed documents (remove disclaimers, translations, etc.)
4. metrics - Compute automated metrics (readability, BERT score, ROUGE, etc.)
5. persuasion - Classify persuasion techniques and news frames
6. behaviour - Analyze model behavior (jailbreak, disclaimer, refusal, etc.)
7. personalisation - Detect personalisation levels for targeted documents
8. entity_recognition - Extract named entities from documents
9. build_dataset - Export final dataset to parquet files

# Generate target-agnostic documents for all models
make target_agnostic

# Generate targeted documents for all models
make targeted

# Run all other pipeline stages
make clean
make metrics
make persuasion
make behaviour
make personalisation
make entity_recognition
make build_dataset

You can process a single model by setting the MODEL variable:

# Process only Llama model
make target_agnostic MODEL=llama
make targeted MODEL=llama
make metrics MODEL=llama

# Process only ChatGPT
make target_agnostic MODEL=chatgpt
make metrics MODEL=chatgpt

Available model names:

• llama - Meta-Llama-3-8B-Instruct
• gemma - google/gemma-2-9b-it
• vicuna - lmsys/vicuna-7b-v1.5
• qwen - Qwen/Qwen2.5-7B-Instruct
• mistral - mistralai/Mistral-Nemo-Instruct-2407
• grok - grok-2
• chatgpt - openai/gpt-4o
• claude - claude-3-5-sonnet-20241022
• all - Process all models (default)

# Target-agnostic generation (no persona targeting)
make target_agnostic

# Targeted generation (with persona instructions)
make targeted

# Clean documents (remove disclaimers, translations)
make clean

# Compute automated metrics
make metrics

# Classify persuasion techniques and news frames
make persuasion

# Analyze model behavior
make behaviour

# Detect personalisation levels
make personalisation

# Extract named entities
make entity_recognition

# Export dataset for a specific model
make build_dataset MODEL=llama

# Export dataset for all models
make build_dataset MODEL=all

The dataset will be saved to data/final/dataset.parquet and organised by language and model in data/final/<language>/<model>.parquet.

The final dataset contains the following columns:

Analysis experiments are executed as individual Jupyter notebooks located in the analysis/ directory:

• distribution_graphics.ipynb: Analysis of overall model behaviour and personalisation levels present in documents generated by the LLMs.
• analysis/base_vs_persona.ipynb: Analysis of the differences between texts generated with and without a target persona.
• analysis/liwc.ipynb: Analysis of LIWC (Linguistic Inquiry and Word Count) features present in the generated texts.
• analysis/co_occurances_analysis.ipynb: Analysis of co-occurrences of personalisation levels.

AI_TRAITS/
├── analysis/              # Analysis notebooks
├── batch_apis/            # Batch API processing scripts
├── clean/                 # Document cleaning scripts
├── doc/                   # Elasticsearch templates and documentation
├── entity_recognition/    # Entity recognition scripts
├── llm_processing/        # LLM-based processing pipelines
├── model_behaviour/       # Model behavior detection
├── personalisation/       # Personalisation level detection
├── pull_from_elastic/    # Dataset export scripts
├── util/                  # Utility package (ai_traits_util)
├── generate_llm_outputs.py  # Main document generation script
├── automated_metrics.py   # Automated metrics computation
├── run_elg_classifiers.py # Persuasion and topic classification
├── prompts.csv           # Base prompts file (user-provided)
└── makefile              # Pipeline automation

• Ensure your CUDA version matches the installed CuPy/spaCy CUDA version
• Check GPU availability with nvidia-smi
• Verify CUDA_DEVICE in the makefile matches your available GPU

• API Models (ChatGPT, Claude, Grok): Ensure API keys are set in .env
• Local Models: Ensure sufficient GPU memory and correct CUDA device is specified
• Batch Processing: API models require additional batch processing step (handled automatically)

See LICENSE file for details.

For now, please cite the pre-print available on ArXiv:

@article{leite2025multilingual,
  title={A Multilingual, Large-Scale Study of the Interplay between LLM Safeguards, Personalisation, and Disinformation},
  author={Leite, Jo{\~a}o A and Arora, Arnav and Gargova, Silvia and Luz, Jo{\~a}o and Sampaio, Gustavo and Roberts, Ian and Scarton, Carolina and Bontcheva, Kalina},
  journal={arXiv preprint arXiv:2510.12993},
  year={2025}
}