Official repository for the paper "Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognition", in which we propose Speech Concept Bottleneck Models (SCBMs), a novel approach for automated hate and counter speech recognition.

✨ SCBM brings interpretability to hate and counter speech recognition by routing decisions through human-readable adjective concepts.

🔗 Paper: https://www.sciencedirect.com/science/article/pii/S030645732500250X

• Adjective tools
  • ✨ Adjective generation: AdjectiveSetGeneration
  • 📖 Adjective definitions: AdjectiveDefinition
• SCBM representations and models
  • 🦙 LLaMA-based feature extraction: Llama
  • 🎯 SCBM and SCBM-T training: SCBM(T)
  • 🧪 Prompt/persona sensitivity: prompt-sensitibity
• Baselines and zero-shot
  • ⚡ Transformer baselines: Transformers_baseline
  • 🔎 Zero-shot (OpenAI/LLaMA): zero-shot-evaluation
• 🧩 ICL & CoT experiments: ICL & CoT experiments
• 🗂️ Datasets overview: Tasks

1. Quickstart
2. Model Architecture
3. Results
4. Training & Evaluation
5. SCBM Representation Computation
6. Training and Evaluation of SCBM and SCBM-T
7. Zero-shot Evaluation on GPT Family
8. Citation

Follow these steps to reproduce the main pipeline end-to-end.

1. Install dependencies:

pip install -r requirements.txt

2. Compute SCBM features with LLaMA (writes .pickle next to CSV):

set -x HF_USER your-username
set -x HF_TOKEN your-token
python Llama/main.py \
  --input_files ./Tasks/germeval/train.csv \
  --use_context false \
  --adjectives_file ./AdjectiveSetGeneration/adjectives.csv

3. Train SCBM variants:

# SCBM (HS_CS)
python "SCBM(T)/SCBM.py" \
  --train_file_name ./Tasks/hs_cs/train.csv \
  --test_file_name ./Tasks/hs_cs/test.csv \
  --use_regularization false \
  --output_dir ./SCBM(T)

SCBM is designed for hate and counter speech recognition by integrating human-interpretable adjective-based concepts as a bottleneck layer between input text and classification.

SCBM leverages adjectives-based representation as semantically meaningful bottleneck concepts derived probabilistically from LLMs, then classifies texts via a transparent, lightweight classifier that learns to prioritize key adjectives. This results in competitive hate and counter speech recognition performance with strong interpretability compared to black-box transformer models.

We summarize quantitative performance across datasets and show qualitative explanation examples.

Performance of all explored approaches in our paper across all employed datasets in terms of macro-$F_1$ score. The best-performing approach in each category is highlighted in italics, and the best-performing approach per dataset is highlighted in bold.

Top-10 most relevant adjectives for individual input samples from each class of the HS-CS dataset provided by SCBM. For comparison, we provide LIME explanations for the same samples generated from the fine-tuned XLM-RoBERTa model.

Transformer baselines are provided in Transformers_baseline/ and operate over the CSVs under Tasks/.

• Baselines (train/dev split): run_transformers.py
• Baselines (5-fold CV, e.g., TSNH): run_transformers-crossval.py

Examples:

# ELF22 split
python Transformers_baseline/run_transformers.py \
  --train_file ./Tasks/elf22/train.csv \
  --dev_file ./Tasks/elf22/test.csv \
  --output_file ./Transformers_baseline/elf22_baselines.pickle

# TSNH cross-validation
python Transformers_baseline/run_transformers-crossval.py \
  --train_file ./Tasks/tsnh/TSNH_uniform.csv \
  --output_file ./Transformers_baseline/tsnh_cv_baselines.pickle

Use Llama/main.py to compute SCBM adjective-probability representations with Llama-3.1-8B-Instruct. The script reads one or more CSVs from Tasks/ and writes a sibling .pickle with id, values (probability vectors), and text (for no-context runs).

Environment (first run clones the model):

set -x HF_USER your-username
set -x HF_TOKEN your-token

Examples:

# No-context (e.g., GermEval)
python Llama/main.py \
  --input_files ./Tasks/germeval/test.csv \
  --use_context false \
  --adjectives_file ./AdjectiveSetGeneration/adjectives.csv \
  --repository meta-llama/Llama-3.1-8B-Instruct \
  --batch_size 244

# Context (e.g., HS_CS)
python Llama/main.py \
  --input_files "[\"./Tasks/hs_cs/train.csv\", \"./Tasks/hs_cs/test.csv\"]" \
  --use_context true \
  --adjectives_file ./AdjectiveSetGeneration/adjectives.csv

SCBM variants live in SCBM(T)/:

• SCBM.py: classifier over adjective-probability features
• SCBMT.py: text + features fusion variant

These scripts expect the .pickle feature files created by the Llama step (same basename as the CSV, e.g., train.csv.pickle).

Examples:

# SCBM (features only)
python "SCBM(T)/SCBM.py" \
  --train_file_name ./Tasks/hs_cs/train.csv \
  --test_file_name ./Tasks/hs_cs/test.csv \
  --use_regularization false \
  --output_dir ./SCBM(T)

# SCBM-T (text + features)
python "SCBM(T)/SCBMT.py" \
  --train_file_name ./Tasks/hs_cs/train.csv \
  --test_file_name ./Tasks/hs_cs/test.csv \
  --use_regularization false \
  --output_dir ./SCBM(T)

Zero-shot baselines live in zero-shot-evaluation/ and support both OpenAI Chat Completions and local LLaMA-3.1.

Scripts

• openai-zero-shot.py: Uses OpenAI Chat Completions. Model is configurable (e.g., gpt-3.5-turbo, chatgpt-4o-latest).
• llama-zero-shot.py: Uses a local pipeline for meta-llama/Llama-3.1-8B-Instruct.

If you use this repository in your research, please cite:

@article{distilling-scbm,
title = {Distilling knowledge from large language models: A concept bottleneck model for hate and counter speech recognition},
journal = {Information Processing & Management},
volume = {63},
number = {2, Part A},
pages = {104309},
year = {2026},
issn = {0306-4573},
doi = {https://doi.org/10.1016/j.ipm.2025.104309},
url = {https://www.sciencedirect.com/science/article/pii/S030645732500250X},
author = {Roberto Labadie-Tamayo and Djordje Slijepčević and Xihui Chen and Adrian Jaques Böck and Andreas Babic and Liz Freimann and Christiane Atzmüller and Matthias Zeppelzauer},
}