Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox

Jiacheng Pang^*, Ashutosh Chaubey^*, Mohammad Soleymani
University of Southern California
_{^* Equal contribution}

Official v1.0.0 code release for our ICML 2026 paper. We introduce VoxParadox, an adversarial benchmark that isolates the paralinguistic understanding of Audio LLMs under controlled linguistic–acoustic contradiction, plus two methods for closing the gap:

• PCLM (Prompt-Conditioned Layer Mixer) — exposes intermediate audio-encoder layers to the LLM via a prompt-conditioned mixture.
• DPO (Direct Preference Optimization) — biases the LLM toward acoustically grounded answers over transcript-implied alternatives.

.
├── af3/audio-flamingo/   # AF3 with PCLM — inference, eval, probing
├── qwen2audio/           # Qwen2-Audio with PCLM — inference, eval, probing
├── clap/                 # CLAP encoder probing
├── hubert/               # HuBERT encoder probing
├── assets/               # logo, figures
└── LICENSE               # USC Research License

Each model has its own environment — the AF3 and Qwen2-Audio toolchains pull incompatible versions of transformers and deepspeed, so don't mix them.

We recommend starting with Qwen2-Audio.

# Qwen2-Audio (recommended starting point)
conda create -n qwen2audio python=3.10 -y && conda activate qwen2audio
pip install torch torchaudio transformers accelerate librosa soundfile

# Audio Flamingo 3
cd af3/audio-flamingo
bash environment_setup.sh af3

Match the torch/CUDA build to your hardware. See each model's README for version pins and probing-specific extras.

from datasets import load_dataset
ds = load_dataset("IHP-Lab/VoxParadox", split="test")

Or download the directory (voxparadox.json + audio/) directly from the dataset page.

Both models ship an entry point that writes a single predictions.jsonl ({"id": ..., "response": ...} per line). Start with Qwen2-Audio — it's the fastest path to a working round-trip.

# Qwen2-Audio
python -m qwen2audio.eval.run_eval \
    --model_path <ckpt> \
    --data_path <data_path> \
    --audio_base <audio_base> \
    --output_dir runs/eval/<run-name>
# multi-GPU: `torchrun --nproc-per-node=N -m qwen2audio.eval.run_eval ...`

# AF3
bash af3/audio-flamingo/scripts/eval_voxparadox.sh <ckpt> [data_path] [audio_base] [output_dir]

• <ckpt> can be the base AF3 / Qwen2-Audio model (e.g. nvidia/audio-flamingo-3, Qwen/Qwen2-Audio-7B-Instruct) or any PCLM / PCLM+DPO checkpoint. PCLM activation is read from the checkpoint's config.json.
• <data_path> can be the VoxParadox JSON (default for the AF3 wrapper) or any MCQ JSON in the same schema — fields id, audio_path, question, choice_a..d. So the scripts can eval MMSU, custom benchmarks, etc., without modification.

The dataset repo ships eval.py, which computes both GT Accuracy and Adversarial-Label Agreement (ALA):

python eval.py --predictions <output_dir>/predictions.jsonl

The layer-wise probing pipeline used in the paper is under */probing/:

• af3/audio-flamingo/probing/ — full 3-stage pipeline (forward pass → cache → MLP probe). Run with bash run_probing.sh.
• qwen2audio/probing/ — forward-pass + tensor dump; reuses the AF3 cache/train scripts.
• clap/probing/, hubert/probing/ — encoder-only probing.

@inproceedings{pang2026voxparadox,
  title     = {Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox},
  author    = {Pang, Jiacheng and Chaubey, Ashutosh and Soleymani, Mohammad},
  booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},
  year      = {2026}
}

Released under the USC Research License (research / non-profit use; commercial use requires a separate license). See LICENSE.

Open a GitHub issue or email pangj@usc.edu or achaubey@usc.edu.