Target Speaker Extraction (TSE) models have demonstrated impressive performance on synthetic datasets such as LibriMix and WSJMix. However, these benchmarks lack the acoustic realism and conversational dynamics of actual human interactions — such as spontaneous speech, overlapping turns, and environmental noise — limiting their relevance to real-world scenarios.

Efforts like REAL-M and LibriCSS have attempted to bridge this gap. REAL-M collects simultaneous speech in shared environments through multi-speaker read-aloud sessions, while LibriCSS re-records isolated utterances via synchronized playback. Though valuable, these datasets still fall short of capturing the nuances of real conversations — including irregular speaker turns, sporadic utterances, and authentic background conditions.

To address these limitations, we introduce REAL-T, the first conversation-centric dataset specifically designed for TSE under real-world conditions. Built from speaker diarization corpora, REAL-T naturally includes overlapping speech, enrollment-ready segments, and complex conversational behaviors.

Key features of REAL-T include:

• Multi-lingual: English and Mandarin recordings
• Multi-genre: Covering diverse conversational scenarios
• Multi-enrollment: Multiple enrollment utterance from different parts of the conversation

To support controlled evaluation, we define two test sets:

• BASE: A filtered, balanced subset for initial testing
• PRIMARY: A more realistic and challenging benchmark

Evaluations reveal that existing TSE models suffer significant performance degradation on REAL-T, highlighting the need for more robust approaches tailored to real conversational speech.

For more details, refer to our paper: REAL-T Paper

Datasets are at huggingface.

git clone https://github.com/REAL-TSE/REAL-T.git
cd REAL-T

# install wesep
git submodule init
git submodule update

conda create -n REAL-T python=3.9
conda activate REAL-T
pip install -r requirements.txt

Please replace $PWD below with the absolute path to this project

$ export PATH=$PWD/FireRedASR/fireredasr/:$PWD/FireRedASR/fireredasr/utils/:$PATH
$ export PYTHONPATH=$PWD/FireRedASR/:$PYTHONPATH
$ export PYTHONPATH=$PWD/wesep/:$PYTHONPATH

Evaluation requires the REAL-T dataset and the ASR model checkpoint FireRedASR-AED-L and whisper-large-v2 from Hugging Face. The dataset must be prepared in a specific format before running evaluation. To automatically set up everything, run:

bash -i ./pre.sh

The run_tse.sh script below demonstrates how to perform TSE inference with the Wesep toolkit using a BSRNN model trained on VoxCeleb1. You can adapt its input/output structure to suit your own TSE model.

➡️ See Bash Script Parameters: run_tse.sh for detailed setup.

cd REAL-T
bash -i run_tse.sh

The transcribe_and_evaluation.sh script automates the evaluation of ASR outputs using Token Error Rate (TER) metrics. You can run transcription, evaluation, or both, depending on your needs. Before execution, make sure to review the script’s parameters—such as whether to include the Fisher corpus, the evaluation subset, input/output directories, and other configurable options. ➡️ See Bash Script Parameters: transcribe_and_evaluation.sh for detailed setup.

cd REAL-T

# Only ASR
bash -i ./transcribe_and_evaluation.sh 1

# Only Evaluation
bash -i ./transcribe_and_evaluation.sh 2

# Both ASR and Evaluation
bash -i ./transcribe_and_evaluation.sh 1 2

This script runs Target Speaker Extraction (TSE) inference for multiple datasets using a specified model. Each dataset will be processed individually, generating separated target speaker audio files.

This script automates two main stages for the REAL-T datasets:

• ASR transcription using different models depending on dataset language.
• ASR evaluation using TER (Token Error Rate) metrics.

The operation mode is controlled by passing one or two arguments:

• 1 → only run ASR transcription
• 2 → only run evaluation
• 1 2 → run both transcription and evaluation sequentially

Typically, run 1 2 for a full process.

BASE_DIRS=(
    "YourPath/output/PRIMARY/bsrnn_vox1"
    "YourPath/output/PRIMARY/bsrnn_hr_vox1"
)

The table below compares the performance of several recently proposed TSE models on the simulated Libri2Mix and PRIMARY test sets.

@inproceedings{li25da_interspeech,
  title     = {{REAL-T: Real Conversational Mixtures for Target Speaker Extraction}},
  author    = {{Shaole Li and Shuai Wang and Jiangyu Han and Ke Zhang and Wupeng Wang and Haizhou Li}},
  year      = {{2025}},
  booktitle = {{Interspeech 2025}},
  pages     = {{1923--1927}},
  doi       = {{10.21437/Interspeech.2025-2662}},
  issn      = {{2958-1796}},
}

For any questions, please contact: shuaiwang@nju.edu.cn