X-ARES (pron /eks ˈeəz/) is an audio encoder benchmark that ​automatically trains task-specific output layers​ (including MLPs and k-NN) ​on user provided encoders. It is heavily inspired by the HEAR benchmark.

Users provide a single pretrained audioencoder, which outputs frame-level embeddings. Embeddings are evaluated using a fine-tuned MLP layer for clip- and frame-level tasks. Further a non-parameterized kNN algorithm is used to evaluate the quality of embeddings. For specialized tasks, pre-trained decoders are incorporated as task-specific components.

• ASV2015
• CREMA-D
• Fluent Speech Commands
• LibriCount
• LibriSpeech-ASR
• LibriSpeech-Male-Female
• RAVDESS
• Speech Commands V2
• speechocean762
• VocalSound
• VoxCeleb1
• VoxLingua107

• Clotho
• DESED
• ESC-50
• FSD18-Kaggle
• FSD50k
• UrbanSound 8k
• Finger snap sound¹
• Inside/outside car¹
• Key scratching car¹
• LiveEnv sounds¹
• Subway broadcast¹

• FMA
• GTZAN Genre
• MAESTRO
• NSynth

# Recommended: using uv
uv sync --extra examples

# Or via pip
pip install xares[examples]

For development, clone and install in editable mode:

git clone <this-repo>
cd xares
uv sync --extra examples   # or: pip install -e .[examples]

Run the benchmark with the baseline encoder (Dasheng) on all tasks:

uv run xares --max-jobs 8 example/dasheng/dasheng_encoder.py src/tasks/*.py

Or run specific tasks:

uv run xares example/dasheng/dasheng_encoder.py src/tasks/esc50_task.py src/tasks/gtzan_task.py

Datasets are downloaded automatically from Zenodo. If that fails, use tools/download_manually.sh.

python -m xares.run also works as an alternative to uv run xares.

X-ARES provides two evaluation methods to assess the quality of audio representations: MLP (Linear Fine-Tuning) and kNN (Unparameterized Evaluation).

MLP: Linear Fine-Tuning on Task-Specific Data. A linear layer will be trained using the provided user embeddings, optimized with predefined hyperparameters for each task. This approach assesses how effectively the fixed representations can be adapted to specific tasks by training an additional linear layer, using predefined hyperparameters tailored for each task. This method evaluates the adaptability and effectiveness of the pre-trained models when applied to new, task-specific contexts without altering the original model parameters.

kNN: Unparameterized Evaluation. Pre-trained model embeddings will be used directly for K-nearest neighbor (KNN) classification without training. This method aims to evaluate the inherent quality of the audio representations without any fine-tuning. While this approach may not always yield the highest performance in real-world applications, it serves as a rigorous test of the fundamental representational power of the embeddings. By avoiding parameterized layers, this method provides a clear view of how well the model captures essential features of the audio data.

Here are the evaluation results for several baseline models using MLP and kNN methods. The weighted average is calculated using the test set size for each dataset.

Examples of audio encoder wrapper could be found at examples, where the baseline encoders are implemented.

We provide a check function to verify if the encoder is correctly implemented:

>>> from xares.audio_encoder_checker import check_audio_encoder

>>> encoder = YourEncoder()
>>> check_audio_encoder(encoder)
True

Then run the benchmark with your own encoder:

uv run xares --max-jobs 8 your_encoder.py src/tasks/*.py

By sure that your encoder supports variable length inference up to 10 minutes of audio. We recommend to simply chunk the input audio in your encoder to mitigate any out-of-memory issues, like:

class MyCustomEncoder(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.sampling_rate = 16000
        self.output_dim = 512
        self.hop_size_in_ms = 10
        self.model = my_model_implementation()
        # This code is only for cases where the model itself does not implement chunking
        self.custom_max_audio_length = int(self.sampling_rate * 10)

    def forward(self, audio: torch.Tensor):
        if audio.ndim == 1:
            audio = audio.unsqueeze(0)

        self.model.eval()
        with torch.inference_mode():
            if audio.shape[-1] > self.custom_max_audio_length:
                embeds = []
                for chunk in audio.split(self.custom_max_audio_length, dim=-1):
                    if chunk.shape[-1] < self.sampling_rate:
                        chunk = torch.nn.functional.pad(
                            chunk, (0, self.sampling_rate - chunk.shape[-1]))

                    embed = self.model(chunk)
                    embeds.append(embed)
                encoded_audio = torch.cat(embeds, dim=1)
            else:
                encoded_audio = self.model(audio)

        return encoded_audio

Adding a new task is easy. Refer to the existing task implementations for guidance. You need to create a TaskConfig tailored to your chosen dataset.

@inproceedings{zhang2025xares,
  title={X-ARES: A Comprehensive Framework for Assessing Audio Encoder Performance},
  author={Zhang, Junbo and Dinkel, Heinrich and Niu, Yadong and Liu, Chenyu and Cheng, Si and Zhao, Anbei and Luan, Jian},
  booktitle={Interspeech 2025},
  year={2025}
}