XARES-LLM trains a typical LALM using the audio encoder provided by the user. The system automatically downloads training data, trains the LALM then tests various downstream tasks, providing scores for each.

uv venv 
uv pip install $THIS_REPO

You can either pass a .py file with your encoder defined inside, requiring the ending "Encoder":

python3 -m xares_llm.run example/dummy/dummyencoder.py 

Or pass the module directly:

python3 -m xares_llm.run example.dummy.dummyencoder.DummyEncoder

python3 -m xares_llm.run example/dummy/dummyencoder.py task1 task1

python3 -m xares_llm.run example/dummy/dummyencoder.py task2 task2

For reprodution purposes Please run the script on one GPU!.

Any Encoder needs to be derived from torch.nn.Module, accepting (audio, audio_attention_mask) as input and outputs (features, feature_attention_mask) and a set variable output_dim that represents the model's embedding size. Masks use the huggingface format, 1 represents keep, 0 represents mask and feature_attention_mask.size(-1) == features.size(1)

class DummyEncoder(torch.nn.Module):
    def __init__(self, **kwargs) -> None:
        super().__init__()
        self.output_dim = 256

    def forward(self, audio, audio_attention_mask=None) -> tuple[torch.Tensor, torch.Tensor | None]:
        output = torch.randn(len(audio), 10, self.output_dim, device=audio.device)
        # Do something with attention mask or just return None, both are alright
        return output, audio_attention_mask

Here we train on clotho and test on clotho.

python3 -m xares_llm.run example/dummy/dummyencoder.py clotho clotho
# Or for the Module interface
# python3 -m xares_llm.run example.dummy.dummyencoder.DummyEncoder clotho clotho

# Using Multiple GPU's with Accelerate:
# accelerate launch -m xares_llm.run example/dummy/dummyencoder.py clotho clotho

All available current datasets can be seen by running python3 -m xares_llm.run -h.

Datasets can also be passed with a custom .yaml:

For training, the format is:

train_data:
  CustomDataName:
    prompt: My prompt
    data:
    - PATH_TO_MY_TARS{00..10}.tar
    key: DATAKEY
num_training_workers: 4

For evaluation:

eval_custom:
  data:
    data:
    - PATH_TO_MY_TARS{00..10}.tar
    key: DATAKEY # Inside the json
    prompt: My prompt
  batch_size: 4
  num_workers: 0
  metric: Accuracy

By default all data is downloaded and stored in ./xares_data from the current directory. During training the data is directly fetched and cached in this directory. One can modify the data path with the environment variable XARES_DATA_HOME.

The data is stored in $XARES_DATA_HOME, which defaults to $PWD/xares_data.

hf download mispeech/xares_llm_data --local-dir xares_data --repo-type dataset
hf download mispeech/MECAT-Caption --local-dir xares_data --repo-type dataset

python3 -m xares_llm.download_data

The download location can also be changed:

# XARES_DATA_HOME="NEW_LOCATION" python3 -m xares_llm.download_data

By default the code uses fp32 precision, which is slow but can be reproduced. Note that results might differ depending on your Graphics card. However, results on a single machine should be completely reproducible.

If one wants to speed up training (and no reproduction) use:

accelerate launch --mixed-precision='bf16' -m xares_llm.run task1