This is the official repository for "TimeAudio: Bridging Temporal Gaps in Large Audio-Language Models".

Recent Large Audio-Language Models (LALMs) exhibit impressive capabilities in understanding audio content for conversational QA tasks. However, these models struggle to accurately understand timestamps for temporal localization (e.g., Temporal Audio Grounding) and are restricted to short audio perception, leading to constrained capabilities on fine-grained tasks. We identify three key aspects that limit their temporal localization and long audio understanding: (i) timestamp representation, (ii) architecture, and (iii) data. To address this, we introduce TimeAudio, a novel method that empowers LALMs to connect their understanding of audio content with precise temporal perception. Specifically, we incorporate unique temporal markers to improve time-sensitive reasoning and apply an absolute time-aware encoding that explicitly grounds the acoustic features with absolute time information. Moreover, to achieve end-to-end long audio understanding, we introduce a segment-level token merging module to substantially reduce audio token redundancy and enhance the efficiency of information extraction. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing audio datasets into a new dataset focused on temporal tasks and establish a series of metrics to evaluate the fine-grained performance. Evaluations show strong performance across a variety of fine-grained tasks, such as dense captioning, temporal grounding, and timeline speech summarization, demonstrating TimeAudio's robust temporal localization and reasoning capabilities.

Overview of the TimeAudio Framework.

git clone https://github.com/lysanderism/TimeAudio
cd TimeAudio
conda create -n TimeAudio python=3.10.16
conda activate TimeAudio
pip install -r requirements.txt
conda install -c conda-forge ffmpeg libsndfile

Download the pretrained model from 🤗 Hugging Face:

1. Download whisper large v2 to whisper_path.
2. Download Fine-tuned BEATs_iter3+ (AS2M) (cpt2) to beats_path.
3. Download vicuna 7B v1.5 to vicuna_path.
4. Download TimeAudio on Hugging Face to ckpt_path
5. Put these checkpoints to the path ./pretrained_model/

To inference the TimeAudio, run:

ppython3 cli_inference.py --cfg-path configs/infer_config.yaml

## stage 1
torchrun --nproc_per_node=8 train.py --cfg-path "configs/config_stage1.yaml"
## stage 2
torchrun --nproc_per_node=8 train.py --cfg-path "configs/config_stage2.yaml"

torchrun --nproc_per_node=8 inference.py --cfg-path "configs/infer_config.yaml"

If you find TimeAudio useful, please cite our paper:

@article{,
  title={TimeAudio: Bridging Temporal Gaps in Large Audio-Language Models},
  author={Hualei Wang, Yiming Li, Shuo Ma, Hong Liu, Xiangdong Wang},
  journal={arXiv preprint arXiv:2511.11039},
  year={2025},
  url={https://arxiv.org/abs/2511.11039}
}

We gratefully acknowledge the creators of:

• SALMONN(Tang et al.)
• Qwen2-Audio(Chu et al.)

Please follow CC-BY-NC.