🎙️ KALL-E: Autoregressive Speech Synthesis with Next-Distribution Prediction

• [2025.08.05] 🔥 🔥 🔥 We release the inference code of KALL-E!
• [2025.09.17] 🎉 🎉 🎉 KALL-E's paper is updated on arxiv, read it now!
• [2025.09.24] 🔥 🔥 🔥 We release the KALL-E weights!

This repository contains the inference utilities for KALL-E, a text-to-speech system that predicts continuous speech representations using a single autoregressive language model.

• Autoregressive Language Modeling: Utilizes an autoregressive approach for next-distribution prediction in text-to-speech synthesis.
• Continuous Speech Distribution: Directly models and predicts continuous speech distributions conditioned on text, avoiding reliance on diffusion-based components.
• FlowVAE: Employs FlowVAE to extract continuous speech distributions from waveforms, rather than using discrete speech tokens.
• Single AR Language Model: Uses a single autoregressive language model to predict continuous speech distributions from text, constrained by Kullback-Leibler divergence loss.
• Simplified Paradigm: Offers a more straightforward and effective approach for using continuous speech representations in TTS.

• Random Speaker Voices - When no speaker prompt is provided, the model is able to generate random voices, either female or male.

• ⚡ Blazing-fast Synthesis Generate up to 5 seconds of audio with a single click in the web UI.

• Context-aware Synthesis KALL-E excels in generating expressive, context-aware speech, showcasing its ability to handle complex linguistic and emotional features with ease.

• Python>=3.9 or higher
• PyTorch with CUDA support
• Transformers==4.49.0
• NumPy
• SciPy
• alias-free-torch

You need download the model in advance and place them like this:

KALL-E
|    ckpt
|    | - flowvae.pt
|    | - model.pt
|    ......
|    model.py
|    infer.py

python infer.py --target_text "<ka li E> is a text-to-speech system that predicts continuous speech representations using a single autoregressive language model."

python infer.py \
--target_text "<ka li E> is a text-to-speech system that predicts continuous speech representations using a single autoregressive language model." \
--prompt_text "oh that's crazy!" \
--prompt_wav_path ./test.wav 

python web.py

• ⚡ Blazing-fast Synthesis

• Context-aware Synthesis

If our work and codebase is useful for you, please cite as:

@misc{kalle,
      title={KALL-E:Autoregressive Speech Synthesis with Next-Distribution Prediction}, 
      author={Kangxiang Xia and Xinfa Zhu and Jixun Yao and Wenjie Tian and Wenhao Li and Lei Xie},
      year={2025},
      eprint={2412.16846},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2412.16846}, 
}