HuggingFace 🤗 | ModelScope 🔮 | Paper 📖 | Demo 🎥

Shanghai Jiao Tong University | Ant Group

VocalNet is a series of high performance, low-latency speech LLMs enabled by a scalable and model-agnostic training framework for real-time voice interaction. VocalNet introduces multi-token prediction (MTP), a novel approach optimized for speech LLMs that simultaneously improves generation speed and quality. VocalNet outperforms many mainstream Omni LLMs and existing open-source speech LLMs despite using significantly less training data.

• VocalNet-1B: Built upon LLaMA-3.2-1B-Instruct. With much fewer model parameters, VocalNet-1B achieves performance comparable to mainstream speech LLMs, including LLaMA-Omni, Freeze-Omni, and GLM-4-Voice.

• VocalNet-8B: Built upon LLaMA-3.1-8B-Instruct. In the evaluation of speech interaction models, VocalNet-8B significantly outperforms most mainstream speech LLMs and Omni LLMs.

Model Architecture & Training Strategy

• VocalNet consists of a speech encoder to convert waves into speech representations, a pre-trained LLM backbone and a speech decoder for speech token generation. A downsample adaptor is added after the speech encoder to achieve a lower frame rate, and a speech projector to bridge the dimension gap between the LLM hidden state and decoder inputs. The generated speech token is sent to the speech vocoder, in which the corresponding speech response is constructed.

• VocalNet adopts a dual-stage training strategy: Multi-Modal Alignment and Generative Supervised Fine-Tuning. In the first stage, VocalNet is trained using speech queries and text responses. The LLM backbone is trained using LoRA along with the downsample adaptor. In the second stage, VocalNet is trained using speech queries and speech responses. The major components are frozen, while the speech projector and speech decoder are trained.

Streaming Decoding

VocalNet employs two attention mask mechanisms tailored for complete sequence processing and real-time speech generation respectively, inspired by MiniCPM-o. For non-streaming mode, the text position attends to whole text positions, and the speech position attends to the whole text positions, itself and its previous speech positions. For streaming mode, the text position attends to itself and its previous text positions, and the speech position attends to chunk-limited text positions, itself and its previous speech positions.

MTP Implementation

VocalNet utilizes N-1 sequential Transformer layers as MTP modules, enabling the prediction of N speech tokens in a single inference step while preserving the temporal relationships. In addition, a layer-wise decaying cross-entropy loss is introduced as the MTP loss.

git clone https://github.com/YHWmz/VocalNet.git
cd VocalNet

conda create -n vocalnet python==3.10
conda activate vocalnet
pip install --upgrade pip
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -e .

# Optional: Install additional packages for training
pip install -e ".[train]"
pip install flash-attn --no-build-isolation

1. We provide two re-synthesized datasets for VocalNet training:

• VoicaAssistant-430K from Mini-Omni:   ModelScope     HuggingFace
• UltraChat from SLAM-Omni:   ModelScope     HuggingFace

Here's an example of a training data instance in JSON format:

[
  {
        "id": "19090",
        "speech": "VoiceAssistant-430K-vocalnet/audios/19090.wav",
        "conversations": [
            {
                "from": "human",
                "value": "<speech>"
            },
            {
                "from": "gpt",
                "value": "The legality of shooting someone who breaks into your house depends on the laws of your specific location. In some places, \"stand your ground\" laws or \"castle doctrine\" laws allow homeowners to use lethal force if they feel threatened. However, other areas may require you to retreat if possible before using force. It's important to familiarize yourself with the laws in your region and consider non-lethal methods of protection as well. Consulting with a legal expert can provide you with guidance tailored to your situation."
            }
        ],
        "units": "VoiceAssistant-430K-vocalnet/cosyvoice2_tokens/19090.npy"
    }
]

After downloading them, organize the data as follows in ./playground/,

├── VoiceAssistant-430K-vocalnet
│   ├── audios
│   └── cosyvoice2_tokens
└── UltraChat-vocalnet
    ├── audios
    └── cosyvoice2_tokens

The speech wave is synthesized with CosyVoice2, and we use CosyVoice 2’s speech tokenizer to extract the speech tokens.

1. Download Base LLM and speech encoder

• Base LLM: LLaMA-3.2-1B-Instruct     LLaMA-3.1-8B-Instruct
• Speech encoder: Whisper-large-v3

2. Path Modification in scripts/mtp/llama_stage1_s2t.sh

# Model checkpoint configuration
CHECKPOINT_NAME="llama32-1B-instruct-s2t"  # Checkpoint name (will be used as the directory name)
CHECKPOINT_DIR="./checkpoints/${CHECKPOINT_NAME}"  # Path to save checkpoints

# Base model configuration
BASE_MODEL="./hf_hub/Llama-3.2-1B-Instruct"  # Path to the pretrained base LLM

# Data path configuration
DATA_PATH="./playground/VoiceAssistant-430K-VocalNet/VoiceAssistant-430K.json"  # Training data index
SPEECH_FOLDER="./playground/"  # Root directory for speech files (contains speech queries and preprocessed speech tokens)

# Speech encoder configuration
SPEECH_ENCODER="./models/speech_encoder/whisper-large-v3"  # Path to the Whisper speech encoder model

3. Start Training

bash scripts/mtp/llama_stage1_s2t.sh

1. Path Modification

# Model Configuration
CHECKPOINT_NAME="llama32-1B-instruct-s2s-mtp5"  # Checkpoint name
CHECKPOINT_DIR="./checkpoints/${CHECKPOINT_NAME}"  # Directory to store training checkpoints

# Model Parameters
BASE_MODEL="./checkpoints/llama32-1B-instruct-s2t"  # Stage I S2T model (used as initialization)

# Dataset Paths
DATA_PATH="./playground/VoiceAssistant-430K-VocalNet/VoiceAssistant-430K.json"  
SPEECH_FOLDER="./playground/"  

# Speech encoder configuration
SPEECH_ENCODER="./models/speech_encoder/whisper-large-v3"  # Whisper-Large-V3 encoder weights

2. Start Training

bash scripts/mtp/llama_stage2_s2s.sh

1. Model Preparartion:

• Download Our Open-source Models: VocalNet-1B from HuggingFace or ModelScope, and VocalNet-8B from HuggingFace or ModelScope.
• Download the Whisper model from HuggingFace and place it in the ./models/speech_encoder/ directory.

2. CosyVoice Preparartion: We utilize CosyVoice2's flow-matching model to convert VocalNet-generated speech tokens into the final audio waveform.

• To infer our VocalNet-1B and VocalNet-8B, you need to download the CosyVoice2-0.5B from HuggingFace.

3. Path Modification

• Modify the path in omni_speech/infer/vocalnet.py.

COSYVOICE_MODEL=""     ## CosyVoice2-0.5B       i.e. /workspace/CosyVoice/pretrained_models/CosyVoice2-0.5B-VocalNet
VOCALNET_MODEL = ""    ## VocalNet speech LLM   i.e. ./checkpoints/VocalNet-1B

4. Local Infer

## stage 1 infer (s2t)
python3 omni_speech/infer/vocalnet.py --query_audio ./omni_speech/infer/llama_questions_42.wav 
## stage 2 infer (s2s)
python3 omni_speech/infer/vocalnet.py --query_audio ./omni_speech/infer/llama_questions_42.wav --s2s --save_dir ./

VocalNet is evaluated on OpenAudioBench, consisting AlpacaEval, LLaMA Questions, TriviaQA, and Web Questions.

Bold indicates the optimal result in each subgroup and underline indicates the suboptimal result in base size models.

• LLaMA-Omni: VocalNet is developed on the codebase of LLaMA-Omni.
• LLaVA: We borrowed a lot of code from LLaVA for model training.
• CosyVoice: VocalNet borrows code about speech generation from CosyVoice.
• Freeze-Omni: We borrow some code about speech decoder.

This repository is released under the Apache-2.0 license as found in the LICENSE file.

If you find our data/model/code/paper helpful, please consider citing our papers 📝 and staring us ⭐️！

@article{wang2025vocalnet,
  title={VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation},
  author={Wang, Yuhao and Liu, Heyang and Cheng, Ziyang and Wu, Ronghua and Gu, Qunshan and Wang, Yanfeng and Wang, Yu},
  journal={arXiv preprint arXiv:2504.04060},
  year={2025}
}