LeapTalk: Breaking the Latency–Quality Trade-off in Talking Head Generation 🎥✨

• Real-time streaming: Generate open-ended talking-head videos from a reference image and speech audio in a chunk-by-chunk streaming pipeline.
• One-step inference: Synthesize each video chunk with only 1 NFE, reaching up to 200 FPS in the Lite setting.
• High-fidelity lip synchronization: Audio-driven classifier-free guidance strengthens mouth motion and speech alignment under extreme step reduction.
• Stable long-video identity: Preserve facial structure, appearance, and visual style over long rollouts by reformulating talking-head generation as a data-to-data transport process via a Brownian bridge model, implemented through a novel Bridge Forcing paradigm.

conda create -n leaptalk python=3.12
conda activate leaptalk

pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1

pip install -r requirements.txt

pip install "huggingface_hub[cli]"

huggingface-cli download Soul-AILab/SoulX-FlashHead-1_3B \
  --local-dir ./models/SoulX-FlashHead-1_3B

huggingface-cli download facebook/wav2vec2-base-960h \
  --local-dir ./models/wav2vec2-base-960h

huggingface-cli download z-rx/leaptalk \
  --local-dir ./models/leaptalk

SoulX-FlashHead-1_3B is used as the base model. The LeapTalk checkpoint directory contains the LoRA weights, audio_proj_step_*.pt, and the Lite TAE checkpoint; the TAE path is resolved automatically when LITE=1.

Edit inf.sh with your local paths:

CKPT_DIR="./models/SoulX-FlashHead-1_3B"
WAV2VEC_DIR="./models/wav2vec2-base-960h"
LORA_DIR="./models/leaptalk"
AUDIO_PROJ="./models/leaptalk/audio_proj_step_10400.pt"
COMPILE="off"
NUM_INFERENCE_STEPS="1"
LITE="1"
COND_IMAGE="YOUR_REFERENCE_IMAGE_PATH"
AUDIO_PATH="YOUR_AUDIO_PATH"

Then run:

bash inf.sh

The web demo provides a real-time digital human conversation experience. After loading a portrait image, users can interact with the digital human through either text messages or microphone speech. The left side shows the streaming speaking video and runtime metrics, while the right side keeps the user input and dialogue history.

cp .env.example .env

Fill the Doubao APP ID/ACCESS TOKEN (Can be obtained from this tutorial) and the LeapTalk model paths in .env:

DOUBAO_APP_ID=YOUR_DOUBAO_APP_ID
DOUBAO_ACCESS_TOKEN=YOUR_DOUBAO_ACCESS_TOKEN
LEAPTALK_CKPT_DIR="./models/SoulX-FlashHead-1_3B"
LEAPTALK_WAV2VEC_DIR="./models/wav2vec2-base-960h"
LEAPTALK_LORA_DIR="./models/leaptalk"
LEAPTALK_AUDIO_PROJ="./models/leaptalk/audio_proj_step_10400.pt"

DOUBAO_API_KEY is kept as an optional fallback, but the web demo prefers the DOUBAO_APP_ID / DOUBAO_ACCESS_TOKEN pair when both are present.

python web_server.py

Open http://localhost:7860, load a portrait image, connect, then type a message or hold the record button to talk with the digital human in real time.

Download the training dataset from Soul-AILab/VividHead:

huggingface-cli download Soul-AILab/VividHead \
  --repo-type dataset \
  --local-dir ./data/VividHead

The training script expects the dataset to contain paired videos and audios:

data/VividHead/
├── videos/
└── audios/

train.sh uses the downloaded base model and wav2vec paths by default:

CKPT_DIR="./models/SoulX-FlashHead-1_3B"
WAV2VEC_DIR="./models/wav2vec2-base-960h"
VIDEO_DIR="./data/VividHead/videos"
AUDIO_DIR="./data/VividHead/audios"
SAVE_DIR="./outputs/train"

bash train.sh

If you find this work useful, please consider citing:

@misc{zhang2026leaptalkbreakinglatencyqualitytradeoff,
      title={LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation}, 
      author={Rongxiang Zhang and Songhua Liu},
      year={2026},
      eprint={2608.00079},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.00079}, 
}