HybridEmo

Yan Zhou, Yun Hong, Yang Feng

HybridEmo is developed by post-training CosyVoice 3 for instruction-following multi-emotion speech synthesis. It is designed to model complex emotional expressions within a single utterance and supports two complementary generation tasks:

• Emotion trajectory: multiple emotions are expressed sequentially and evolve over the course of an utterance.
• Emotion blending: multiple emotions are expressed simultaneously throughout an utterance.

Given a natural-language instruction and synthesis text, HybridEmo generates speech that follows the specified emotional trajectory or blended emotional state. Audio samples are available on our demo page.

HybridEmo is post-trained with Group Relative Policy Optimization (GRPO) using a sample-aware hybrid reward. Given synthesis text, a natural-language emotion instruction, and timbre reference audio, the TTS policy generates candidate speech samples. A task-type router then selects the appropriate reward branch: all samples receive an ASR reward for intelligibility, while trajectory samples additionally receive a segment-level emotion consistency reward and blending samples receive a GMM-based mixture-density reward. These rewards are combined according to the sample type and used to guide the GRPO update toward instruction-aligned multi-emotion speech generation.

The inference code is based on CosyVoice. We recommend using Python 3.10 and a CUDA-enabled Linux environment.

# Clone HybridEmo together with the pinned CosyVoice submodule.
git clone --recursive https://github.com/ictnlp/HybridEmo.git
cd HybridEmo

conda create -n hybridemo python=3.10 -y
conda activate hybridemo

# Install the CosyVoice dependencies.
pip install -r third_party/CosyVoice/requirements.txt

If you have already cloned HybridEmo without --recursive, initialize the submodule with:

git submodule update --init --recursive

On Ubuntu, sox may also be required:

sudo apt-get install sox libsox-dev

Download the HybridEmo model checkpoint from our Hugging Face Collection, which also hosts the MultiEmo-Test evaluation set:

pip install -U huggingface_hub
hf download ICTNLP/HybridEmo --local-dir pretrained_models/HybridEmo

HybridEmo takes three inputs:

• text to synthesize;
• a natural-language emotion instruction;
• a prompt audio file that provides the target speaker timbre.

For emotion trajectory, describe how the emotions should evolve sequentially over the utterance:

python inference.py \
  --model-dir pretrained_models/HybridEmo \
  --prompt-audio /path/to/prompt.wav \
  --instruction "Start with a happy tone, then gradually transition into a fearful tone." \
  --text "I finally reached the top of the mountain, but looking over the edge made my heart race." \
  --output outputs/trajectory.wav

For emotion blending, describe the same emotions as occurring simultaneously throughout the utterance:

python inference.py \
  --model-dir pretrained_models/HybridEmo \
  --prompt-audio /path/to/prompt.wav \
  --instruction "Blend happiness and fear simultaneously throughout the utterance." \
  --text "I finally reached the top of the mountain, but looking over the edge made my heart race." \
  --output outputs/blending.wav

By default, the script looks for CosyVoice at third_party/CosyVoice. Use --cosyvoice-dir if it is installed elsewhere.

If you use HybridEmo in your research, please cite our paper:

@misc{zhou2026sequentialtrajectoriessimultaneousblending,
  title={Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS},
  author={Yan Zhou and Yun Hong and Yang Feng},
  year={2026},
  eprint={2608.30325},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2608.30325},
}

If you have questions, please contact: zhouyan23z@ict.ac.cn.