This is the official implementation of paper "EmoCtrl: Controllable Emotional Image Content Generation".

Jingyuan Yang, WeiBin Luo, Hui Huang^*
Shenzhen University

Fig 1. Controllable Emotional Image Content Generation with EmoCtrl. Given a content condition (“Ocean”) and a target emotion (“Contentment”), EmoCtrl generates images that maintain semantic content while vividly expressing accurate emotions.

Download the pre-trained weights required for the program to run.

Qwen3-0.6B: Qwen3-0.6B · 模型库

SDXL: stable-diffusion-xl-base-1.0 · 模型库

If you want to train your own model, you need to download both datasets.

EmoSet: JingyuanYY/EmoSet

EmoEditSet: JingyuanYY/EmoEdit

First, you need to configure the environment.

conda create -n emoctrl python=3.11
conda activate emoctrl

# Install PyTorch with CUDA 11.8 support (must be installed before other dependencies)
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu118

pip install -r requirement.txt

Then you can get started quickly and get results. Test-related parameters can be modified in the file.

python test.py

Fig 2. Overview of EmoCtrl. (a) Textual Emotion Enhancement: Emotion tokens are fused with content text in the LLM to enhance emotion at the semantic level. (b) Visual Emotion Enhancement: Emotion tokens are fused with the textual emotional condition within Stable Diffusion to produce affective images through visual cues. (c) Emotion-driven Preference Optimization: Refines the model with emotion-driven reward for better human alignment.

Stage 1: Train the textual emotion enhancement module, which includes emotion tokens and LoRA for LLM.

# Prepare training data
python data/data_process.py
# Start training
torchrun --nproc_per_node=8 train_stage1.py

Stage 2: Train the visual emotion enhancement module.

# Prepare training data
python offline_data_gen.py
# Start training
accelerate launch --num_processes 8 --multi_gpu --mixed_precision "fp16" train_stage2.py

Stage 3: Fine-tune using EDPO.

accelerate launch train_stage3.py --config configs/grpo.py:emoset_sdxl_llm

You can evaluate the generated images after training.

# Emo-A, CLIP-A and EC-A
python metrics/joint.py
# LPIPS and Sem-C
python metrics/semantic.py

You can visualize the trained textual emotion tokens.

python textual_visualization.py

Fig 3. Comparison with state-of-the-art methods, showing EmoCtrl is superior in both content and emotion aspects.

Fig 4. Ablation study of EmoCtrl, showing the contributions of textual emotion tokens (v_t) and visual emotion tokens (v_v).

Fig 5. Visualization of textual emotion tokens. Each row shows images generated from a specific emotion, while each column reflects diverse semantic expressions. The results show that EmoCtrl produces rich content diversity while maintaining consistent emotional tone.

Fig 6. EmoCtrl can be applied to creative art generation, including (a) style control and (b) multi-emotion condition.

If you find this work useful, please kindly cite our paper:

@misc{yang2026emoctrlcontrollableemotionalimage,
      title={EmoCtrl: Controllable Emotional Image Content Generation}, 
      author={Jingyuan Yang and Weibin Luo and Hui Huang},
      year={2026},
      eprint={2512.22437},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.22437}, 
}