HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enhancement

TL;DR: HOMIE is a subject-consistent video customization model enhanced by multimodal large language models (MLLMs) that can generate videos with complex human-object combination, including general human-object interaction, abstract content personalization. HOMIE can generate videos with both inter- and intra-subject video generation.

HOMIE supports diverse combination of human-object references, both intra- and inter-subject video generation.

🚀 [2026-07-21] We release the technical report, inference codes as well as checkpoints tuned on Wan2.1-T2V-14B.

# Ensure torch >= 2.4.0 with a matching CUDA build.
bash set_env.sh

HOMIE loads its feature-extraction models (VAE, text encoder, tokenizer) from the public HuggingFace Wan2.1-T2V-14B Diffusers checkpoint. The trained HOMIE weights (DiT + task/reference embeddings + qwen connector) are provided separately. Furthermore, the Qwen3-VL-2B-Thinking model should be downloaded for MLLM feature extraction.

pip install "huggingface_hub[cli]"

mkdir weights

# 1) Wan2.1-T2V-14B (Diffusers layout: provides vae/ text_encoder/ tokenizer/)
huggingface-cli download Wan-AI/Wan2.1-T2V-14B-Diffusers --local-dir ./weights/Wan2.1-T2V-14B-Diffusers

# 2) HOMIE trained weights: place Homie_Wan_14B-*.safetensors and
#    Homie_Wan_14B.safetensors.index.json into a directory, e.g. ../HOMIE-Wan-Model
huggingface-cli download yychai/homie-r2v-wan2.1 --local-dir ./weights/HOMIE-Wan-Model

# 3) Qwen3-VL-2B-Thinking (MLLM used to extract the qwen_feature)
huggingface-cli download Qwen/Qwen3-VL-2B-Thinking --local-dir ./weights/Qwen3-VL-2B-Thinking

{"reference_paths": [["eval_examples/images/1/human.png"], ["eval_examples/images/1/hat.png"], ["eval_examples/images/1/bottle.png"]], "prompt": "A man wearing a cowboy hat..."}

• reference_paths — list per subject; each entry's outer index becomes that subject's reference-id letter (a, b, c, ...).
• prompt — prompt string which describes the video content.

See eval_examples/meta_file.jsonl for an example.

Generate MLLM features with the Qwen3-VL-2B-Thinking model. This reads a meta jsonl, writes one .pt per sample into $FEATURE_DIR, and emits a new meta jsonl with an mllm_feature path added to each line.

MLLM_CKPT=${MLLM_CKPT:-./weights/Qwen3-VL-2B-Thinking}
INPUT_JSON=${INPUT_JSON:-eval_examples/meta_file.jsonl}
FEATURE_DIR=${FEATURE_DIR:-eval_examples/mllm_features}
OUTPUT_JSON=${OUTPUT_JSON:-eval_examples/meta_file_with_mllm.jsonl}

python generate_mllm_feature.py \
    --meta_file "$INPUT_JSON" \
    --output_meta "$OUTPUT_JSON" \
    --feature_dir "$FEATURE_DIR" \
    --model_path "$MLLM_CKPT"

See extract_mllm_feature.sh for ready-to-copy commands.

python generate.py \
    --task s2v-14B --size 832*480 --frame_num 97 --sample_fps 24 \
    --ckpt_dir ./weights/Wan2.1-T2V-14B-Diffusers \
    --homie_ckpt ./weights/HOMIE-Wan-Model \
    --input_json ./eval_examples/meta_file_with_mllm.jsonl \
    --save_path ./video_results_480p/ --base_seed 6666

torchrun --nproc_per_node=8 generate.py \
    --task s2v-14B --size 832*480 --frame_num 97 --sample_fps 24 \
    --ckpt_dir ./weights/Wan2.1-T2V-14B-Diffusers \
    --homie_ckpt ./weights/HOMIE-Wan-Model \
    --input_json ./eval_examples/meta_file_with_mllm.jsonl \
    --save_path ./video_results_480p/ --dit_fsdp --t5_fsdp --base_seed 6666

See infer.sh and infer_720p.sh for ready-to-copy commands.

Context parallel (CP) shards the DiT token sequence across the GPUs in a CP group. On an 8×A100 machine, set --ulysses_size 8 and --ring_size 1 so all eight GPUs cooperate on each generated sample. The example below also enables FSDP for the DiT and T5 weights:

bash infer_cp.sh

infer_cp.sh defaults to NPROC_PER_NODE=8, ULYSSES_SIZE=8, and RING_SIZE=1. You can override these variables when launching the script, for example MASTER_PORT=23456 ./infer_cp.sh.

This project is licensed under the Apache License 2.0.

This project is based on the work of Wan2.1 and Phantom. We thank their authors for the open-source contributions.

If you use this project in your research, please cite the following paper:

@misc{cai2026homiehumanobjectcentricvideo,
      title={HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enchancement}, 
      author={Yiyang Cai and Nan Chen and Rongchang Xie and Junwen Pan and Chunyang Jiang and Cheng Chen and Wen Zhou and Zhenbang Sun and Wei Xue and Wenhan Luo and Yike Guo},
      year={2026},
      eprint={2607.18217},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.18217}, 
}