Fork of HOMIE with Quanto qint8 DiT inference so the 14B model fits on lower-VRAM GPUs (e.g. RTX 3090 / 24 GB). Use the pre-quantized weights from IanB/homie-r2v-wan2.1_q8.
Download the q8 model
# q8 DiT: Homie_Wan_14B_q8.safetensors + quantization_map.json
huggingface-cli download IanB/homie-r2v-wan2.1_q8 --local-dir ./weights/HOMIE-Wan-Model-q8Follow the Quick Start guide, but generate with the q8 model
python generate.py \
--task s2v-14B --size 832*480 --frame_num 33 --sample_fps 24 \
--ckpt_dir ./weights/Wan2.1-T2V-14B-Diffusers \
--homie_ckpt ./weights/HOMIE-Wan-Model-q8 \
--homie_ckpt_basename Homie_Wan_14B_q8 \
--quantized_dit --t5_cpu \
--input_json ./eval_examples/meta_file_with_mllm.jsonl \
--save_path ./video_results_q8/ --base_seed 6666
Yiyang Cai* · Nan Chen* · Rongchang Xie · Junwen Pan · Chunyang Jiang · Cheng Chen · Wen Zhou · Zhenbang Sun · Wei Xue · Wenhan Luo · Yike Guo
Hong Kong University of Science and Technology
mailto: ycaicj@connect.ust.hk
TL;DR: HOMIE is a subject-consistent video customization model enhanced by multimodal large language models (MLLMs) that can generate videos with complex human-object combination, including general human-object interaction, abstract content personalization. HOMIE can generate videos with both inter- and intra-subject video generation.
demo_multi_subject.mp4 |
demo_logo.mp4 |
demo_intra_ocr.mp4 |
demo_intra_multiview.mp4 |
HOMIE supports diverse combination of human-object references, both intra- and inter-subject video generation.
🚀 [2026-07-21] We release the technical report, inference codes as well as checkpoints tuned on Wan2.1-T2V-14B.
# Ensure torch >= 2.4.0 with a matching CUDA build.
bash set_env.shHOMIE loads its feature-extraction models (VAE, text encoder, tokenizer) from the public HuggingFace Wan2.1-T2V-14B Diffusers checkpoint. The trained HOMIE weights (DiT + task/reference embeddings + qwen connector) are provided separately. Furthermore, the Qwen3-VL-2B-Thinking model should be downloaded for MLLM feature extraction.
pip install "huggingface_hub[cli]"
mkdir weights
# 1) Wan2.1-T2V-14B (Diffusers layout: provides vae/ text_encoder/ tokenizer/)
huggingface-cli download Wan-AI/Wan2.1-T2V-14B-Diffusers --local-dir ./weights/Wan2.1-T2V-14B-Diffusers
# 2) HOMIE trained weights: place Homie_Wan_14B-*.safetensors and
# Homie_Wan_14B.safetensors.index.json into a directory, e.g. ../HOMIE-Wan-Model
huggingface-cli download yychai/homie-r2v-wan2.1 --local-dir ./weights/HOMIE-Wan-Model
# 3) Qwen3-VL-2B-Thinking (MLLM used to extract the qwen_feature)
huggingface-cli download Qwen/Qwen3-VL-2B-Thinking --local-dir ./weights/Qwen3-VL-2B-Thinking{"reference_paths": [["eval_examples/images/1/human.png"], ["eval_examples/images/1/hat.png"], ["eval_examples/images/1/bottle.png"]], "prompt": "A man wearing a cowboy hat..."}reference_paths— list per subject; each entry's outer index becomes that subject's reference-id letter (a,b,c, ...).prompt— prompt string which describes the video content.
See eval_examples/meta_file.jsonl for an example.
Generate MLLM features with the Qwen3-VL-2B-Thinking model. This reads a meta jsonl,
writes one .pt per sample into $FEATURE_DIR, and emits a new meta jsonl with an mllm_feature path added to each line.
MLLM_CKPT=${MLLM_CKPT:-./weights/Qwen3-VL-2B-Thinking}
INPUT_JSON=${INPUT_JSON:-eval_examples/meta_file.jsonl}
FEATURE_DIR=${FEATURE_DIR:-eval_examples/mllm_features}
OUTPUT_JSON=${OUTPUT_JSON:-eval_examples/meta_file_with_mllm.jsonl}
python generate_mllm_feature.py \
--meta_file "$INPUT_JSON" \
--output_meta "$OUTPUT_JSON" \
--feature_dir "$FEATURE_DIR" \
--model_path "$MLLM_CKPT"See extract_mllm_feature.sh for ready-to-copy commands.
python generate.py \
--task s2v-14B --size 832*480 --frame_num 97 --sample_fps 24 \
--ckpt_dir ./weights/Wan2.1-T2V-14B-Diffusers \
--homie_ckpt ./weights/HOMIE-Wan-Model \
--input_json ./eval_datasets/meta_file_with_mllm.jsonl \
--save_path ./video_results_480p/ --base_seed 6666torchrun --nproc_per_node=8 generate.py \
--task s2v-14B --size 832*480 --frame_num 97 --sample_fps 24 \
--ckpt_dir ./weights/Wan2.1-T2V-14B-Diffusers \
--homie_ckpt ./weights/HOMIE-Wan-Model \
--input_json ./eval_datasets/meta_file_with_mllm.jsonl \
--save_path ./video_results_480p/ --dit_fsdp --t5_fsdp --base_seed 6666See infer.sh and infer_720p.sh for ready-to-copy commands.
This project is licensed under the Apache License 2.0.
This project is based on the work of Wan2.1 and Phantom. We thank their authors for the open-source contributions.
If you use this project in your research, please cite the following paper:
@misc{cai2026homiehumanobjectcentricvideo,
title={HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enchancement},
author={Yiyang Cai and Nan Chen and Rongchang Xie and Junwen Pan and Chunyang Jiang and Cheng Chen and Wen Zhou and Zhenbang Sun and Wei Xue and Wenhan Luo and Yike Guo},
year={2026},
eprint={2607.18217},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2607.18217},
}