Senmao Li1,3, Kai Wang2, Salman Khan3, Fahad Shahbaz Khan3,4, Jian Yang1, Yaxing Wang1
1 Nankai University, 2 City University of Hong Kong (Dongguan), China, 3 MBZUAI, 4 Linkoping University
Figure 1. Overview of the proposed FasterVAR framework. We retain the original VAR inference process for the semantic and structure establishment stages, while exploiting semantic irrelevance and low-rank properties in the fidelity refinement stage to accelerate inference.
Figure 2. Qualitative comparison with the vanilla Infinity-2B, Infinity-8B, and STAR models (1st, 3rd, and 5th rows). Our FasterVAR (2nd, 4th, and 6th rows) achieves a 3.4x, 2.7x, and 1.74x speedup while maintaining performance.
Quantitative Results on the GenEval and DPG benchmarks
We apply FasterVAR to three text-to-image VAR models: Infinity, STAR, and HART. FasterVAR can also be easily integrated into other VAR models for efficient inference acceleration.
Infinity
cd ./Infinity
python tools/interactive_infer.pySTAR
cd ./STAR-T2I
python sample.py --model_path taocrayon/STAR/star_rope_d30_1024_drop_3-ar-ckpt-ep1-iter21000.pth \
--text_model_path taocrayon/STAR/CLIP \
--vae_path FoundationVision/var/vae_ch160v4096z32.pthHART
cd ./hart
python sample_fastervar.py --model_path mit-han-lab/hart-0.7b-1024px/llm \
--text_model_path mit-han-lab/Qwen2-VL-1.5B-Instruct \
--shield_model_path mit-han-lab/hart-0.7b-1024px \
--prompt "A cinematic shot of robot with colorful feathers"InfinityStar
FasterVAR can be integrated into video generative models based on next-scale prediction, such as InfinityStar, delivering 1.8× and 2.7× speedups at 480p and 720p, respectively, with negligible degradation in visual quality. See ./InfinityStar for details.
Please cite our paper if you find this work useful for your research:
@inproceedings{lifastervar,
title={FasterVAR: Plug-and-Play Acceleration for Visual Autoregressive Models},
author={Li, Senmao and Wang, Kai and Khan, Salman and Khan, Fahad Shahbaz and Wang, Yaxing and others},
booktitle={Forty-third International Conference on Machine Learning}
}⭐ If FasterVAR is helpful to your projects, please help star this repo. Thanks! 🤗