📑 Technical Report · 👋 Playground · 💬 Discord · 🤗 Hugging Face · 🤖 ModelScope · 🚀 Demo

Welcome to the SkyCaptioner-V1 repository! Here, you'll find the structural video captioning model weights and inference code for our video captioner that labels the video data efficiently and comprehensively.

• May 07, 2025: 🚀 Added a web demo implementation based on Gradio and the online demo is now available!
• Apr 21, 2025: 👋 We release the vllm batch inference code for SkyCaptioner-V1 Model and caption fusion inference code.
• Apr 21, 2025: 👋 We release the first shot-aware video captioning model SkyCaptioner-V1 Model. For more details, please check our paper.

• SkyCaptioner-V1

  • Checkpoints
  • Batch Inference Code
  • Caption Fusion Method
  • Web Demo (Gradio)

SkyCaptioner-V1 is a structural video captioning model designed to generate high-quality, structural descriptions for video data. It integrates specialized sub-expert models and multimodal large language models (MLLMs) with human annotations to address the limitations of general captioners in capturing professional film-related details. Key aspects include:

1. ​​Structural Representation​: Combines general video descriptions (from MLLMs) with sub-expert captioner (e.g., shot types,shot angles, shot positions, camera motions.) and human annotations.
2. ​​Knowledge Distillation​: Distills expertise from sub-expert captioners into a unified model.
3. ​​Application Flexibility​: Generates dense captions for text-to-video (T2V) and concise prompts for image-to-video (I2V) tasks.

Our Video Captioning model captures multi-dimensional details:

• ​​Subjects​: Appearance, action, expression, position, and hierarchical categorization.
• ​​Shot Metadata​: Shot type (e.g., close-up, long shot), shot angle, shot position, camera motion, environment, lighting, etc.

• ​​Shot Captioner​: Classifies shot type, angle, and position with high precision.
• ​​Expression Captioner​: Analyzes facial expressions, emotion intensity, and temporal dynamics.
• ​​Camera Motion Captioner​: Tracks 6DoF camera movements and composite motion types,

• Trained on ~2M high-quality, concept-balanced videos curated from 10M raw samples.
• Fine-tuned on Qwen2.5-VL-7B-Instruct with a global batch size of 512 across 32 A800 GPUs.
• Optimized using AdamW (learning rate: 1e-5) for 2 epochs.

• Adapts output length based on application (T2V/I2V).
• Employs LLM Model to fusion structural fields to get a natural and fluency caption for downstream tasks.

SkyCaptioner-V1 demonstrates significant improvements over existing models in key film-specific captioning tasks, particularly in ​shot-language understanding and ​​domain-specific precision​. The differences stem from its structural architecture and expert-guided training:

1. ​​Superior Shot-Language Understanding​:
   • ​Our Captioner model outperforms Qwen2.5-VL-72B with +11.2% in shot type, +16.1% in shot angle, and +50.4% in shot position accuracy. Because SkyCaptioner-V1’s specialized shot classifiers outperform generalist MLLMs, which lack film-domain fine-tuning.
   • ​+28.5% accuracy in camera motion vs. Tarsier2-recap-7B (88.8% vs. 41.5%): Its 6DoF motion analysis and active learning pipeline address ambiguities in composite motions (e.g., tracking + panning) that challenge generic captioners.
2. ​​High domain-specific precision​:
   • ​​Expression accuracy​: ​68.8% vs. 54.3% (Tarsier2-recap-7B), leveraging temporal-aware S2D frameworks to capture dynamic facial changes.

Our SkyCaptioner-V1 model can be downloaded from SkyCaptioner-V1 Model. We use Qwen2.5-32B-Instruct as our caption fusion model to intelligently combine structured caption fields, producing either dense or sparse final captions depending on application requirements.

# download SkyCaptioner-V1
huggingface-cli download Skywork/SkyCaptioner-V1 --local-dir /path/to/your_local_model_path
# download Qwen2.5-32B-Instruct
huggingface-cli download Qwen/Qwen2.5-32B-Instruct --local-dir /path/to/your_local_model_path2

Begin by cloning the repository:

git clone https://github.com/SkyworkAI/SkyReels-V2
cd skycaptioner_v1

We recommend Python 3.10 and CUDA version 12.2 for the manual installation.

pip install -r requirements.txt

export SkyCaptioner_V1_Model_PATH="/path/to/your_local_model_path"

python scripts/vllm_struct_caption.py \
    --model_path ${SkyCaptioner_V1_Model_PATH} \
    --input_csv "./examples/test.csv" \
    --out_csv "./examples/test_result.csv" \
    --tp 1 \
    --bs 4

export LLM_MODEL_PATH="/path/to/your_local_model_path2"

python scripts/vllm_fusion_caption.py \
    --model_path ${LLM_MODEL_PATH} \
    --input_csv "./examples/test_result.csv" \
    --out_csv "./examples/test_result_caption.csv" \
    --bs 4 \
    --tp 1 \
    --task t2v

Note:

• If you want to get i2v caption, just change the --task t2v to --task i2v in your Command.

Launch the Gradio web demo for SkyCaptioner-V1:

export SkyCaptioner_V1_Model_PATH="/path/to/your_local_model_path"
python scripts/gradio_struct_caption.py \
    --skycaptioner_model_path ${SkyCaptioner_V1_Model_PATH}

Launch the Gradio web demo for Caption Fusion:

export LLM_MODEL_PATH="/path/to/your_local_model_path2"
python scripts/gradio_fusion_caption.py \
    --fusioncaptioner_model_path ${LLM_MODEL_PATH} \

We would like to thank the contributors of Qwen2.5-VL, tarsier2 and vllm repositories, for their open research and contributions.

@misc{chen2025skyreelsv2infinitelengthfilmgenerative,
author = {Guibin Chen and Dixuan Lin and Jiangping Yang and Chunze Lin and Junchen Zhu and Mingyuan Fan and Hao Zhang and Sheng Chen and Zheng Chen and Chengcheng Ma and Weiming Xiong and Wei Wang and Nuo Pang and Kang Kang and Zhiheng Xu and Yuzhe Jin and Yupeng Liang and Yubing Song and Peng Zhao and Boyuan Xu and Di Qiu and Debang Li and Zhengcong Fei and Yang Li and Yahui Zhou},
title = {Skyreels V2:Infinite-Length Film Generative Model},
year = {2025},
eprint={2504.13074},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2504.13074}
}