Larger dataset should not be the work only for large group with more resources.

LoMo-Pipe is the first open-sourced LLM-assist video understanding benchmark construction pipeline. Here are four stages for data construction:

• Stage 1: Raw Data Collection - a downloader to download videos from the Internet as raw data;
• Stage 2: Subtitles Processing - a program to convert the audio into the text description with time stamps and a translator from Chinese to English;
• Stage 3: Information Compression - a program to compress the substitles into a conlusion and extract effective informations;
• Stage 4: QA Pairs Generation - we provide some manuscripts to generate different kinds of QA pairs.

LoMo-Pipe is a automatic data annotation pipeline with Lower cost and More efficiency. We carefully select the models without high computational resources requirements and sucussfully run the whole project with on 2 RTX 4090 GPUs, which is practical for most AI researchers (Sorry for if not 😢).

To run this project, you need do the preparation for the following steps:

You need to put the SESSDATA into the file bilibili_downloader.py:

SESSDATA = "Put Your SESSDATA of Bilibili here"

You can get your SESSDATA of Bilibili account from this tutorial (only for Chinese). If you want to get the video from Youtube, you can go through youtube-dl project.

⚠️ Attention: This code is only for research use. You can not utilize this code for commercial use!

For audio2text.py, please follow the instruction in faster-whisper to use the faster version OpenAI whisper, or run the following line:

pip install faster-whisper

Also, you can use original OpenAI whisper, or run the following line:

pip install -U openai-whisper

For chinese2english_translator.py, please following the instruction in hf_hub_ctranslate2, or run the following line:

pip install hf-hub-ctranslate2

We utilize Qwen to automaticly generate the data, please following the instruction Qwen2.5-7B-Instruct.

We load the weight in offline way. You need to download the weights to the folder weights by the following line:

huggingface-cli download --resume-download Qwen/Qwen2.5-7B-Instruct --local-dir Qwen2.5-7B-Instruct

# If huggingface cannot use
modelscope download --model Qwen/Qwen2.5-7B-Instruct

The argument for the manuscripts:

• bilibili_downloader.py

  • --bvid: bvid number of the video in Bilibili. You can get the bvid of the video from Bilibili following the instruction (Only Chinese)
  • --output: output folder
  • --resolution: the resolution of the video, you can choose from [1080P, 720P]

• audio2text.py

  • --video_path: path to the video need to be convert into text
  • --sub_path: path to save the text
  • --audio_path: path to save the audio
  • --log_path: path to save the log file

• other manuscripts

  • --input_folder: path to the folder containing Chinese subtitles
  • --output_folder: path to the folder to save English subtitles

We utilize this pipeline to form the LoMo Video benchmark with 14,000+ videos. You can find the LoMo benchmark in HuggingFace.

If you have any question, please contect by E-mail: huchengyang@sjtu.edu.cn

We thank the following open-source projects:

• youtube-dl
• faster-whisper
• whisper
• CTranslate
• Qwen