Compact, modality-rich captions for scalable, agentic video understanding in RAG pipelines
Video-Enriched Retrieval Augmented Generation Using Aligned Video Captions [Paper]
Kevin Dela Rosa
Smart Routing for Multimodal Video Retrieval: When to Search What [Paper]
Kevin Dela Rosa
Stay tuned for follow up works using this dataset and future extensions.
- [2025/07/04] 🎞️ Smart Routing for Multimodal Video Retrieval: When to Search What accepted to Multimodal Representation and Retrieval at ICCV 2025 | First usage of upcoming AVC 1.8M dataset
- [2025/04/28] 🎞️ Video-Enriched Retrieval Augmented Generation Using Aligned Video Captions accepted to Agent4IR at ACM WWW 2025 | [Paper]
- [2024/07/18] Early version of work presented at Multimodal Representation and Retrieval at SIGIR 2024
The following are datasets artifacts supporting the 🎞️ Video-Enriched Retrieval Augmented Generation Using Aligned Video Captions publication.
We curated a dataset based on public youtube videos sampled from Panda-70M, which contains individual clip segments and a general visual scene caption learned from a set of open source video captioning models. Specifically we sampled roughly 2,000 videos from each of 15 top level YouTube categories, resulting in a dataset of 29,259 videos (1.5M video clips and corresponding visual captions) or roughly 215 days of footage. We then augmented that dataset with video subtitles gathered from the source videos and created the aligned transcripts.
Statistics for Aligned Video Caption Dataset
| DATASET DIMENSION | TOTAL | MEDIAN |
|---|---|---|
| Video Count | 29,259 | - |
| Scene Count | 1,476,462 | 31.00 |
| Video Duration (seconds) | 18,584,396 | 478.00 |
| Text Character Length | ||
| Title | 1,548,810 | 51.00 |
| Description | 30,565,705 | 780.00 |
| Title + Description | 32,114,515 | 833.00 |
| Visual Video Captions | 96,888,187 | 2,016.00 |
| Subtitles / ASR | 141,926,062 | 3,472.00 |
| Aligned Captions | 276,019,918 | 6,461.00 |
Dataset Artifacts
| Artifact | Download |
|---|---|
| Video Ids | link (343KB) |
| Aligned Video Captions + Limited Metadata | link (602MB) |
| Generated Questions from RAG feasibility Study | link (252KB) |
If you wish to download the original videos and full video metadata, follow instructions from here. The ChatGPT prompts used for generating video summaries, questions, and automatic judging are provided in prompts folder.
If you find Aligned Video Captions useful for your research and applications, please cite using this BibTeX:
@inproceedings{10.1145/3701716.3716890,
author = {Dela Rosa, Kevin},
title = {Video-Enriched Retrieval Augmented Generation Using Aligned Video Captions},
year = {2025},
isbn = {9798400713316},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3701716.3716890},
doi = {10.1145/3701716.3716890},
booktitle = {Companion Proceedings of the ACM on Web Conference 2025},
pages = {1663–1667},
numpages = {5},
keywords = {agentic information retrieval, chatbots, multimodal retrieval},
location = {Sydney NSW, Australia},
series = {WWW '25}
}