🧐📄 VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal RAG 🎯🤖

Files for the NAACL 2025 paper, VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation. 📚📊🔍

This repo contains the 5 data splits in VisDoMBench, a cutting-edge multi-document, multimodal QA benchmark 🚀🔎 designed for answering questions across visually rich document content like:

📊 Tables | 📉 Charts | 🖼️ Slides

Perfect for evaluating multimodal, multi-document QA systems in a comprehensive way! ✅📖

This repository includes visdomrag.py, our implementation of the VisDoMRAG framework, a multimodal retrieval-augmented pipeline specifically designed for visual document understanding and question answering.

1. Visual Retrieval: Using models like ColPali and ColQwen for image-to-image and text-to-image retrieval
2. Text Retrieval: Supporting BM25, MiniLM, MPNet, and BGE embeddings
3. Multi-Stage Pipeline:
   • Document caching and preprocessing
   • Visual and textual index building
   • Context retrieval based on queries
   • Response generation from each modality
   • Response combination for final answers

config = {
    "data_dir": "path/to/data",
    "output_dir": "path/to/output",
    "llm_model": "gpt4",  # Options: "gpt4", "gemini", "qwen"
    "vision_retriever": "colpali",  # Options: "colpali", "colqwen"
    "text_retriever": "bm25",  # Options: "bm25", "minilm", "mpnet", "bge"
    "top_k": 5,  # Number of contexts to retrieve
    "chunk_size": 3000,  # Text chunk size for retrieval
    "chunk_overlap": 300,  # Overlap between chunks
    "force_reindex": False,  # Whether to rebuild indexes
    "qa_prompt": # Refer to context dataset specific prompts in the code
}

📌 Table: Summary of data splits included in VisDoMBench.

from visdomrag import VisDoMRAG

# Configure the pipeline
config = {
    "data_dir": "./path/to/dataset",
    "output_dir": "./results",
    "llm_model": "gpt4",
    "vision_retriever": "colpali",
    "text_retriever": "bm25",
    "api_keys": {
        "openai": "your-openai-key",
        "gemini": "your-gemini-key"
    }
}

# Initialize and run
pipeline = VisDoMRAG(config)
pipeline.run()  # Process all queries
# Or process a specific query
pipeline.process_query(query_id)

Main requirements:

• PyTorch
• pandas, numpy
• pdf2image, PyPDF2, pytesseract (for PDF processing)
• chromadb, langchain (for text retrieval)
• Optional model-specific dependencies:
  • google.generativeai for Gemini
  • openai for GPT-4
  • colpali_engine for ColPali/ColQwen visual retrievers
  • transformers for Qwen models

@misc{suri2024visdommultidocumentqavisually,
      title={VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation}, 
      author={Manan Suri and Puneet Mathur and Franck Dernoncourt and Kanika Goswami and Ryan A. Rossi and Dinesh Manocha},
      year={2024},
      eprint={2412.10704},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.10704}, 
}