Savaal

Savaal generates questions that test understanding, not recall — from documents spanning dozens to hundreds of pages. Instead of stuffing a whole document into one prompt (which yields shallow, repetitive questions), Savaal extracts and ranks the document's main ideas, retrieves grounding passages for each, and generates a question per idea:

Each LLM call stays small and self-contained, so quality holds as documents grow.

• 📈 Scalable — handles hundreds of pages with cost that grows sub-linearly.
• 🔍 Deep — targets ranked key concepts, so questions require reasoning, not memorization.
• 🌐 Domain-independent — no fine-tuning or domain-specific prompting.

We evaluated Savaal with human experts on research papers and PhD dissertations. Compared to a direct-prompting LLM baseline, Savaal generates questions that better test depth of understanding by 6.5× for dissertations and 1.5× for papers — and its advantage in question quality and cost grows as documents get longer.

Human ratings on PhD dissertations: experts judged Savaal's questions poor far less often than the baseline's (lower is better).

A Savaal question on the Attention Is All You Need paper:

{
  "main_idea": "Transformers use multi-headed self-attention to model dependencies regardless of distance...",
  "question": {
    "question": "Which of the following best explains why Transformer models are considered advantageous for tasks beyond text translation, such as processing images, audio, and video?",
    "options": [
      "A. Transformer models utilize multi-headed self-attention, which allows them to model dependencies without regard to their distance, making them suitable for handling large inputs and outputs like images and audio.",
      "B. The sequential nature of Transformer models makes them ideal for processing continuous data streams such as video and audio, where maintaining order is crucial.",
      "C. The reliance on recurrent layers in Transformer models enables them to efficiently process non-textual data by leveraging sequence-aligned recurrence.",
      "D. Transformer models are specifically designed to handle text data, and their application to other modalities is limited due to their text-centric architecture."
    ],
    "answer": "A. Transformer models utilize multi-headed self-attention, which allows them to model dependencies without regard to their distance, making them suitable for handling large inputs and outputs like images and audio."
  }
}

Option A — conda (recommended)

conda env create -f environment.yml
conda activate savaal

Option B — uv / pip

uv venv --python 3.11 .venv
uv pip install -r requirements.txt
source .venv/bin/activate

docker run -d --name grobid -p 8070:8070 lfoppiano/grobid:0.8.0

cp .env.example .env.local   # then add your OPENAI_API_KEY

OPENAI_API_KEY is required. WANDB_API_KEY / WEAVE_PROJECT are optional (tracing) — leave WEAVE_PROJECT unset to run without it.

Index the document once, then generate:

python src/main.py index    -f data/example.pdf -i example
python src/main.py generate -f data/example.pdf -o questions.json -n 5 -i example

@article{
  noorbakhsh2026savaal,
  title={Savaal: Scalable Concept-Driven Question Generation to Enhance Human Learning},
  author={Kimia Noorbakhsh and Joseph Chandler and Pantea Karimi and Mohammad Alizadeh and Hari Balakrishnan},
  journal={Transactions on Machine Learning Research},
  issn={2835-8856},
  year={2026},
  url={https://openreview.net/forum?id=2DWDQTsz7K},
  note={}
}