Project QA benchmarks onto LLM pre-training corpora.

NanoKnow identifies which benchmark questions have answers in a model's training data, enabling controlled studies of parametric knowledge vs. retrieval-augmented generation (RAG).

🎉 NanoKnow was accepted to SIGIR '26!

Update: We have new projections for ClimbMix-400B! ClimbMix-400B is the corpus for the 2026 TREC RAG track, which aims to build reusable collections that enable controlled studies of parametric and retrieval knowledge. See the ClimbMix projection guide for details on how these projections were produced.

Arxiv: https://arxiv.org/abs/2602.20122

Given a QA benchmark and a pre-training corpus, NanoKnow produces relevance judgments (qrels) that partition questions into:

• Supported: The answer exists in the training data (the model could have memorized it).
• Unsupported: The answer does not appear in the training data.

The pipeline has three stages:

1. BM25 Retrieval — Search the corpus for candidate documents using the question as a query.
2. Answer String Matching — Filter to documents that contain the gold answer as a substring.
3. LLM Verification — Use an LLM judge to filter out coincidental matches (e.g., "Paris" in a passage about Paris, Texas).

We provide pre-built qrels for karpathy/fineweb-edu-100b-shuffle, the corpus used to train nanochat, and for ClimbMix-400B:

The pre-built files are organized by dataset under questions-and-qrels/:

questions-and-qrels/
├── nq/
│   ├── answers.nanoknow-nq.jsonl
│   ├── climbmix/
│   │   ├── qrels.nanoknow-nq-climbmix.supported.txt
│   │   ├── topics.nanoknow-nq-climbmix.supported.tsv
│   │   └── topics.nanoknow-nq-climbmix.unsupported.tsv
│   └── fineweb-edu/
│       ├── qrels.nanoknow-nq-fineweb.supported.txt
│       ├── topics.nanoknow-nq-fineweb.supported.tsv
│       └── topics.nanoknow-nq-fineweb.unsupported.tsv
└── squad/
    ├── answers.nanoknow-squad.jsonl
    ├── climbmix/
    │   ├── qrels.nanoknow-squad-climbmix.supported.txt
    │   ├── topics.nanoknow-squad-climbmix.supported.tsv
    │   └── topics.nanoknow-squad-climbmix.unsupported.tsv
    └── fineweb-edu/
        ├── qrels.nanoknow-squad-fineweb.supported.txt
        ├── topics.nanoknow-squad-fineweb.supported.tsv
        └── topics.nanoknow-squad-fineweb.unsupported.tsv

Each dataset directory contains:

• answers.nanoknow-<dataset>.jsonl: gold answers as one JSON object per line, e.g. {"qid": "0", "answer": ["14 December 1972 UTC", "December 1972"]}.
• <corpus>/topics.nanoknow-<dataset>-<corpus>.supported.tsv: corpus-supported questions as qid<TAB>question.
• <corpus>/topics.nanoknow-<dataset>-<corpus>.unsupported.tsv: corpus-unsupported questions as qid<TAB>question.
• <corpus>/qrels.nanoknow-<dataset>-<corpus>.supported.txt: TREC-format qrels for supported questions, e.g. 0 Q0 shard_01177_50695 1.

FineWeb-Edu files use the corpus slug fineweb in filenames and the directory name fineweb-edu.

pip install -r requirements.txt

For BM25 retrieval, you also need Java 11+:

# Ubuntu/Debian
sudo apt install openjdk-11-jdk

We release a pre-built Lucene index over karpathy/fineweb-edu-100b-shuffle (326 GB):

Download: LingweiGu/NanoKnow-Fineweb-Edu-Index

huggingface-cli download LingweiGu/NanoKnow-Fineweb-Edu-Index --repo-type dataset --local-dir ./fineweb-edu-index

To build the index yourself using Anserini:

bin/run.sh io.anserini.index.IndexCollection \
  -collection FinewebCollection \
  -input /path/to/corpus \
  -index /output/directory \
  -generator DefaultLuceneDocumentGenerator \
  -threads 16

# Stage 1: BM25 retrieval + answer matching (CPU only)
python scripts/project.py \
    --dataset squad \
    --stage 1 \
    --index_path /path/to/lucene-index \
    --output output/squad_stage1.pkl

# Stage 1 using the Pyserini REST API, e.g. ClimbMix
export PYSERINI_API_TOKEN="..."
python scripts/project.py \
    --dataset squad \
    --stage 1 \
    --retriever api \
    --index_path climbmix-400b \
    --output output/squad_climbmix_stage1.pkl

# Stage 2: LLM verification (requires GPU)
python scripts/project.py \
    --stage 2 \
    --input output/squad_stage1.pkl \
    --output output/squad_stage2.pkl

# Export Stage 2 output to NanoKnow topics/qrels files
python scripts/export_stage2_qrels.py \
    --input output/squad_climbmix_stage2.pkl \
    --dataset squad \
    --corpus climbmix

# Or run both stages together
python scripts/project.py \
    --dataset squad \
    --stage both \
    --index_path /path/to/lucene-index \
    --output output/squad_projected.pkl

NANOCHAT_DIR="${NANOCHAT_DIR:-/path/to/nanochat}"
CHECKPOINT_DIR="${CHECKPOINT_DIR:-/path/to/nanochat-checkpoint}"
STEP="${STEP:?Set STEP to the checkpoint step to evaluate}"
DATASET="${DATASET:-nq}"
QRELS_DIR="${QRELS_DIR:-questions-and-qrels/${DATASET}}"
CORPUS="${CORPUS:-fineweb}"
FINEWEB_INDEX_PATH="${FINEWEB_INDEX_PATH:-/path/to/fineweb-index}"
OUTPUT_DIR="${OUTPUT_DIR:-output}"
DEVICE="${DEVICE:-cuda}"

python scripts/nanochat_inference.py \
    --nanochat-dir "${NANOCHAT_DIR}" \
    --checkpoint_dir "${CHECKPOINT_DIR}" \
    --step "${STEP}" \
    --dataset "${DATASET}" \
    --qrels_dir "${QRELS_DIR}" \
    --corpus "${CORPUS}" \
    --fineweb_index_path "${FINEWEB_INDEX_PATH}" \
    --output_dir "${OUTPUT_DIR}" \
    --device "${DEVICE}"

If nanochat is already importable in your Python environment, --nanochat-dir can be omitted. For repeated runs, you can set NANOCHAT_DIR=/path/to/nanochat instead of passing the argument each time.

The output file is named from the checkpoint directory basename and dataset, for example output/karpathy_nanochat_d32_nq.

python scripts/evaluate_model_predictions.py \
    --input_file output/karpathy_nanochat_d32_nq \
    --output_file nanochat_evaluations/karpathy_nanochat_d32_nq_scored.pkl

python scripts/get_eval_scores.py nanochat_evaluations/karpathy_nanochat_d32_nq_scored.pkl

Example output:

{
  "supported_closed_book": {
    "count": 2389,
    "exact_match_accuracy": 0.19589786521557137,
    "llm_judge_accuracy": 0.2293846797823357
  },
  "supported_w_fineweb_context": {
    "count": 2389,
    "exact_match_accuracy": 0.46923398911678527,
    "llm_judge_accuracy": 0.46672247802427796
  },
  "unsupported_closed_book": {
    "count": 1221,
    "exact_match_accuracy": 0.00819000819000819,
    "llm_judge_accuracy": 0.04914004914004914
  }
}

Replication check: on June 1, 2026, we re-ran the NQ evaluation for karpathy_nanochat_d32 at step 650. Inference completed in 36m56s, scoring completed in 25m40s, and the reproduced scores were:

{
  "supported_closed_book": {
    "count": 2389,
    "exact_match_accuracy": 0.19715362076182502,
    "llm_judge_accuracy": 0.22980326496442027
  },
  "supported_w_fineweb_context": {
    "count": 2389,
    "exact_match_accuracy": 0.4679782335705316,
    "llm_judge_accuracy": 0.47216408539137716
  },
  "unsupported_closed_book": {
    "count": 1221,
    "exact_match_accuracy": 0.00819000819000819,
    "llm_judge_accuracy": 0.04504504504504504
  }
}

NanoKnow/
├── nanoknow/                  # Core library
│   ├── retriever.py           # Stage 1: BM25 retrieval + answer matching
│   ├── verifier.py            # Stage 2: LLM-based verification
│   └── evaluator.py           # Evaluation utilities
├── scripts/                   # Runnable scripts
│   ├── project.py             # Run the projection pipeline
│   ├── export_stage2_qrels.py # Export Stage 2 pickles to topics/qrels files
│   ├── nanochat_inference.py  # Run nanochat checkpoint inference
│   ├── evaluate_model_predictions.py  # Score predictions
│   └── get_eval_scores.py     # Summarize scored evaluation results
├── questions-and-qrels/       # Pre-built benchmark questions, answers, and qrels
│   ├── nq/
│   │   ├── answers.nanoknow-nq.jsonl
│   │   ├── climbmix/
│   │   │   ├── qrels.nanoknow-nq-climbmix.supported.txt
│   │   │   ├── topics.nanoknow-nq-climbmix.supported.tsv
│   │   │   └── topics.nanoknow-nq-climbmix.unsupported.tsv
│   │   └── fineweb-edu/
│   │       ├── qrels.nanoknow-nq-fineweb.supported.txt
│   │       ├── topics.nanoknow-nq-fineweb.supported.tsv
│   │       └── topics.nanoknow-nq-fineweb.unsupported.tsv
│   └── squad/
│       ├── answers.nanoknow-squad.jsonl
│       ├── climbmix/
│       │   ├── qrels.nanoknow-squad-climbmix.supported.txt
│       │   ├── topics.nanoknow-squad-climbmix.supported.tsv
│       │   └── topics.nanoknow-squad-climbmix.unsupported.tsv
│       └── fineweb-edu/
│           ├── qrels.nanoknow-squad-fineweb.supported.txt
│           ├── topics.nanoknow-squad-fineweb.supported.tsv
│           └── topics.nanoknow-squad-fineweb.unsupported.tsv
├── pyproject.toml
├── requirements.txt
├── LICENSE
└── README.md

We evaluated eight checkpoints across three model scales:

@article{gu2026nanoknow,
  title={NanoKnow: How to Know What Your Language Model Knows},
  author={Gu, Lingwei and Jedidi, Nour and Lin, Jimmy},
  journal={arXiv preprint arXiv:2602.20122},
  year={2026}
}

Apache 2.0