RAG Index Benchmark Suite

Stop Guessing. Measure Your Retrieval Layer.

Most RAG systems don't fail because of embeddings. They fail because the index silently drops the documents that matter.

This repository contains a fully reproducible benchmark that shows exactly how Flat, IVF, and HNSW behave across different dataset sizes, different query loads, and different parameter sweeps. If you care about recall, latency, or scaling real systems, this suite gives you the numbers, not the vibes.

🔗 Full article: “RAG Isn't About Embeddings — It's About the Index You Chose” https://nidly.substack.com/p/rag-isnt-about-embeddings

This benchmark suite is built to answer one question:

“Which index should I actually use for my dataset and my latency budget?”

Inside the suite:

• Flat (Exact)
• IVF (tunable nlist, nprobe)
• HNSW (M, efConstruction, efSearch)

We compute ground truth using Flat search, then measure how IVF and HNSW degrade relative to perfection.

We record:

• p50
• p95
• p99
• throughput impact under concurrent load

One file, already configured:

• IVF → nprobe sweep across 1 → 100
• HNSW → efSearch sweep across 10 → 300

You get the Pareto Frontier: Recall vs Latency, the only curve that actually matters in Retrieval.

Fully self-contained, reproducible, and clean enough to drop into your own experiments or articles.

Because every team that builds RAG eventually hits this moment:

“The LLM is fine. The embeddings are fine. So why the hell are we getting incomplete answers?”

This repo is the answer: because your index configuration is silently sabotaging you.

If you don't measure:

• the true recall gap
• the latency ceiling
• the cluster imbalance
• the graph search depth trade-offs

…you're building blind.

This suite shows you the real behavior of your retrieval layer, not the marketing slides.

With this benchmark, you'll see things like:

• IVF with nprobe=1 can drop recall into the 40–60% range.
• HNSW becomes unreliable at low efSearch, even though it “looks fast”.
• Flat is unbeatable under 300k vectors but collapses past 1M.
• Increasing nlist without rebalancing clusters kills IVF accuracy.
• Latency curves behave differently under concurrent load vs single-shot queries.

None of these show up in blog posts. All of them show up in real production systems.

pip install -r requirements.txt

python bench_faiss_indices.py

Results are saved as JSON + console output.

You can plug outputs directly into your notebook or monitoring tools.

• Search engineers shipping RAG into production
• AI engineers who want numbers, not vibes
• CTOs & architects modeling system latency budgets
• Researchers studying ANN behavior under real constraints
• Anyone who's tired of guessing which index to use

If you care about correctness, performance, or scale, this repo saves you weeks.

Read the full article series about retrieval engineering, index tuning, and real-world RAG failures:

👉 https://nidly.substack.com

Here's the star of the show for this repo: “RAG Isn't About Embeddings — It's About the Index You Chose” A breakdown of why embeddings get too much credit and indexes get too little.

If this repo helps you:

⭐ Star the repo 📝 Subscribe to the Substack 🔗 Share the articles with your team

It keeps the research flowing and the benchmarks improving.

• GPU acceleration benchmarking
• PQ & OPQ compression benchmarks
• Milvus & Weaviate benchmark adapters
• Hybrid retrieval (BM25 + dense vectors)
• Cross-encoder re-ranking scoring suite

Stay tuned. This is just version zero.