🎙️ LennyHub RAG

A production-ready RAG (Retrieval-Augmented Generation) system built on transcripts from Lenny's Podcast, featuring conversations with top product leaders and growth experts.

• 🚀 One-Command Setup: Automated installation and indexing with setup_rag.py
• 🎨 Visual Web Interface: Beautiful Streamlit app for querying and exploration
• 🕸️ Interactive Graph Viewer: Explore 544 people and their connections with clickable network visualization
• 🗄️ Qdrant Vector Database: Production-grade local vector storage (no Docker needed)
• 📊 Knowledge Graph RAG: Advanced retrieval with LightRAG entity and relationship extraction
• 🔍 Multiple Search Modes: Hybrid, local, global, and naive search strategies
• 📚 297 Podcast Transcripts: Comprehensive knowledge base from industry leaders
• 💡 Interactive Queries: Both CLI and web-based query interfaces
• ⚡ Fast & Efficient: Caching, parallel processing, and optimized embeddings

Featuring conversations with:

• Product Leaders: Julie Zhuo, Shreyas Doshi, Adam Fishman
• Growth Experts: Brian Balfour, Elena Verna, Kevin Kwok
• Founders: Patrick Collison, Amjad Masad, Andrew Wilkinson
• Executives: Ada Chen Rekhi, Claire Hughes Johnson, Gokul Rajaram
• And many more!

Topics covered: Product management, growth strategy, career development, startup advice, leadership, decision-making frameworks, and more.

git clone https://github.com/traversaal-ai/lennyhub-rag.git
cd lennyhub-rag

pip install -r requirements.txt

# Copy example environment file
cp .env.example .env

# Edit .env and add your OpenAI API key
# OPENAI_API_KEY=your-key-here

Sequential Mode (Reliable)

# Process first 10 transcripts (quick test - 5 min)
python setup_rag.py --quick

# Process first 50 transcripts (30-40 min)
python setup_rag.py --max 50

# Process all 297 transcripts (2-3 hours)
python setup_rag.py

Parallel Mode (5-10x Faster!) ⚡

# Process 50 transcripts in parallel (6-8 min)
python setup_rag.py --max 50 --parallel

# All 297 transcripts in parallel (25-35 min)
python setup_rag.py --parallel

# Custom workers (default: 5, max: 10)
python setup_rag.py --parallel --workers 8

What this does:

• ✅ Installs Qdrant locally (if needed)
• ✅ Starts Qdrant server
• ✅ Builds embeddings and knowledge graph (sequential or parallel)
• ✅ Automatically resumes from where you left off
• ✅ Tests the system automatically

Launch the beautiful web UI:

Windows:

# First time: Install Qdrant
.\install_qdrant_windows.ps1

# Run the app (Qdrant starts automatically!)
.venv\Scripts\Activate.ps1
streamlit run streamlit_app.py

macOS/Linux:

./run_streamlit.sh

Features:

• 🔍 Query Tab: Ask questions with AI-powered search
• 📊 Statistics Tab: View system health and metrics
• 📖 Transcripts Tab: Browse and preview all transcripts
• ⚙️ Sidebar: Real-time status, settings, and quick links
• 💡 Sample Questions: Pre-built queries to get started
• 🔄 Auto-Start Qdrant: No manual startup needed
• 🛡️ Stable Queries: Run unlimited queries without errors

Screenshot Features:

• Clean, modern interface
• Multiple query modes (hybrid, local, global, naive)
• Real-time Qdrant status monitoring
• Query timing and metadata
• Transcript filtering and preview

Explore the network of people and their connections from Lenny's podcasts with an interactive graph visualization.

Knowledge Graph

Launch the Graph Viewer:

python serve_graph.py

The graph will automatically open in your browser at http://localhost:8000/graph_viewer_simple.html

Features:

• 🎯 Interactive Network: 544 people with all names visible
• 🌟 Top Connectors Highlighted: See the 5 most connected people
  • Lenny Rachitsky (292 connections) - Gold
  • Bob Moesta (98 connections) - Amber
  • April Dunford (85 connections) - Orange
  • Arielle Jackson (82 connections) - Deep Orange
  • Andrew Wilkinson (78 connections) - Red
• 🔍 Smart Search: Find people with exact match prioritization
• 👆 Clickable Legend: Click any top connector to focus on them
• 🖱️ Interactive Controls:
  • Drag nodes to reposition
  • Scroll to zoom in/out
  • Click for detailed information
  • Double-click to focus on a person
• 🎨 Beautiful Design: Dark theme with Mona Sans font
• ⚡ Auto-Centers: Graph starts focused on Lenny Rachitsky

What You'll Discover:

• Who are the most connected people in Lenny's podcast network
• Relationships between guests, experts, and industry leaders
• Connection patterns across 544 people from the transcripts

python query_rag.py --interactive

python query_rag.py "What is a curiosity loop?"
python query_rag.py "What is the growth competency model?"

python query_with_sources.py "What are best practices for onboarding?"
python query_rag_with_chunks.py "How do you build a great product team?"

"What is the explore and exploit framework for career development?"
"How do you avoid being the boiled frog in your career?"
"What advice does Ada give about early career strategy?"
"How should you use values to make career decisions?"

"What are the four components of the growth competency model?"
"Why is onboarding important for growth?"
"How can onboarding improve retention?"
"What are opinionated defaults?"

"What is a curiosity loop and how does it work?"
"What is the PMF framework for choosing a company?"
"What is the inner vs outer scorecard concept?"
"What is the eating your vegetables concept?"

"Should you start a company with your partner?"
"How do you build trust with your team?"
"What makes a great product leader?"

• SETUP_GUIDE.md - Complete setup instructions
• STREAMLIT_QUICKSTART.md - Launch web UI in 3 steps
• VISUAL_APP_SUMMARY.md - Streamlit app features overview

• QDRANT_SETUP.md - Qdrant installation and configuration
• STREAMLIT_README.md - Full Streamlit app documentation
• OVERVIEW.md - Technical architecture deep dive
• ADDING_TRANSCRIPTS.md - Guide for adding more transcripts

• sample_questions.txt - 70+ curated questions
• MULTI_TURN_QUESTIONS.md - 85+ complex queries

lennyhub-rag/
├── 📊 Data & Storage
│   ├── data/                         # 297 podcast transcripts
│   ├── rag_storage/                  # Knowledge graph & metadata
│   └── qdrant_storage/               # Vector embeddings (local DB)
│
├── 🚀 Setup & Configuration
│   ├── setup_rag.py                  # One-command automated setup
│   ├── install_qdrant_local.sh       # Install Qdrant (macOS/Linux)
│   ├── install_qdrant_windows.ps1    # Install Qdrant (Windows)
│   ├── start_qdrant.sh               # Start Qdrant server
│   ├── stop_qdrant.sh                # Stop Qdrant server
│   ├── status_qdrant.sh              # Check Qdrant status
│   ├── qdrant_config.yaml            # Qdrant configuration
│   ├── qdrant_config.py              # Python Qdrant config
│   ├── requirements.txt              # Python dependencies
│   └── .env                          # API keys & settings
│
├── 🎨 User Interfaces
│   ├── streamlit_app.py              # Visual web interface
│   ├── query_worker.py               # Subprocess query handler
│   ├── run_streamlit.sh              # Launch Streamlit app
│   ├── query_rag.py                  # CLI query interface
│   ├── query_with_sources.py         # Query with source attribution
│   └── query_rag_with_chunks.py      # Query with chunk details
│
├── 🔧 Building & Processing
│   ├── build_transcript_rag.py       # Build RAG (all transcripts)
│   ├── build_rag_quick.py            # Quick build (10 transcripts)
│   └── build_transcript_rag_parallel.py  # Parallel processing
│
└── 📚 Documentation
    ├── README.md                     # This file
    ├── SETUP_GUIDE.md                # Setup instructions
    ├── STREAMLIT_QUICKSTART.md       # Streamlit quick start
    ├── STREAMLIT_README.md           # Streamlit full docs
    ├── VISUAL_APP_SUMMARY.md         # Streamlit features
    ├── QDRANT_SETUP.md               # Qdrant documentation
    ├── OVERVIEW.md                   # Technical architecture
    └── ADDING_TRANSCRIPTS.md         # Adding transcripts

• Research frameworks and best practices
• Learn from top PMs at Airbnb, Stripe, Meta
• Study product strategy and execution
• Interview preparation

• Understand growth competency models
• Learn retention and onboarding strategies
• Study successful growth strategies
• Framework deep-dives

• Career strategy frameworks
• Decision-making guidance
• Leadership insights
• Personal values exploration

• Startup advice from successful founders
• Leadership frameworks
• Team building strategies
• Strategic decision-making

User Query
    ↓
Streamlit UI / CLI
    ↓
RAG System (RAG-Anything)
    ↓
LightRAG (Knowledge Graph)
    ↓
├─→ Entity Extraction (GPT-4o-mini)
├─→ Relationship Mapping
├─→ Embeddings (text-embedding-3-small)
└─→ Qdrant Vector Storage
    ↓
Hybrid Search (local + global + vector)
    ↓
Answer Synthesis (GPT-4o-mini)
    ↓
Results with Sources

The system supports parallel transcript processing for significantly faster indexing:

Sequential Processing:

• Processes one transcript at a time
• Safer, more predictable
• ~2-3 minutes per transcript

Parallel Processing (⚡ 5-10x faster):

• Processes 5-10 transcripts simultaneously
• Uses asyncio semaphore for concurrency control
• Rate-limit safe (max 10 workers)
• Smart resume: skips already-processed transcripts
• ~20-30 seconds per transcript (with 5 workers)

# Enable parallel mode
python setup_rag.py --max 50 --parallel

# Custom concurrency
python setup_rag.py --parallel --workers 8

Hybrid (Recommended)

• Combines local + global + vector search
• Best overall results
• Balanced speed and accuracy

Local

• Entity-focused search
• Fast and precise
• Great for specific concepts

Global

• Relationship-focused
• Broader context
• Best for understanding connections

Naive

• Pure vector similarity
• Fastest mode
• Simple semantic search

This RAG system demonstrates advanced agent engineering with knowledge graphs, vector databases, and LLM orchestration. Want to learn how to build systems like this from scratch?

Join the Agent Engineering Bootcamp: Developers Edition

• Learn to build production-ready AI agents
• Master RAG, LLMs, and agentic workflows
• Taught by Hamza Farooq (Ex-Google, Prof UCLA & UMN)
• 4.8★ rating from 95+ students

Get $200 OFF with code 200OFF →

• RAG Framework: RAG-Anything v1.2.9+
• Knowledge Graph: LightRAG v1.4.9+
• Vector Database: Qdrant v1.16+
• LLM: OpenAI GPT-4o-mini
• Embeddings: OpenAI text-embedding-3-small (1536 dims)
• Web UI: Streamlit 1.28+
• Language: Python 3.8+

*Small batches don't benefit much from parallelization

• Per Query: $0.001-0.01
• Cached Queries: Free (stored responses)
• Typical Session: ~$0.05-0.10

• LLM response caching (saves ~80% on repeated queries)
• Efficient chunking and embedding strategies
• Smart query routing

• OS: Windows, macOS, or Linux
• Python: 3.8 or higher (3.11 recommended)
• RAM: 2GB+ recommended (4GB+ for all transcripts)
• Disk Space:
  • Base: ~500MB
  • 10 transcripts: ~1GB
  • 50 transcripts: ~2GB
  • 297 transcripts: ~5GB
• Internet: Required for OpenAI API calls

Edit .env file:

# Vector Database
USE_QDRANT=true
QDRANT_URL=http://localhost:6333
QDRANT_COLLECTION_NAME=lennyhub

# Working Directory
WORKING_DIR=./rag_storage

# Start Qdrant
./start_qdrant.sh

# Check status
./status_qdrant.sh

# View logs
tail -f qdrant.log

# Stop Qdrant
./stop_qdrant.sh

# Dashboard
open http://localhost:6333/dashboard

Sequential Processing:

# Build all transcripts
python build_transcript_rag.py

# Quick build (10 transcripts)
python build_rag_quick.py

Parallel Processing (5-10x faster):

# Using setup_rag.py (recommended)
python setup_rag.py --max 50 --parallel --workers 5

# Using standalone parallel script
python build_transcript_rag_parallel.py

Performance Comparison:

• Sequential: ~2-3 min/transcript
• Parallel (5 workers): ~20-30 sec/transcript
• Parallel (10 workers): ~15-20 sec/transcript

from raganything import RAGAnything, RAGAnythingConfig
from qdrant_config import get_lightrag_kwargs
import asyncio

# Initialize
config = RAGAnythingConfig(working_dir="./rag_storage")
lightrag_kwargs = get_lightrag_kwargs()
rag = RAGAnything(config=config, lightrag_kwargs=lightrag_kwargs)

# Query
response = await rag.aquery("Your question here", mode="hybrid")
print(response)

# Check if running
curl http://localhost:6333/

# Restart Qdrant
./stop_qdrant.sh && ./start_qdrant.sh

# View logs
tail -f qdrant.log

# Reinstall dependencies
pip install -r requirements.txt --upgrade

# Check API key
echo $OPENAI_API_KEY

# Clear cache and rebuild
rm -rf rag_storage/ qdrant_storage/
python setup_rag.py --quick

# Clear cache
streamlit cache clear

# Run on different port
streamlit run streamlit_app.py --server.port 8502

# Check logs
streamlit run streamlit_app.py --logger.level debug

# Install Qdrant if not found
.\install_qdrant_windows.ps1

# Check if Qdrant is running
Invoke-RestMethod -Uri "http://localhost:6333/"

# Activate virtual environment
.venv\Scripts\Activate.ps1

# Install missing packages
uv pip install nest-asyncio

1. Use Parallel Mode: 5-10x faster with --parallel flag
2. Adjust Workers: More workers = faster (up to 10 for rate limits)
3. Smart Resume: System automatically skips processed transcripts
4. Start Small: Test with --quick before full indexing
5. Monitor Resources: Check RAM usage with large datasets

1. Use Hybrid Mode: Best balance of speed and accuracy
2. Enable Caching: Responses are cached automatically (saves ~80%)
3. Batch Queries: Process multiple questions in one session
4. Choose Right Mode: Naive is fastest, hybrid is most accurate
5. Reuse Sessions: Keep Streamlit app running for instant queries

Contributions welcome! Areas for improvement:

• Add more transcripts
• Improve query templates
• Enhance UI features
• Add new search modes
• Optimize performance
• Expand documentation

See LICENSE file for details.

• Transcripts: Lenny's Podcast
• RAG Framework: RAG-Anything by HKUDS
• Knowledge Graph: LightRAG by HKUDS
• Vector Database: Qdrant
• LLM & Embeddings: OpenAI

Ada Chen Rekhi, Adam Fishman, Adam Grenier, Andrew Wilkinson, Annie Duke, Brian Balfour, Casey Winters, Claire Hughes Johnson, Elena Verna, Gokul Rajaram, Jeff Weinstein, Julie Zhuo, Kevin Kwok, Lenny Rachitsky, Maggie Crowley, Marily Nika, Patrick Collison, Shreyas Doshi, and 279 more amazing guests!

• Check SETUP_GUIDE.md for detailed instructions
• See TROUBLESHOOTING section
• Review sample_questions.txt for query examples

• 🪟 Windows Support: Full Windows compatibility with PowerShell scripts
• 🔄 Auto-Start Qdrant: Streamlit app automatically starts Qdrant
• 🛡️ Stable Multi-Query: Subprocess isolation prevents event loop conflicts
• ⚡ Parallel Processing: 5-10x faster indexing with --parallel flag
• ✨ One-Command Setup: Automated setup_rag.py script
• 🎨 Streamlit Web UI: Beautiful visual interface
• 🗄️ Local Qdrant: Production vector DB (no Docker)
• 📚 297 Transcripts: Complete podcast library
• 🚀 Smart Resume: Automatically skips processed transcripts
• 📊 Statistics Dashboard: Real-time system monitoring
• 💡 Sample Questions: Built-in query examples

Ready to explore?

git clone https://github.com/traversaal-ai/lennyhub-rag.git
cd lennyhub-rag
python setup_rag.py --quick

Enjoyed building with this RAG system? Take your skills to the next level!

Agent Engineering Bootcamp: Developers Edition teaches you to build production-ready AI agents like this one (and more advanced systems). Learn RAG pipelines, knowledge graphs, LLM orchestration, and agentic workflows from industry experts.

Taught by Hamza Farooq

• Founder & Ex-Google Engineer
• Professor at Stanford Continuing Studies, UCLA & UMN
• 4.8★ rating (95+ students)

Enroll now with $200 OFF →

Built with ❤️ using RAG-Anything, LightRAG, Qdrant, and Streamlit