A small, working Retrieval-Augmented Generation API. You give it documents, then ask questions, and it answers using only what is in those documents, with sources. Built to learn the moving parts of a real RAG system.
An LLM does not know your private documents. RAG fixes that: we split each document into chunks, turn every chunk into an embedding (a vector that captures meaning), and store them in a vector database. When you ask a question, we embed the question, find the chunks whose vectors are closest, and hand those chunks to the LLM as context, so it answers from your data instead of guessing.
ingest: text -> chunks -> embeddings -> vector store
ask: question -> embedding -> nearest chunks -> Claude -> answer + sources
Each layer has exactly one job. This is the "separation of concerns" a senior reviewer looks for.
| File | Job |
|---|---|
app/config.py |
all settings in one place |
app/models.py |
request/response shapes (Pydantic), input validation |
app/embeddings.py |
turns text into vectors (local model, no paid API) |
app/vector_store.py |
the vector DB (Chroma); swap to Pinecone here only |
app/llm.py |
the only file that talks to Claude |
app/rag.py |
the service: chunk, ingest, retrieve, answer |
app/main.py |
the HTTP routes: validate input and delegate, no logic |
- Python 3.10+
python -m venv venv && source venv/bin/activatepip install -r requirements.txtcp .env.example .envand put yourANTHROPIC_API_KEYin ituvicorn app.main:app --reload
The first run downloads a small embedding model (~80MB). Then open http://127.0.0.1:8000/docs for an interactive UI you can click through.
Ingest a document:
curl -X POST http://127.0.0.1:8000/ingest \
-H "Content-Type: application/json" \
-d '{
"doc_id": "policy1",
"title": "Refund Policy",
"text": "Customers may request a refund within 30 days of purchase. Digital goods are non-refundable once downloaded. Refunds are issued to the original payment method within 5 business days."
}'Ask a question:
curl -X POST http://127.0.0.1:8000/ask \
-H "Content-Type: application/json" \
-d '{ "question": "How long do customers have to request a refund?" }'You get back an answer grounded in the document, plus the source chunks it used. Try asking something NOT in the document (for example "what is your phone number?") and watch it correctly say it does not have that information.
- The question and the documents must use the same embedding model, or the vectors are not comparable.
- Chunk overlap keeps an idea from being cut in half between chunks.
- The system prompt in
llm.pyforces Claude to answer only from context and to admit when it does not know. That single rule is what keeps a RAG system from hallucinating. - top_k controls how many chunks feed the model: too few misses context, too many adds noise and cost.