A hands-on reference notebook comparing 5 text-chunking strategies used in RAG pipelines, implemented from scratch and benchmarked on a synthetic Persian contract sample.
Chunking is the step before embedding/retrieval in any RAG pipeline: splitting a large document into smaller pieces that can be individually embedded and searched. The right chunking strategy depends heavily on document type, and picking the wrong one silently hurts retrieval quality. This repo implements and compares the main strategies side by side, so the trade-offs are visible instead of assumed.
| # | Method | Core idea | Best for |
|---|---|---|---|
| 1 | Fixed-size | Cut every N characters, with overlap | Quick prototyping only |
| 2 | Recursive character | Try paragraph -> sentence -> word boundaries first | Generic documents (safe default) |
| 3 | Token-based | Cut by real token count (tiktoken / HF tokenizer) | Strict context-window / cost budgets |
| 4 | Structure-aware (clause-based) | Cut on the document's own structural markers (e.g. numbered contract clauses) | Documents with known, consistent structure |
| 5 | Semantic | Embed sentences, cut where cosine similarity between consecutive sentences drops sharply | Documents with messy/unknown structure |
Each method is implemented manually (not just called as a black box), timed, and compared on chunk count, average/min/max chunk size, and execution time.
sample_contract.txt- synthetic Persian service contract (7 numbered clauses) used as the test documentchunking_comparison.ipynb- the full comparison notebook, one method per section, with a final side-by-side results tablerequirements.txt- dependencies
python -m venv venv
venv\Scripts\activate # Windows
pip install -r requirements.txt
jupyter notebook chunking_comparison.ipynbNote on network access (relevant from Iran): tiktoken (Method 3) and
sentence-transformers (Method 5) both download files on first use -- tiktoken
from Azure blob storage, sentence-transformers from Hugging Face. Both need a
VPN/proxy active the first time they run. After that, results are cached
locally. If unavailable, both methods fall back automatically to a lighter
offline approximation (whitespace tokenizer / TF-IDF vectors), so the notebook
always runs end to end.
| Method | Chunks | Time (ms) |
|---|---|---|
| fixed_size | 6 | 0.16 |
| recursive | 9 | 0.61 |
| token_based | 19 | 4.11 |
| structure_aware | 8 | 0.12 |
| semantic | 9 | 7.60 |
structure_awareandfixed_sizeare fastest (simple string/regex ops).semanticis slowest, since it requires an embedding pass before chunking can start.token_basedproduced far more chunks here because its size parameter (60 tokens) and the other methods' size parameter (300 characters) aren't directly comparable -- Persian text uses more tokens per character than English, so this isn't an apples-to-apples comparison out of the box. A fair comparison requires normalizing all methods to the same underlying unit (e.g. target token count).
- Generic document, no time to think about it -> recursive character splitting.
- Need a hard guarantee on chunk size relative to a model's context window -> token-based.
- Document has known, consistent structure (contracts, markdown, code) -> structure-aware.
- Document structure is messy or unknown, topic boundaries matter more than fixed size -> semantic.
- Fixed-size -> prototyping only; avoid in anything user-facing.