Review a chunking choice before the first index build
RolesLLM Product EngineeringAI Data SystemsCustomer AI Deployment
The question
A team is about to index a new document corpus and asks you to review the chunker. They picked semantic splitting, 1000 tokens per chunk and 20 percent overlap, because a blog post recommended it. What do you tell them?
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
Start from the query type, not from a post. Factoid questions want 256 to 512 tokens, analytical or multi-hop questions 512 to 1024, and whole-section questions 1024 to 2048. Begin with recursive splitting at 512 tokens and no overlap, then measure recall at 5 on a 50-query set before you move any knob.
Show full answer
Why size follows the question
A chunk must be large enough to hold the answer plus its local context, and small enough that the top k returns the answer and not noise. Too small, at 64 to 128 tokens, and a chunk says "it increased 15% last quarter" with no subject. Too large, at 2048 tokens and more, and one chunk is 10% about revenue and 90% about headcount, so its embedding points nowhere.
| Query type | Chunk size |
|---|---|
| Factoid | 256 to 512 tokens |
| Analytical or multi-hop | 512 to 1024 tokens |
| Whole-section comprehension | 1024 to 2048 tokens |
What the measurements say
- Recursive splitting at 512 tokens beat semantic splitting 69% to 54% in the chunking study.
- Overlap gave zero measurable benefit in the sparse retrieval study, and every overlapped token is embedded and stored again, so the index cost grows with the overlap.
- Response quality drops sharply around 2,500 tokens of context, so bigger chunks hit a cliff.
- Semantic splitting without a minimum size produces 40-token fragments that hurt retrieval.
Pick a default without a benchmark
Three properties decide the first chunker: the document type, the paragraph length, and whether the format carries structure. Prose with no structure gets recursive splitting. Markdown and API docs get the header splitter. Code gets a syntax-aware splitter. When in doubt, recursive splitting is the strongest single baseline.
Then measure
Build a 50-query eval set, stratified by query type, and read recall at 5 for each candidate. Clamp every chunker with a minimum and a maximum size. Never let a chunk span two documents.
Follow-ups
L2WorkingWhy does the chunk size depend on the kind of question?
Show expected answer
the chunk has to contain the answer with its context, and still be small enough that the top k is about the answer and not about the surrounding noise.
L2WorkingThe corpus is markdown API documentation. What do you try first?
Show expected answer
the structural header splitter, because the format carries explicit structure; drop or merge header-only chunks.
L3SeniorWhich two guards does every chunker need, whatever the strategy?
Show expected answer
a minimum and a maximum size, because chunks of 5 tokens or 5000 tokens both break retrieval.
L3SeniorYou switch the embedding model later. What do you run again?
Show expected answer
the chunking eval, because the winner depends on the corpus and the model, and semantic splitting moves its boundaries when the model changes.
Wrong answers
Show the wrong answers that interviewers hear
“Semantic splitting is best because it follows meaning.”
It lost to recursive splitting at 512 tokens in the measured study, and it makes tiny fragments unless you set a floor.
“Overlap is a safety margin, keep it.”
The studies found that overlap often adds nothing, and it adds index cost in proportion to its size; justify any overlap with an ablation.
“Bigger chunks give the model more context.”
Quality drops around 2,500 tokens of context, and a chunk that covers many topics dilutes its own embedding.
Taught in
The lesson chunking strategies gives the size table and the measurements, the lesson RAG explains the too-small and too-large failures, and the lesson chunking comparison picks a default from three properties.