RAG Chunking: Why Chunk Size Is the Wrong Knob

You've swept chunk size from 256 to 1024 tokens. You tried 20% overlap, then 50%. You swapped in a semantic splitter. Your eval score moved a point and a half, and the answers that were wrong are mostly still wrong.

Here's a pattern worth checking before you touch another setting: pull up the chunk your system cited for one of those wrong answers and read it on its own. Often the chunk is correct. It just doesn't say what it's about.

That's the thing chunk-size tuning can't fix. Every chunk in a RAG pipeline is embedded alone, retrieved alone, and handed to the model alone. If a chunk only makes sense next to its neighbours, your system never sees it make sense.

The setup: what each component actually sees

A quick map, so the failure is precise (full walkthrough in How RAG works):

• Ingestion splits documents into chunks and embeds each chunk. The embedding is computed from that chunk's tokens only.

• Retrieval compares the question's embedding (and, in hybrid setups, its keywords) against each chunk independently.

• Generation gets the top-k chunk texts, usually without the headings or pages they came from, often from several documents at once.

No step ever sees the page. So the unit that has to carry meaning is the chunk, and nothing else.

In this hypothetical handbook, the question "Can a contractor carry over unused leave days?" matches chunk 14 strongly (carry over, unused, leave days). The exception lives in chunk 15, which mentions contractors but never says leave or carry-over. The model answers "Yes, up to 5 days," with a real citation.

One orphaned chunk, two failure points

Anthropic's contextual retrieval write-up (Sept 2024) uses a chunk from an SEC filing: "The company's revenue grew by 3% over the previous quarter." No company, no quarter.

1. Retrieval failure. A question naming the company can't keyword-match it, and semantically it's indistinguishable from every other company's revenue sentence in the index.

2. Generation failure. If it is retrieved next to similar chunks from other filings, the model has to guess which "the company" is which. Whatever it picks gets cited, so the answer looks grounded. (More on why models fill gaps this confidently: Why AI confidently lies.)

Keep both in mind. Some fixes only address the first.

The four kinds of orphaned chunk

The split rule deserves a note: writers put the rule first and the exception after, which places exceptions right where boundaries tend to fall. It's the same sentence class AI summaries drop, covered in The Exception Pass.

What the chunking research actually says

• Chroma's technical report (Evaluating Chunking Strategies for Retrieval, July 2024) found recall in a narrow band of roughly 85–90 across the sizes tested, while smaller chunks markedly improved precision. The common 800-token / 400-overlap default scored lowest on the efficiency metrics. Small dataset, so directional.

• A Vectara-led study (Is Semantic Chunking Worth the Computational Cost?, Oct 2024) found semantic chunking's gains inconsistent; on realistic documents, fixed-size chunking often matched or beat it.

Read together: size and splitter change how much text you fetch and where cuts land. They don't put the missing subject back into a chunk. Overlap rescues a sentence cut in half but not a title four pages up. A higher top-k adds more context-free chunks and pushes useful text into the middle of the prompt, where recall is weakest (context windows explained).

The Orphan Test: measure before tuning

Twenty chunks, read cold. Prefer chunks your system actually retrieved for recent real questions.

# `store` is your vector store client; adapt the call to your library.
chunks = store.sample(n=20)  # or: chunks retrieved for your last 20 real queries

for i, c in enumerate(chunks, 1):
    print(f"--- {i} ---")
    print(c.text)          # the stored text, NOT the source document
    print()

# For each chunk, answer three yes/no questions:
#   1. Address: can I tell the document, product and version?
#   2. References: does every "it / this / the above" resolve inside the chunk?
#   3. Completeness: does any rule, number or table row carry its condition,
#      subject and headers?
# Any "no" = orphan. Count them.

There's no published benchmark for an acceptable orphan rate. What matters is whether you're at 2/20 or 12/20, because those are different projects.

Five fixes, cheapest first

1. Split on structure

Break on headings, list items and table boundaries. If a table has to be split, repeat its header row in every piece.

def split_table(header_row, rows, max_rows=20):
    for i in range(0, len(rows), max_rows):
        yield "\n".join([header_row, *rows[i:i + max_rows]])

2. Stamp the address

Deterministic, cheap, and it fixes most nameless numbers.

def with_address(chunk, doc):
    path = " > ".join(chunk.heading_path)
    stamp = f"{doc.title} > {path} (version {doc.version}, updated {doc.updated})"
    return f"{stamp}\n\n{chunk.text}"

# Embed AND keyword-index the stamped text; keep plain text for citations.
vector = embed(with_address(chunk, doc))
bm25_index.add(chunk.id, with_address(chunk, doc))

3. Add one written context line (contextual retrieval)

For dangling references, a heading path isn't enough. Have a model read the full document plus the chunk and write one or two sentences situating it. Prepend that before embedding and keyword indexing.

CONTEXT_PROMPT = """Here is a document:
{document}

Here is one chunk from it:
{chunk}

In one or two sentences, say where this chunk sits in the document and what any
pronouns or references in it point to. Output only those sentences."""

On Anthropic's benchmarks, top-20 retrieval failure fell from 5.7% to 3.7% with contextual embeddings, to 2.9% adding contextual BM25 (-49%), and to 1.9% with reranking (-67%), at an estimated one-time ~$1.02 per million document tokens with prompt caching. Measure on your own data. Trade-offs: the line is model-written (spot-check it) and must be regenerated when documents change.

Late chunking (Günther et al., Jina AI) is an embedding-layer alternative: embed the whole document with a long-context model, then pool token embeddings per chunk, so "its 3.85 million inhabitants" lands closer to "Berlin." It addresses the retrieval failure only. The model still reads the bare chunk.

4. Retrieve small, read big

def expand(hit, store, window=1):
    ids = range(hit.position - window, hit.position + window + 1)
    return [store.get(hit.doc_id, p) for p in ids if store.exists(hit.doc_id, p)]

Search precise chunks; send the model the neighbours or the parent section. Cap expansion, or prompt cost climbs.

5. Test the boundaries

Add eval questions whose answer needs the heading, the next paragraph, or a table header. Score retrieval separately from the answer:

def retrieval_hit(question, needed_chunk_ids, k=5):
    got = {c.id for c in search(question, k=k)}
    return needed_chunk_ids <= got   # all needed chunks must come back

If retrieval misses, prompt changes won't recover it.

When to skip chunking entirely

Anthropic notes that knowledge bases under ~200,000 tokens (about 500 pages) can go straight into the prompt, with prompt caching keeping it practical. Keep the prefix stable or the cache breaks; see prompt caching for AI agents. Long-context recall is uneven, so test it.

Key Takeaways

• Wrong answers with real citations usually mean the chunk lost its context, not that the model failed.

• Orphaned chunks fail twice: at retrieval and at generation.

• Chunk size, overlap and splitter type change where cuts land, not what a chunk knows.

• Run the Orphan Test (20 chunks, three questions) before tuning anything.

• Stamp every chunk with its address first; add written context and neighbour expansion where needed.

• Add boundary questions to evals and score retrieval separately.

FAQ

What is the best chunk size for RAG?

There isn't a universal one. Chroma's 2024 evaluation found recall fairly stable across sizes, with smaller chunks improving precision. Pick a size that respects your document structure, then fix context loss, which size alone can't address.

Does chunk overlap fix lost context in RAG?

Only partly. Overlap repeats neighbouring text, so a sentence cut in half survives. It doesn't bring back a document title, a section heading, or a table header that sits further away.

Is semantic chunking better than fixed-size chunking?

Not consistently. A 2024 study led by Vectara found semantic chunking's gains depended on the data, and fixed-size chunking often matched it on realistic documents at lower cost.

What is contextual retrieval?

A technique Anthropic published in 2024: a model writes a short context line for each chunk, describing where it sits in its document, and that line is prepended before embedding and keyword indexing. It reduced retrieval failures in Anthropic's benchmarks.

How do I know if my RAG chunks are losing context?

Read 20 stored or retrieved chunks on their own. If you can't tell what document they're from, what "this" refers to, or what a number or table row describes, those chunks are orphaned.

What is the difference between late chunking and contextual retrieval?

Late chunking adds document context to the chunk's embedding, which helps search. Contextual retrieval adds context to the chunk's text, which helps both search and the model reading the chunk.

Related reading

• How RAG works: the full pipeline this piece builds on

• Why AI forgets: context windows explained: why more chunks isn't the fix

• The Exception Pass: catching what AI summaries delete: the same sentences, lost a different way

• Start Here: a plain-English map to understanding AI

Get one of these in your inbox each week: subscribe to Simply Explained.

The bottom line

Your retriever, your embedding model and your LLM can all be working correctly while the answer is wrong, because the one sentence that mattered was stored without its subject.

Before you change the size of your chunks, read one. Alone. Then ask what it forgot.

Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype.