When RAG Finds the Right Document but Gives the Wrong Answer: Chunk Size, Overlap, and Retrieval Quality

Abstract

A RAG system can have a strong embedding model, a fast vector index, and a reasonable prompt, then still miss the sentence that answers the user's question. The answer often vanishes before nearest-neighbor search starts, when the document is cut into chunks that are too small, too large, too noisy, or stripped away from the metadata that made the text retrievable.

RAG chunking strategy sits before vector search (HNSW and IVF-PQ vector search mechanics) and before answer-quality diagnosis (attribution traces and retrieval coverage metrics). It decides what the embedding model sees, what the index stores, what the retriever can rank, and what the generator receives as evidence. A 200-token chunk can make retrieval precise and brittle. A 1,200-token chunk can preserve context and dilute the matching signal. Overlap can rescue boundary cases and also inflate the index with near-duplicates.

1. Chunking Changes the Object Being Retrieved

Vector search ranks chunks. Documents only matter after the ingestion pipeline turns them into retrievable units. That sounds obvious until a retrieval trace returns the right file and the wrong evidence.

Take a support policy page with this structure:

For the query "Can an enterprise customer return engraved hardware after 30 days?", neither isolated section is enough. The standard return window, the engraved-hardware exception, and the enterprise-contract caveat all matter. If the retriever only returns the Returns section, the model may answer with the standard rule. If it only returns the Exceptions section, it may answer without the baseline rule.

Now change the chunking:

The larger chunk improves answerability for that question. It also adds more unrelated words to the embedding. A broad chunk can match many queries weakly, crowding out a smaller chunk that contains a more exact answer.

That is the central trade-off:

The goal is to make the retrievable unit match the evidence unit. A paragraph may be enough for a definition. A whole section may be needed for a policy. A table row may need its header. A code function may need imports, type definitions, and the comment that explains the invariant.

2. Chunk Size Is a Recall-Precision Dial

Chunk size controls the amount of text embedded as one vector. In most systems, it is the first dial people change because frameworks expose it directly.

Most chunking systems implement the same abstraction with different boundary rules. LlamaIndex treats chunking as token-budgeted node parsing: it keeps text under a target token size, carries overlap forward, and falls back through separators when natural boundaries do not fit. LangChain's recursive splitter starts from the largest textual unit and descends through smaller separators, preserving paragraphs and sentences when they fit before cutting more aggressively. OpenAI vector stores make chunking an ingestion policy on each file: the static strategy sets the maximum token budget and overlap before chunks are embedded and stored, with the current default at 800-token chunks and 400-token overlap.

Those defaults are starting points. A handbook, a codebase, financial filings, call transcripts, research PDFs, and product documentation all deserve separate measurement.

A useful first-order model:

where:

• $T$ is document length in tokens.
• $S$ is chunk size.
• $O$ is overlap.
• $S - O$ is stride.

For a 20,000-token manual, the storage and ranking surface changes quickly:

The 800-token chunk with 400-token overlap creates almost 50% more chunks than the same size with 200-token overlap. That affects storage, embedding spend, indexing time, and the number of near-duplicate candidates the retriever must rank.

The retrieval quality curve is usually non-monotonic:

Small chunks favor precision. Large chunks favor context. The right size depends on query distribution.

For "What is the refund window?", a short chunk with the exact sentence is enough. For "Can enterprise customers return engraved hardware?", a larger chunk or parent-section expansion is safer. For "Which API parameter controls vector store chunk overlap?", a precise chunk with code and parameter names will beat a broad page-level chunk.

3. Overlap Fixes Boundaries and Creates Duplicate Evidence

Overlap exists because real documents ignore artificial chunk boundaries. A definition starts at the end of one chunk. A caveat begins in the next. A table header sits above the row. A function call is separated from the type that explains it.

Overlap carries some trailing context forward:

The overlapped version makes each boundary chunk more useful. It also means the same sentence can appear in several vectors. When top-k retrieval returns three variants of the same local passage, the model receives less diverse evidence than the retrieval count suggests.

A practical overlap policy:

Measure overlap as a boundary recall tool. If increasing overlap raises hit rate while the top results become mostly duplicates, the retriever may need deduplication, parent-child retrieval, or max marginal relevance.

A simple duplicate-ratio smoke test compares unique parent ids with the returned result count:

Here, one of four results repeats an already represented parent, so the duplicate ratio is 0.25. That number is not a universal metric. If the duplicate ratio climbs as overlap increases, the recall gain may be coming from repeated evidence rather than broader coverage.

4. Semantic Boundaries Beat Blind Token Windows

A fixed 800-token window is easy to operate. It also cuts through document structure.

The better default is a hierarchy: document, section, subsection, paragraph, sentence, then token fallback. Split at the highest boundary that keeps the chunk under the size budget. Use token splitting only when the natural unit is too large.

For documentation, a page like "Vector stores > File upload > Chunking strategy" should keep that heading path attached to the local paragraph. The chunk can carry headings as metadata, prefix text, or both:

That heading context matters because many local paragraphs are written with assumed context. A paragraph that starts with "You can customize this..." may embed poorly when detached from the section that defines "this."

For code, syntax-aware chunking usually beats paragraph splitting:

For tables, the unit is often a row plus headers, or a small group of rows plus headers. A chunk containing "$0.10/GB/day" without the storage tier, product, and pricing context is too weak to support a grounded answer.

The same rule applies to PDFs. Page boundaries are implementation details. If the question is answered by a caption, a footnote, or a row that continues across pages, the chunker has to preserve that relationship explicitly.

5. Retrieval Quality Needs Chunk-Level Metrics

A chunking change should be evaluated before it reaches production. The basic question is: for the queries users actually ask, does the retriever return chunks that contain enough evidence to answer correctly?

Start with a small evaluation set:

Then compare chunking configurations:

Use retrieval metrics before answer metrics:

LlamaIndex's retrieval evaluator exposes standard ranking metrics such as hit rate, MRR, precision, recall, average precision, and NDCG. Ragas separates context recall into claim support over retrieved context, and also supports non-LLM or ID-based context recall when reference contexts are available.

The key comparison is the metric profile:

The answer model should be evaluated after the retriever passes this first gate. If the retriever misses the required source, answer grading mostly measures the model's willingness to guess.

6. Parent-Child Retrieval Separates Search From Context

One common fix is to search over small chunks and answer with larger parent chunks.

This gives the retriever a sharp vector while giving the generator enough context. It is especially useful for:

• API documentation where a parameter paragraph needs the endpoint section.
• Legal text where a clause needs neighboring exceptions.
• Tutorials where one step depends on the previous step.
• Tables where a row needs headers and notes.
• Code where a function needs class-level state or imports.

The trade-off is context expansion. If every hit expands to a large parent, top-k can become expensive quickly:

where $P$ is parent size. With $k=8$ and 1,200-token parents, retrieval can inject 9,600 tokens before the user question, instructions, citations, or conversation history.

A safer policy:

1. Retrieve 20 child chunks.
2. Deduplicate by parent id.
3. Rerank child chunks.
4. Expand the top four unique parents.
5. Trim parent text around the matched child when possible.

That policy keeps high recall during search, preserves enough context for generation, and avoids filling the prompt with repeated sibling chunks from the same section.

Parent-child retrieval also improves citations. The child chunk tells you the exact evidence span. The parent chunk gives the model surrounding context. The citation can point to the child span while the answer uses the parent text for disambiguation.

7. Chunking Policy Should Follow Query Distribution

Different products ask different questions of the same corpus.

A developer-docs chatbot sees exact parameter names, error messages, version constraints, and code snippets. A customer-support bot sees policy questions, account-specific constraints, and procedural steps. A research assistant sees broad synthesis queries and multi-hop evidence gathering.

That changes the chunking strategy:

The query distribution should be sampled from real traces when possible. When real traces are unavailable, create a synthetic set that includes the queries that usually break retrieval:

• exact IDs, SKUs, error codes, and parameter names
• "except when" policy questions
• comparisons across two sections
• table lookups
• questions that depend on headings
• stale or deleted facts
• multi-hop questions where evidence lives in separate chunks

The "Lost in the Middle" long-context result is relevant here. Longer context windows can still use retrieved passages unevenly. Liu et al. found that model performance can change when relevant information moves inside the input context, with weaker use of information in the middle for many evaluated models. Chunking and ranking therefore decide both what evidence enters the prompt and where it lands.

That makes retrieval ordering part of chunking policy. If a large parent expansion pushes the best evidence into the middle of a long context, a nominal recall win can become an answer-quality loss.

8. A Practical RAG Chunking Strategy Tuning Loop

Treat chunking as an experiment with a small grid and a fixed evaluation set.

Start with four candidates:

Then run the same queries through each candidate and record:

A useful decision rule:

Chunking should also be versioned. Store the chunking policy with each indexed document:

Without that metadata, retrieval regressions are hard to diagnose. A production incident may look like an embedding model problem when the real change was a parser update that removed headings, a chunk-size change that split tables, or a reindex that dropped parent ids.

9. Decision Rules That Hold Up in Practice

Use these defaults as a starting point, then validate against your corpus:

Keep chunks self-contained enough to answer a narrow question, but small enough that the embedding still describes a specific idea.

The chunk should usually include:

• document title
• section path
• stable source id
• page, line, or byte offsets where available
• timestamps for transcripts
• permissions and tenant metadata
• parent-child relationship
• chunking policy version

The chunk should usually avoid:

• repeated navigation text
• boilerplate footers
• unrelated neighboring sections
• tables without headers
• code fragments without symbols
• PDF extraction artifacts
• duplicated overlap returned repeatedly to the generator

Those rules are more durable than a single magic chunk size.

10. What to Benchmark Before You Trust a RAG Chunking Strategy

Before promoting a new RAG chunking strategy, run a benchmark that forces the chunker to show its weaknesses.

Use at least these slices:

Then inspect real retrieved chunks alongside aggregate scores. The most useful debugging view connects each query to the expected source ids, top retrieved chunks, chunk text, parent id, score, metadata filters, final context, answer, and citations.

That trace connects chunking to retrieval and retrieval to generation. It also separates the common cases:

Chunking is upstream of every RAG quality metric. The retriever can only rank the objects the ingestion pipeline created.

Conclusion

RAG chunking strategy is the document-design layer of vector search. Chunk size controls recall and precision. Overlap protects boundaries and adds duplicates. Semantic splitting keeps human structure intact. Parent-child retrieval separates search precision from generation context.

The practical path is simple: choose two or three chunking policies, run them against a fixed retrieval eval set, inspect the misses, and promote the policy that improves answerability without flooding the prompt with duplicated or diluted evidence.

References

1. Patrick Lewis et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." arXiv, 2020. https://arxiv.org/abs/2005.11401
2. Nelson F. Liu et al. "Lost in the Middle: How Language Models Use Long Contexts." arXiv, 2023. https://arxiv.org/abs/2307.03172
3. OpenAI. "Retrieval." OpenAI API documentation. https://platform.openai.com/docs/guides/retrieval
4. LlamaIndex. "Token text splitter." LlamaIndex documentation. https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/token_text_splitter/
5. LangChain. "Splitting recursively." LangChain documentation. https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter
6. LlamaIndex. "Retrieval Evaluation." LlamaIndex examples documentation. https://developers.llamaindex.ai/python/examples/evaluation/retrieval/retriever_eval/
7. Ragas. "Context Recall." Ragas documentation. https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/
8. Pinecone. "Chunking Strategies for LLM Applications." Pinecone Learning Center. https://www.pinecone.io/learn/chunking-strategies/