Are Bad Chunks Hurting Your RAG Answers? Evaluate Chunk Quality in 5 Steps

Article

Why Good Retrieval Still Gives Bad Answers

You upgraded your embeddings, raised top-k to 8 or 10, and rewrote your prompt, yet your RAG app still returns vague, incomplete, or slightly off answers. Retrieval scores look acceptable, and the language model is capable, so it is hard to know what to fix next.

In many of these cases, the weak link is the chunk itself. If a key definition is split across two chunks, buried in a 2,000-character block with three unrelated topics, or stripped of its heading, date, or section context, even accurate retrieval cannot rebuild the full meaning for the model.

Evaluating chunk quality before you retune embeddings or rewrite prompts saves you a lot of guessing. A short, structured review will tell you whether your chunks are self-contained and retrievable, or whether they are quietly capping your answer quality.

What Good Chunks Actually Look Like

A high-quality chunk for RAG does one job well: it gives the retriever and the model everything needed to use a single idea without needing to hunt elsewhere. That usually means one clear topic per chunk, intact sentences and tables, and just enough surrounding context to stay interpretable on its own.

Practically, this comes down to four traits. Completeness means no cut-off sentences, lists, or procedures. Self-containment means headings, entity names, and conditions travel with the chunk. Focus means the chunk is not mixing pricing, troubleshooting, and legal disclaimers together. Retrievability means the wording contains the terms and questions users are likely to search for.

If you evaluate chunk quality for RAG against those four traits, you can diagnose most problems quickly. A chunk that fails completeness needs better boundaries. A chunk that fails self-containment needs metadata or context enrichment. A chunk that fails focus or retrievability usually needs smaller or more semantically coherent splits.

A 5-Step Process to Evaluate Chunk Quality

You do not need a full evaluation harness to get a reliable signal. A focused sample of 20 to 30 real user questions is enough to reveal whether chunking is your bottleneck. Work from questions to chunks, not the other way around, so you judge quality by usefulness for answering.

Run this process on a copy of your indexed content or your chunk preview, and take notes per query. You are looking for patterns across failures, not scoring a single perfect chunk. Three similar failures are more valuable than ten perfect scores.

Once you see a pattern, fix one variable at a time and re-test the same set. Changing chunk size, boundaries, and metadata all at once makes it impossible to know what helped.

Step 1: Collect 20 to 30 real failed or borderline questions, plus the correct answer you expected for each.

Step 2: For each question, inspect the top 3 to 5 retrieved chunks. Ask: does any single chunk or combination contain the full answer?

Step 3: Label each failure as split context, missing context, bloated chunk, duplicate or near-duplicate, or no relevant chunk retrieved.

Step 4: Open the source document and compare. Check if the answer was split by your splitter, stripped of a heading, or buried in an unrelated section.

Step 5: Tally the labels. Your most frequent label is your priority fix for the next iteration.

Run a 20-Minute Manual Chunk Review

Automated metrics like faithfulness and answer relevance are useful, but a quick human review catches chunking issues that scores hide. Set a timer, pull 15 random chunks plus the chunks for your 5 worst recent answers, and read them as if you had no access to the source file.

For each chunk, ask whether a new teammate could understand and act on it alone. If you need to open the original PDF or page to make sense of it, the chunk is not self-contained enough for reliable generation.

Use the checklist below to keep the review consistent. If more than a third of your sample fails the same check, treat that as a systemic chunking problem rather than an edge case.

Can I understand what this chunk is about without seeing the previous chunk?

Are headings, product or section names, dates, and units preserved in the text or metadata?

Does the chunk cover one idea or task instead of three unrelated topics?

Are sentences, steps, table rows, and list items intact, not cut mid-thought?

Would the key search terms a user would type actually appear in this chunk?

If this chunk was retrieved alone, could the model answer correctly without guessing?

5 Common Chunk Failures and How to Fix Them

Most teams see the same handful of failure modes when they evaluate RAG chunk quality. Naming the pattern makes the fix obvious and prevents you from increasing top-k to compensate for chunks that will never work well together.

Start with the least invasive fix that resolves the pattern. Adding context and improving boundaries is usually cheaper and safer than doubling chunk size or reindexing everything with a new embedding model.

Pay special attention to split procedures, tables, and definitions, because those cause the most visible answer errors. If steps 4 and 5 of a process land in different chunks, the model will often summarize only what it saw and present it as complete.

Split context: answer is divided across two chunks. Fix with structure-aware splitting, larger windows for procedures, or parent-child retrieval where small chunks retrieve a larger parent.

Missing context: chunk is readable but ambiguous, like it says it instead of the product name. Fix by prepending document title and heading path and storing source section in metadata.

Bloated chunk: 1,500 plus characters covering multiple topics dilutes retrieval. Fix by tightening max size and splitting on headings or semantic shifts so each chunk holds one idea.

Orphaned table or list: rows retrieved without headers or preceding paragraph. Fix by keeping tables with captions and intro text, or converting key rows into self-describing sentences.

Duplicates and near-duplicates: headers, footers, or repeated disclaimers crowd out useful chunks. Fix by deduplicating boilerplate and filtering very low-signal chunks before indexing.

Should You Re-Chunk or Fix Retrieval?

Not every retrieval miss means you need to re-chunk. If the right chunk exists and reads well on its own but was ranked sixth instead of second, try improving ranking first with better metadata filtering, hybrid search, or query rewriting before you rebuild your index.

Re-chunk when your review shows the answer does not exist in any clean, usable form. Split sentences, context-free pronouns, mixed topics, and broken tables are chunking problems. No retriever can consistently fix content that was damaged at split time.

A practical rule is to re-chunk if more than 30 to 40 percent of your sampled failures are split context, missing context, or bloated chunks. If most failures are ranking issues with good chunks available, focus on retrieval and reranking, then re-evaluate chunk quality on the remaining failures.

Get Better Answers Without Starting Over

Chunk quality is not a one-time setting. It is a short feedback loop between real questions, retrieved chunks, and source structure. Review a small sample regularly, fix the dominant pattern, and keep the same test set so you can see whether answers actually improve.

If you take one habit from this guide, make it question-led evaluation. Ten real user questions will teach you more about your chunks than any abstract debate about ideal chunk size.

As a next step, pick your five most important recent failures and run the 5-step review on ChunkCraft resources. If split or context-free chunks keep appearing, adjust boundaries and context enrichment first, then retest before scaling the change to your full corpus.