Chunking Strategy for RAG: The Passage That Cannot Stand Alone

Pick 512 tokens with ten percent overlap and move on. That is the answer you will be given, and it is the wrong shape of answer, because the setting it names is not what decides whether your system replies correctly.
Here is the strategy, in one paragraph. Split so that no passage, read on its own with nothing around it, asserts a rule it does not finish. Respect the boundaries the author already wrote (headings, clauses, numbered items) rather than counting tokens to a boundary of your own. Keep each passage's path back to its source page attached to it. Then accept that you cannot settle this before launch: the peer reviewed paper on how these systems fail concludes that "validation of a RAG system is only feasible during operation", so commission a second pass after real questions arrive and build the pipeline so that re-indexing is cheap.
The reason the size question misleads you sits in a 2025 preprint marked for publication at SIGIR. Semantic independence between passages is what performance depends on. Concept unity inside a passage, the property most teams optimise for, shows minimal impact. A bigger chunk is a concept-unity fix. It moves the boundary. It does not make the passages either side of that boundary able to stand alone.
So the budget question is not what size to buy. It is whether you are buying a pipeline you can change after you learn something.
The returns policy that answers in half
Here is the shape of the failure, as a constructed example rather than a client document. A passage is retrieved that reads as a complete rule, while the exception that qualifies it sits on the other side of a chunk boundary. The answer comes back confident, sourced and wrong, and the document held the right answer all along.
A twelve-page returns policy. The return window is stated on page 7: thirty days. The exception that extends it for faulty goods, sixty days, is the first line of page 8. A fixed-size splitter counts tokens, reaches its limit near the foot of page 7, and puts a boundary between the rule and the exception.
A customer writes in. Their blender failed after six weeks. The system retrieves the page 7 passage, which reads as a complete and confident statement of policy, and replies that the window has closed.
The document contains the correct answer. The system had the wrong half of it. Nothing in that passage signals that it is incomplete, which is exactly why the failure survives review: a human reading the retrieved text alongside the answer will agree with the answer.
The commercial version of this is not a benchmark score. It is a refund wrongly refused, a quoted price that omits the surcharge, a leave entitlement stated without the probation clause, and a compliance answer that is true on its own page and wrong in the document. Any product built on a document question answering workflow inherits this risk the moment the source material contains exceptions, and nearly all of it does, because exceptions are what policies are for.
Why is chunking important in RAG?
Chunking decides what the system is able to retrieve at all. A model can only answer from the passages it receives, so a split that separates a rule from its exception caps the best possible answer before ranking, prompt wording or model choice get a say. Chunking sets the ceiling. Everything downstream works beneath it.
That is also why chunking is the wrong thing to look at last. Teams reach for a better model or a longer prompt when answers come back thin, because those are the levers that feel adjustable. The passage boundary is the lever nobody wants to touch, since moving it means re-indexing the corpus, so it stays where whoever set up the pipeline left it. Stated as judgement rather than measurement, and consistent with the ordering we argue for in evaluating retrieval systems: the index is usually where the recoverable quality sits, and it is usually the last place anyone looks.
Two of seven failure points, and they need opposite fixes
There is a peer reviewed name for each version of this, which makes it something you can put in a specification instead of describing with your hands.
Barnett, Kurniawan, Thudumu, Brannelly and Abdelrazek published "Seven Failure Points When Engineering a Retrieval Augmented Generation System" at CAIN 2024, the 3rd International Conference on AI Engineering, in Lisbon. It is an experience report across three systems: a research tool, a university tutor, and biomedical question answering.
Two of their seven failure points describe the returns policy case, and the paper's own wording matters here.
FP2, Missed the Top Ranked Documents. "The answer to the question is in the document but did not rank highly enough to be returned to the user." Only the top K are passed on. Page 8 existed, scored lower than page 7 for that question, and never reached the model.
FP7, Incomplete. "Incomplete answers are not incorrect but miss some of the information" that was present in the context. Both passages reached the model and the reply used one of them.
Same refund, same angry customer, two different defects. FP2 is a retrieval problem and you fix it in the index, the splitter or the ranking. FP7 is a generation problem and you fix it in the prompt, the answer format or a verification step. We have written at length on telling a retrieval failure apart from a writing failure, and this is the cheapest reason to do it: the two cures are different and guessing wastes a sprint.
Ask your team which of the two they are looking at before anybody changes a setting. A team that cannot tell you does not have provenance logging, and that is the gap to close first.

What is the best chunk size for RAG?
There is no single correct size. Published research finds that semantic independence between passages matters to performance, while concept unity within a passage shows minimal impact, so the test is whether a passage read alone states a rule it does not finish. Split on the document's own structure, then measure against real questions.
Now the part that argument displaces. The instinctive response to the returns policy is to make the chunks bigger so the rule and its exception land together. Hold that against the research.
Brådland, Goodwin, Andersen, Nossum and Gupta, in "A New HOPE: Domain-agnostic Automatic Evaluation of Text Chunking" (arXiv 2505.02171, submitted 4 May 2025 and marked to be published at SIGIR 2025), report two findings that sit awkwardly beside common practice. "Semantic independence between passages proves essential for system performance." And: "On the contrary, traditional assumptions about maintaining concept unity within passages show minimal impact."
Read that as a buyer. The property being sold to you as the fix, keeping related ideas inside one chunk, is the one the paper finds matters least. The property nobody mentions in a proposal, that each passage can be read alone without misleading you, is the one it finds essential.
The arithmetic of boundaries is against you anyway. Twelve pages split at any fixed size still has boundaries. Doubling the size halves their number and does nothing about where they land, because where they land is a property of the document rather than of the setting. A policy that states rules and then qualifies them will put a qualification immediately after a boundary sooner or later, and the longer the document the more certain that becomes.
This is also why the usual sweep, trying 256 then 512 then 1,024 tokens and reading the scores, can show you a winner and still leave the defect in place. The sweep is worth running, and the mechanics of it belong in a retrieval evaluation suite rather than in a one-off chunking decision. What the sweep does not surface is a passage that quietly asserts half a rule, because on average it scores fine.
A related reflex is worth heading off here. Teams who cannot get the passages right sometimes propose training the model on the documents instead, which swaps a retrieval problem for a different one, as we set out in retrieval against fine-tuning. A model trained on a policy manual still has no way to tell you that page 8 qualifies page 7.
What is semantic chunking, and is it the fix?
Semantic chunking splits text where the meaning shifts, using the similarity between neighbouring sentences to choose the break instead of counting tokens to a fixed limit. It helps on messy documents. It is not the same thing as semantic independence, and the difference is the whole argument of this post.
A semantically coherent passage can still assert an unfinished rule. "Items may be returned within thirty days" is one clean idea, which is precisely why a meaning-based splitter is happy to end a passage there. The exception on the next page is a different idea, so the splitter is doing its job correctly and the retrieved passage is still misleading on its own.
Independence is a property of the pair, not of the passage. Two passages are independent when neither changes the meaning of the other. That is harder to detect automatically than a topic shift, and it is why the cheap version of this check is a person reading twenty real retrieved passages rather than a better splitter.
Semantic chunking is still usually an improvement over fixed-size on prose with no headings, such as call transcripts or scraped pages. Where documents carry their own structure, splitting on that structure is cheaper and more predictable, and it brings the heading path with it for free.
One caveat on how far a splitter can take you. No boundary rule handles a question whose answer is assembled from four documents, and that is where a system that retrieves, reads and retrieves again earns its cost, which we cover in agentic retrieval. Reach for it when the corpus is genuinely fragmented, not when a single clause got cut.
Validation is only feasible during operation
The Barnett paper's first takeaway is the sentence to carry into your commissioning conversation: "validation of a RAG system is only feasible during operation."
That is a conclusion drawn from building three of these systems, not a disclaimer. It says the thing you want, a chunking strategy signed off before the build starts, is not available at any price.
What follows from it is specific, and it changes what you buy.
You are buying the ability to re-index, not a configuration. If changing the splitter means a two-week re-run that somebody has to be talked into, you will not change it, and the first configuration becomes permanent by exhaustion. Ask how long a full re-index takes and what it costs to run. Ask whether it can be triggered without an engineer present.
You are buying provenance. Every answer needs to carry which passages produced it, stored, queryable, and readable by somebody who is not an engineer. Without it you cannot separate FP2 from FP7 after the fact, and the first serious complaint becomes an argument rather than an investigation. The same requirement shows up wherever a system acts on retrieved material, which is the thread running through evaluating AI agents.
You are buying a second pass with a date on it. Not a support retainer. A scheduled review of real questions and the passages that answered them, with authority to change the splitter and re-index. Put it in the statement of work with a week number.
A build that ends at go-live has skipped the only validation the research says is available. That is worth being blunt about with whoever is quoting you, including us.
Three figures about chunking that are not on this page
Search this topic and you will meet three numbers. All three were checked while this post was researched and none of them survived. The reasons are more useful than the figures.
"Naive RAG fails about 40% of the time." One blog post, no study behind it. There is no stated corpus, no task, no model and no definition of failure, so there is nothing to agree or disagree with.
"A 9% recall gap between the best and worst chunking methods." A vendor's own figure for its own product. It may well be accurate inside that vendor's evaluation. It is not a measurement anyone else can reproduce, and procurement is a poor place to use a number you cannot check.
"A context cliff at roughly 2,500 tokens." One preprint, reported secondhand, never verified against the paper. A specific threshold with a round number is the most quotable kind of claim and the easiest to pass along unread.
Two smaller exclusions, for the same discipline. The HOPE paper reports gains of up to 56.2% in factual correctness and 21.1% in answer correctness; those are not quoted here, because the abstract's statistical reporting could not be reconciled from the material read, and a figure whose basis is unclear is a figure that gets quoted back at you. The Barnett paper gives two different sizes for its biomedical corpus, 15,000 documents in one place and 4,017 in another, so this post prints no corpus size at all.
The rule we work to: where a source contradicts itself, use no figure. Not the larger one, not the average, not a hedge. Nothing.
That matters to you commercially rather than academically. A number in your pitch deck becomes a number in a customer contract, and then a number in a dispute. The qualitative findings above sound weaker and stand up better.
What to specify instead of a chunk size, and how to accept the work
You do not need to understand embeddings to write a specification that catches this. None of the following asks is a number.
What to ask for instead of a number
- The standalone test. Pick twenty passages the system actually retrieved last week. Read each one alone, with no document around it. Mark any that state a rule, a price, a period or an entitlement without the qualification that follows it. That count is your defect rate, and it is the only chunking metric a non-technical owner can audit without help. Calling it the highest-value check available to you is engineering judgement, not a measured result.
- Structure before size. Does the splitter break on headings, clause numbers and list items, or does it count tokens? Documents written by people carry boundaries the author already decided were meaningful, and a splitter that ignores them discards free information.
- Exceptions as a named class. "Unless", "except where", "this does not apply to", "subject to". Your documents have a vocabulary for qualifying rules, and a splitter aware of it, even crudely, stops making the specific mistake that costs you refunds.
- The heading path travels with the passage. A passage arriving with "Returns policy, section 4, Faulty goods" attached is a different piece of evidence from the same words arriving bare. Where the corpus already lives in an AI document management system, that path usually exists in the metadata and is simply being dropped at indexing time.
- Re-index on demand, timed in hours, from the team, before contract.
Then write the acceptance test, because most of these projects are accepted on a demo instead. Somebody asks six questions in a meeting, the answers are good, and the invoice clears. Those six questions are drawn, consciously or not, from the middle of the document where nothing is qualified.
Draw the question set from real traffic or from the questions your support team already answers by hand, and insist that a stated share of them target exceptions, edge dates and conditional entitlements. Twenty such questions are worth more than two hundred general ones. Score the passages, not only the answers: for each question, record whether the passage that should have answered it was retrieved at all, because that single column separates FP2 from FP7 across the whole set. Run the test after go-live as well as before it, with a named week and a budget attached. And keep the set, because it becomes the regression check for every later change to the splitter, the model or the prompt.
One more thing to settle in the contract: who owns the index and the logs. If the passages and their provenance live only inside a vendor's platform, the standalone test becomes something you request rather than something you run. That applies whether you are commissioning a bespoke build or buying a chat with documents product and configuring it.

What the fix costs, and the figure we will not invent
Our published rate band is $40 to $100 per hour by role, blending to $60 to $70 for a mixed team. We publish no module hours for this work, and an hour range with no measurement behind it would tell you nothing useful, so there is not one here. A discovery conversation prices the work.
What it prices is usually three pieces. A structure-aware splitter with a re-indexable pipeline for one document set. Provenance logging that records which passages produced each answer. A question set of around sixty real questions, half of them chosen because they land on exceptions. How long those take depends on how structured the source documents are and on what already exists.
Two cost traps are worth naming. The first is paying for a chunking strategy as a one-off deliverable, which buys a configuration and no way to change it. The second is paying for a pipeline only the vendor can re-index, which is the same trap in different clothes.
The expensive version of this problem has no figure attached, and inventing one would be worse than leaving the gap. It is the period between go-live and the day somebody notices, during which a confident, well-formatted, incorrect answer went out to customers under your name. How long that period runs depends entirely on whether you built the logging.
What we would do first
Before touching the splitter, read the documents. Which of them state rules and then qualify them elsewhere? A product catalogue with independent entries has almost none of this problem. A policy manual, an insurance wording, a staff handbook or a tariff schedule is built from it. That answer changes the size of the job more than any technical choice downstream, and it takes an afternoon with the actual files rather than a workshop.
Then turn on logging, even against a system you consider temporary. Retrieval you did not record is a week of guessing later.
Then split on structure, attach the heading path, and run the standalone test on real retrieved passages rather than on a sample you assembled yourself. The passages your users pull are not the passages you would pick.
Defer the interesting work. Re-ranking, query rewriting, a graph over the corpus and multi-step retrieval are all real and all cheaper to add once you can measure whether they helped. An agentic retrieval loop that searches, reads and searches again is a reasonable answer to a hard corpus and the wrong first answer to a split clause. Training is a different tool for a different problem, which is the argument in retrieval against fine-tuning, and it does nothing for a rule the model never saw. A document management build that can re-index on demand and show its sources is worth more than a cleverer first version that can do neither, and how documents and tools reach the model, including through the Model Context Protocol, is a question you can answer once the corpus behaves.
What we will not tell you before a discovery is your chunk size. Not because it needs a workshop, but because the only honest version of that answer arrives a few weeks after your users do.
Frequently asked questions
There is no size that is correct across documents, and choosing one is not the decision that fixes wrong answers. A 2025 preprint marked for SIGIR 2025 (Brådland et al., arXiv 2505.02171) finds that semantic independence between passages is essential to performance while concept unity within a passage shows minimal impact. The rule that follows: split so no passage, read alone, states a rule it does not finish, and split on the document's own headings and clauses rather than on a token count. Sweeping 256, 512 and 1,024 tokens is still worth doing inside an evaluation suite, but treat the result as a tuning gain rather than a fix.
Because the information it needed was split across a boundary and only one side of it was used. The CAIN 2024 paper by Barnett et al. names two distinct causes. FP2, Missed the Top Ranked Documents, is when "the answer to the question is in the document but did not rank highly enough to be returned to the user". FP7, Incomplete, is when the information did reach the model and the answer missed some of it anyway. The first is a retrieval defect and the second is a generation defect, and they take opposite fixes, so record which passages produced each answer before changing anything.
Not reliably. A larger chunk moves the boundary and reduces how many boundaries exist; it does not control where they fall, and where they fall is a property of the document. A long policy that states rules and then qualifies them will eventually put a qualification immediately after a boundary at any fixed size. Splitting on the structure the author wrote, and keeping the heading path with each passage, addresses the cause instead of the symptom.
It is the property that a passage means the same thing read alone as it does inside the document. A passage stating "items may be returned within 30 days" is not semantically independent if the next line, outside the passage, extends that window to 60 days for faulty goods, because reading it alone misleads you. Brådland et al. report that this property "proves essential for system performance", in contrast to the more commonly pursued goal of keeping one concept inside one passage.
Not completely. The first takeaway of Barnett et al., CAIN 2024, is that "validation of a RAG system is only feasible during operation", drawn from building three such systems. Pre-launch testing finds real defects and is worth running. It cannot tell you which passages your users will actually pull, which is the information the audit needs. Commission a scheduled post-launch pass with authority to change the splitter and re-index, and make retrieval logging a day-one requirement so that pass has data to work with.
Split on structure where the documents have any. Headings, clause numbers, list items and table rows are boundaries a human author already judged meaningful, and a token counter discards that information for free. Fixed-size or meaning-based splitting is a reasonable default for unstructured text such as transcripts or scraped pages. Whichever you use, test it by reading real retrieved passages on their own and counting how many assert a rule they do not finish.
Our published rate band is $40 to $100 per hour by role, blending to $60 to $70 for a mixed team, and we publish no module hours for this work, so we are not going to put an hour figure on it here. The work is usually three pieces: a structure-aware splitter with a re-indexable pipeline, provenance logging, and a question set of real questions weighted towards exceptions. The real variable is whether retrieval was logged from the start, because without logs the first step is recreating evidence rather than reading it.

Vikas Choudhary
Vikas has around fifteen years of experience building software and now builds generative AI systems at Zyneto. His work covers retrieval augmented generation, agentic AI, knowledge graphs, AI memory, and the evaluation and guardrails that decide whether any of it is safe to put in front of customers. He has shipped enterprise copilots, document AI, chatbots and predictive analytics for e-commerce, fintech and marketing teams, and works day to day in Python, JavaScript and SQL. He follows multimodal models, business process automation and enterprise AI security closely, and mentors engineers moving into AI. He writes about architecture, inference cost and the failure modes that only show up at production scale.
Zyneto builds custom software, AI and automation
Tell us what you are building and we will reply with a clear next step.









