The RAG Setting That Does Nothing, Except When It Might

Chunk overlap is one of those RAG settings people turn on without asking much from it.

Ten percent feels safe. Twenty percent feels safer. The index gets bigger, ingestion takes longer, and everyone assumes retrieval improved somewhere.

A 2026 paper tested that assumption. In its setup, overlap produced no measurable improvement in BERTScore or exact match. It only created more chunks to index.

I saw the same result on a client system. We turned overlap off. Recall did not move, and indexing became cheaper.

That does not prove overlap is useless. It proves something more practical: overlap should earn its place in your pipeline.

What the paper measured

Sofia Bennani and Charles Moslonka published A Systematic Analysis of Chunking Strategies for Reliable Question Answering in January 2026. A conference version appeared in the ECIR 2026 proceedings in March.

The researchers built a two-stage RAG pipeline with:

• the short-answer subset of Natural Questions;
• English Wikipedia as the document collection;
• SPLADE as the sparse retriever;
• Ministral-8B-Instruct-2410 as the generator;
• low-temperature generation at 0.1;
• no reranker;
• chunk sizes from 50 to 500 tokens;
• overlap settings of 0% and 20%;
• context budgets of 500, 1,000, 2,500, 5,000, and 10,000 tokens.

They compared token, sentence, semantic, and code-aware chunking. Quality was measured with BERTScore, exact match, and the rate at which the model returned "NONE" because the supplied context was insufficient.

This setup matters as much as the result.

Finding 1: overlap added cost, not quality

The paper reports no measurable improvement from 10% to 20% overlap across paired configurations. The reported BERTScore differences stayed within 0.004, and exact-match differences stayed within 0.001.

Overlap did increase index size.

The paper gives a simple estimate. If the overlap ratio is r, the number of chunks grows by roughly:

1 / (1 - r)

At 20% overlap, that produces about 1.25 times as many chunks. In plain terms, you add roughly 25% more items to index, store, and search.

That cost can appear in several places:

• document ingestion takes longer;
• the index needs more storage;
• near-duplicate chunks compete during retrieval;
• reindexing becomes slower;
• dense systems create and store more embeddings;
• downstream rerankers may process repeated evidence.

If quality stays flat, every one of those costs is waste.

The limitation people leave out

The paper used SPLADE, which is a sparse neural retriever. It represents text through weighted vocabulary terms and learned lexical expansion.

It did not test the common setup where documents and queries are converted into dense embedding vectors. It also excluded rerankers and late-interaction models such as ColBERT.

That difference stops us from turning the result into a universal rule.

With dense retrieval, an idea split across a chunk boundary may produce weaker embeddings on both sides. Overlap could preserve enough surrounding meaning to improve retrieval. That is a reasonable hypothesis, not a result from this paper.

Hybrid retrieval may behave differently again because it combines lexical and semantic signals. A reranker can also change which duplicates or boundary fragments reach the model.

So the accurate conclusion is:

In this sparse SPLADE setup, overlap produced no measurable quality gain and increased indexing cost. Other retrieval systems still need their own test.

That sentence is less dramatic than "overlap is useless." It is also much more useful in production.

Finding 2: sentence chunking was the cheaper default

Sentence chunking matched semantic chunking up to context budgets of about 5,000 tokens. The paper recommends sentence chunking as the cost-effective default for text-based question answering.

This is worth testing before adopting a semantic chunker.

Semantic chunking sounds more advanced because it groups nearby sentences based on similarity. In the experiment, the semantic method used all-MiniLM-L12-v2 and merged adjacent sentences when cosine similarity exceeded 0.5, up to the target chunk size.

That extra work did not produce a measurable advantage over sentence chunking at context sizes up to about 5,000 tokens. Semantic chunking showed a slight edge only beyond that range.

The study's practical defaults were:

Setting

Paper's default

Reason

Overlap

0%

No measurable benefit in the tested setup

Chunker

Sentence

Matched semantic up to about 5,000 context tokens

Chunk size

150 to 300 tokens

Balanced retrieval and abstention

QA context

About 2,500 tokens

Improved exact match without excessive context

Summary context

About 500 tokens

Favored semantic faithfulness

Context above 5,000

Consider semantic

Small measured advantage at large context sizes

These are starting points from one experiment, not production defaults for every company.

Finding 3: more context eventually reduced quality

The researchers also found a "context cliff."

For sentence chunks with a target size of 300 tokens and no overlap, BERTScore stayed fairly stable between 500 and 2,500 context tokens. At 10,000 tokens, it fell by roughly 4% to 5% relative to the earlier range.

The best context size depended on the goal:

• BERTScore tended to peak near 500 tokens.
• Exact match tended to peak near 2,500 tokens.
• The model returned "NONE" less often as more context was supplied.

This creates a real product choice. A larger context may help the system find an exact fact and answer more often. It may also introduce enough irrelevant material to weaken semantic quality.

Do not optimize only for the number of answered questions. A system that answers everything can still answer badly.

Check your retriever before touching overlap

Before changing a production setting, identify the retrieval path that is actually running.

Record:

1. Retriever type: sparse, dense, hybrid, or late interaction.
2. Model or index: SPLADE, BM25, embedding model, hybrid weights, or another method.
3. Chunking method: token, sentence, semantic, layout-aware, or custom.
4. Chunk size and overlap, including units.
5. Retrieval rule: top-k, score threshold, or fill-to-budget.
6. Reranking method, if present.
7. Generator model, prompt version, and context budget.
8. Current index size, ingestion time, latency, and cost.

Do not trust the configuration file alone. Trace one real question through production and inspect the chunks that reached the model. Old indexes, environment overrides, and untracked defaults can make the live system different from the documented one.

A two-hour chunking test

You do not need a month-long RAG study to decide whether overlap belongs in one application. You need a fixed corpus, representative questions, and a paired comparison.

Step 1: define the decision

Write one sentence before running anything:

We will remove overlap if answer quality stays within our accepted margin and indexing cost falls.

Set the margin before seeing the results. The acceptable difference depends on the risk of the workflow. A small loss may be fine for an internal search tool and unacceptable for a legal or medical assistant.

Step 2: freeze a production sample

Take a fixed snapshot of the documents. Remove duplicates and record a checksum or version for the corpus.

Build the question set from real usage where possible. Include:

• common questions;
• questions whose answer sits near a section boundary;
• questions that require two nearby passages;
• questions with no answer in the corpus;
• documents with tables, lists, headings, and long paragraphs.

A quick test can start with 100 to 200 questions. A higher-risk launch needs a larger set and a formal power analysis based on the smallest quality change the business cares about.

For each answerable question, record the expected answer and the source document. Page, section, or paragraph references make retrieval scoring easier.

Step 3: create the smallest useful comparison

Keep the retriever, generator, prompt, context budget, and question set fixed. Change only the chunking configuration.

Start with:

Variant

Chunker

Overlap

Purpose

A

Current production method

Current value

Baseline

B

Current production method

0%

Measures the value of overlap

C

Sentence

0%

Tests the cheaper default

D

Semantic

0%

Tests whether semantic grouping earns its cost

If your current system uses dense retrieval, add a small boundary-heavy subset. This is where overlap has the strongest reason to help.

Step 4: measure retrieval and answers separately

An answer score alone cannot tell you where the pipeline failed.

Measure retrieval with:

• source recall at your production cutoff;
• the rank of the first correct source;
• duplicate or near-duplicate chunks in the returned context;
• total retrieved tokens;
• percentage of questions with no relevant source retrieved.

Measure generation with:

• exact match or a task-specific grading rubric;
• citation accuracy;
• groundedness against the retrieved text;
• correct abstention when the answer is missing;
• human preference on ambiguous answers.

Measure operations with:

• number of chunks;
• index size;
• ingestion time and cost;
• retrieval latency at the median and 95th percentile;
• reranking and generation tokens;
• full cost per answered question.

Step 5: use paired results

Compare every configuration on the same questions. Report the difference per question, then calculate a 95% bootstrap confidence interval for the paired change.

The paper used this approach when judging whether differences were measurable. It is better than comparing two overall averages with no uncertainty.

Keep the generator deterministic if possible. If the model remains stochastic, repeat each run and separate retrieval variance from generation variance.

Step 6: inspect the disagreements

Overall scores can hide the only cases you care about.

Review questions where:

• overlap retrieves the source and zero overlap misses it;
• zero overlap works and overlap introduces duplicates;
• sentence and semantic chunking disagree;
• retrieval succeeds but the answer fails;
• a larger context changes a correct answer into a weak one.

Read the source around each boundary. This tells you whether the problem belongs to chunking, retrieval, reranking, or generation.

Step 7: make the production decision

Remove overlap when all three conditions hold:

1. Quality stays within the limit you set before the test.
2. Boundary-heavy questions do not show a meaningful regression.
3. Index size, latency, or ingestion cost improves enough to matter.

Keep overlap when it produces a repeatable gain on questions that represent real user value. Do not keep it because one hand-picked demo looks better.

If the result is mixed, segment the corpus. Policies, manuals, code, transcripts, and scanned reports do not always need the same chunker.

Roll it out without rebuilding everything at once

Changing chunking normally requires a new index. Treat that as a migration.

1. Build the new index beside the current one.
2. Replay recent queries in shadow mode.
3. Compare retrieved sources and answer scores.
4. Send a small share of live traffic to the new index.
5. Monitor quality, abstention, latency, and cost by document type.
6. Keep the old index available for rollback.
7. Increase traffic only after the agreed acceptance window passes.

Do not delete the old index on launch day. A cheaper configuration is not a win if one important document type stops working.

The production checklist

• [ ] We know whether retrieval is sparse, dense, hybrid, or late interaction.
• [ ] We measured overlap on our own documents and questions.
• [ ] The test changed one major variable at a time.
• [ ] Retrieval and answer quality were scored separately.
• [ ] We included questions near chunk boundaries.
• [ ] We measured index size, ingestion cost, and latency.
• [ ] We defined the acceptable quality margin before testing.
• [ ] We reviewed disagreements, not only averages.
• [ ] We tested context length as well as chunk size.
• [ ] The new index has a staged rollout and rollback path.

What to use as your default

If you run a sparse retrieval system similar to the paper, sentence chunking with no overlap is a strong baseline. A target size of 150 to 300 tokens and a QA context near 2,500 tokens are sensible first experiments because they come directly from the measured setup.

If you run dense or hybrid retrieval, do not copy the no-overlap result as a rule. Test zero overlap against your current setting, with extra attention to boundary-heavy questions.

If you use semantic chunking, compare it with sentence chunking before paying its extra ingestion cost. The paper found no measurable advantage up to about 5,000 context tokens.

And if your answer quality drops, do not assume the chunk size is wrong. The context budget, retrieval method, reranker, prompt, and generator can all change the outcome.

Overlap is not insurance. It is a hypothesis with a monthly bill attached.

Measure it.

Sources

• Bennani and Moslonka, A Systematic Analysis of Chunking Strategies for Reliable Question Answering, arXiv:2601.14123
• Peer-reviewed conference version in the ECIR 2026 proceedings, Springer