Chunking PDFs for RAG: Why Fixed-Size Splitting Fails
Every RAG tutorial reaches for a fixed character count with some overlap. It is easy to implement and it quietly wrecks retrieval quality. Structure-aware chunking is barely harder and substantially better.
RAG · · 4 min read
The default strategy, and what it does
The standard recipe is a recursive character splitter: cut the document every N characters, overlap consecutive chunks by some smaller number, embed each chunk, store the vectors. It appears in nearly every tutorial because it works on any input and requires no understanding of the document.
What it produces is chunks that begin and end at arbitrary positions. A chunk starts mid-sentence, in the middle of a thought whose subject appeared 200 characters earlier. It ends mid-clause. A table is split from the caption explaining it. A heading lands at the tail of one chunk while the section it introduces sits in the next.
Overlap is the patch applied to this, and it is a poor one. It increases storage and embedding cost proportionally, duplicates content across neighbours so the same passage competes with itself in the result set, and still does not guarantee any individual chunk makes sense on its own.
Why arbitrary boundaries hurt retrieval specifically
An embedding is a summary of what a passage is about. That summary is only meaningful if the passage has a coherent topic. A fragment beginning "…and therefore the second condition does not apply, provided that" has no topic — its meaning lives entirely in text that is not present. Whatever vector it produces will be close to nothing useful and will surface for the wrong queries.
The failure is silent, which is what makes it dangerous. Retrieval returns something, the model produces a fluent answer from it, and nothing in the output signals that the retrieved passage was a decontextualised fragment. Debugging a RAG system that answers plausibly but wrongly usually leads back to the chunker.
There is a second, subtler cost. When one section is split across three chunks with no shared context, a query about that section may retrieve one of the three and miss the other two — so the model answers from a third of the available evidence while appearing to have consulted the document.
Chunking on structure instead
Documents already contain the boundaries you want. Sections are what an author decided belongs together, and they are almost always the right unit for retrieval. The problem is only that a PDF does not hand you its sections directly.
That is what our PDF to JSON converter produces: every line typed as heading, list_item or text, with its page number, font size and vertical position. Headings are inferred from relative font size, and the raw size is included so you can apply your own threshold when a document uses an unusual scale.
The algorithm is short. Walk the lines. Start a new chunk at every heading. Prepend the heading text — and ideally the parent heading too — to the chunk body, so the embedding captures what the section is about and not merely what it says. Then post-process: merge chunks below a floor into their neighbour, and split chunks above your model's comfortable ceiling at line boundaries rather than mid-sentence.
Prepending the heading is the highest-value line in that algorithm. A section body frequently never restates its own subject — a chunk under "Termination for Convenience" may discuss notice periods for three paragraphs without using the word "termination" once. Carrying the heading into the chunk fixes that mismatch entirely.
Citations, for free
Because every line carries its page number, each chunk knows which page it came from. Store that alongside the vector and your system can tell a user that an answer came from page 34 of a specific document.
This is not a cosmetic feature. It is the difference between an answer a reader can verify and one they have to trust. In any domain where being wrong is expensive — legal, medical, financial, compliance — an uncheckable answer is often worth less than no answer at all, and page-level citation is the cheapest possible way to make checking viable.
What structure-aware chunking will not fix
It does not repair bad extraction. If the source PDF is a two-column layout whose columns interleave, chunking correctly on a scrambled document gives you tidy chunks of scrambled text. Check the extraction first — see why PDF text extraction fails.
It does not help documents with no headings. Some PDFs — transcripts, correspondence, single-flow reports — have no hierarchy to find. There, fall back to paragraph and sentence boundaries, which are still better boundaries than character counts.
And it does not survive tables well. Tabular data rarely belongs in a prose chunk at all; extracting tables separately and storing them as structured records usually beats trying to embed a flattened grid.