Chunking Markdown for RAG: Why the Format Already Did Half the Work
Markdown arrives at your pipeline with the hard part of chunking already solved: the boundaries are written in the text. Most pipelines throw that away by applying the same fixed-size splitter they use on PDFs. Here is how to chunk Markdown so the structure survives.
RAG · · 3 min read
Why Markdown is already the easy case
Chunking exists to give the retriever clean, self-contained passages. With PDFs and Word files, the boundaries have to be guessed from font sizes and indentation, and that reconstructed hierarchy is frequently wrong. Markdown states the hierarchy right in the characters: a heading line says what it is, a list marker says what it is, and no parser has to infer either one.
That is what makes the mistake with Markdown so frustrating. The default RAG recipe applies the same recursive character splitter to every format, so a Markdown document with clean heading boundaries gets sliced at arbitrary mid-paragraph positions, exactly like a PDF whose structure had to be guessed. The format hands you perfect boundaries, and the pipeline ignores them.
What fixed-size splitting does to a Markdown document
A heading lands at the tail of one chunk while its section starts in the next. A table is cut between its header row and its data. A fenced code block, often the one thing in the file a model needs to see whole, is split across two chunks, so neither contains runnable code. Ordered lists lose their numbers mid-sequence, which a model reading a chunk cannot reconstruct.
None of this is hypothetical. Every one of those boundaries is right there in the source text. The splitter is not unlucky, it just is not looking.
Heading-aware chunking
The fix is to split on structure instead of character count: cut at heading boundaries, keep each heading with the content it introduces, and let chunk size vary within a sane range. Two things make this work in practice rather than just sound good.
First, choose the split depth deliberately. Splitting at third-level headings gives many small chunks suited to point-answer retrieval; splitting at second-level headings gives fewer, larger chunks with more context per hit. Both are defensible. What matters is deciding once and sticking to it, instead of letting the split land wherever the character count happens to fall.
Second, handle the two elements that must never split: tables and fenced code blocks. A table is atomic, so keep it whole. If it sits inside a section too large to ship whole, give the table its own chunk and use the preceding heading as its title. A code block is likewise atomic, for the same reason: the code is useless in fragments. This is the Markdown equivalent of the rule in chunking PDFs for RAG: respect the boundaries the document gives you.
The recipe
Split at second-level headings by default. Keep each heading line at the start of its own chunk, because it is free context that names the chunk for the retriever. If a section exceeds the maximum chunk size, sub-split at the next heading level. If a single paragraph or element exceeds the limit on its own, ship it oversized rather than cut mid-sentence. A chunk of 2,000 characters that makes sense beats two 1,000-character fragments that do not.
Then the two special cases: tables and fenced code blocks get their own chunks, titled by their nearest heading, never split. Lists stay intact with their intro sentence. A numbered list whose first item is missing reads as fiction to a model.
When you have enough documents, store the heading path alongside each chunk as metadata. It costs nothing at ingest, and it turns "find passages about X" into "find passages about X in the deployment section", which is where retrieval actually pays off.
Getting any source into this shape
The recipe assumes Markdown input. If your corpus is PDFs, convert first. The PDF to Markdown converter extracts headings and lists in the browser, and the same applies to Word files through DOCX to Markdown. The quality of the chunking is then decided by the quality of that conversion, which is why it is worth skimming the Markdown for real headings before you build the index.
For pipelines that need page numbers or positions for citations, use PDF to JSON instead. Typed lines with coordinates give you chunk boundaries you can cite back to the original page, at the cost of losing the clean heading structure Markdown provides.