Hey everyone, welcome to the thirtieth issue of The Main Thread. In this article, we will talk about various ways we can chunk data to be ingested in a RAG system.
Our RAG system is only as good as our chunks. Most tutorials on the web skip this part. They explain how to wire up an embedding model, load it into a vector store, and run a similarity search. “Just 50 lines of code, impressive demo, ship it”.
Shit gets real when production happens. Users ask questions that get answered with fragments that don’t have proper context. A perfect relevant passage is missed because it was split across two chunks. The LLM confidently answers from a chunk that’s adjacent to the right information but doesn’t contain it. The system doesn’t work reliably enough to matter.
Most of the time, the failure is in the chunking.
Chunking is a step where we take a raw document and divide it into retrievable fragments. Every decision we make here - how big, where to split, how much overlap, what metadata to attach - directly determines what our retrieval system can and cannot find.
After having built a few production-grade RAG systems, I wish this guide existed when I built the first RAG system.
TL; DR
Fixed Size Chunking
This is fast and predictable but semantically careless. This is fine for dense and uniform content, but bad for structured content.
Recursive Chunking
This approach is the practical default and it respects natural boundaries and degrades gracefully.
Semantic Chunking
This is the ceiling on quality but expensive computationally. We should use it only at places where precision matters the most.
Overlap
This prevents boundary loss but inflates the index. 10-20% overlap is usually the right range.
Chunk Size
This is a retrieval precision vs context coverage tradeoff. Smaller means more precise, larger means more coherent.
Metadata Enrichment
This is often the highest leverage improvement after we have a working chunking baseline working
Evaluate with recall, not just precision
A chunk we don’t retrieve is a failure we will never see in our logs.
Why Chunking is Hard
Before diving into the strategies, we must understand the problem.
Embedding models have a token limit (usually 512 to 8192 tokens) depending on the model. We cannot embed an entire document in one vector. We must split it. But splitting is a lossy operation because the meaning of a sentence often depends on what came before it. Similarly, a paragraph’s significance often depends on the section it’s in.
There is no universally optimal chunk size. In fact, the right size depends on the embedding model, query patterns, document types, and what a “good answer” looks like in a specific domain.
What follows are the strategies to navigate these tradeoffs deliberately rather than accidentally.
Strategy 1: Fixed size chunking
This is the simplest approach: split the document every N characters or N tokens, regardless of content.
def fixed_size_chunks(
text: str,
chunk_size: int = 512,
overlap: int = 50
) -> list[str]:
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunks.append(text[start:end])
start += chunk_size - overlap
return chunksThis approach works with dense, uniform content such as transcripts, logs, financial filings, data-heavy reports, where every sentence is equal in information density and there are no meaningful structural boundaries.
This approach fails for anything with structure, such as a technical blog post with headers, code blocks, and paragraphs. Fixed-size splitting will cut mid-sentence, across code blocks, through headers. The chunk boundary is arbitrary w.r.t. meaning.
There is one more problem: fixed size splitting is semantically careless. It doesn’t know that “Therefore” starts a conclusion that references previous three paragraphs. It doesn’t know that a code snippet and its explanation belong together. It just counts characters.
We can use this as a baseline to beat but not a production default.
Strategy 2: Recursive Character Text Splitting
This approach is the practical default for most RAG systems and the right starting point for new projects.
The idea is to define a priority-ordered list of separators. We try to split on the highest priority separator first (example: double new line \n\n for paragraphs). If a resulting chunk is still too large, we split on the next separator in the priority list (single new line, then space, then character). We recurse until all chunks are within the size limit.
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000, # Target chunk size in characters
chunk_overlap=200, # Overlap between chunks
separators=[
"\n\n", # Paragraph break (top priority)
"\n", # Line break
". ", # Sentence end
", ", # Clause break
" ", # Word break
"" # Character (last resort)
]
)
chunks = splitter.split_text(document_text)This approach works well in practice because it respects natural language boundaries in priority order. A paragraph split is always preferred over a sentence split, which is preferred over a word split. The chunk size constraint is a ceiling - chunks can be smaller if the document structure allows it.
However, the graceful degradation property is key. If the document is well-structured with clear paragraph breaks, most chunks will be coherent paragraphs. If it is not well-structured, the splitter falls back to sentence breaks. It never does anything outright wrong.
Customizing separators for content type
# For Markdown documents
markdown_separators = ["## ", "### ", "\n\n", "\n", ". ", " ", ""]
# For code files
code_separators = ["\nclass ", "\ndef ", "\n\n", "\n", " ", ""]
# For legal documents
legal_separators = ["\nSection ", "\nArticle ", "\n\n", "\n", ". ", " ", ""]We must always tune separators to our content type. Defaults can take you only so far; domain-specific chunks are better.
Strategy 3: Semantic Chunking
Semantic chunking splits the text on meaning boundaries, unlike fixed-size and recursive chunking which split on syntactic (whitespace, punctuation, delimiters, etc.) boundaries.
Semantic chunking works like this:
Embed each sentence separately
Then measure the cosine similarity between consecutive sentence embeddings
Wherever similarity drops sharply, we insert a chunk boundary because those sentences are about different topics.
from sentence_transformers import SentenceTransformer
import numpy as np
def semantic_chunks(
text: str,
model: SentenceTransformer,
breakpoint_threshold: float = 0.3
) -> list[str]:
sentences = text.split(". ")
embeddings = model.encode(sentences)
chunks = []
current_chunk = [sentences[0]]
for i in range(1, len(sentences)):
# Cosine similarity between consecutive sentences
sim = np.dot(embeddings[i-1], embeddings[i]) / (
np.linalg.norm(embeddings[i-1]) * np.linalg.norm(embeddings[i])
)
if sim < breakpoint_threshold:
# Semantic shift detected — close current chunk
chunks.append(". ".join(current_chunk))
current_chunk = [sentences[i]]
else:
current_chunk.append(sentences[i])
if current_chunk:
chunks.append(". ".join(current_chunk))
return chunksThis approach works better for documents with variable topic density, such as blog posts that transition between concepts, research papers with distinct methodology and result sections, support documentation that mixes problem description with resolution steps. Semantic chunking finds natural topic boundaries that syntactic splitting misses.
But this approach has drawbacks too - we are running an embedding model twice (once to chunk, once to index). For large document corpora, this is expensive. Semantic chunking at index time is often practical but doing it at query time is not.
If the threshold is too low, we over-segment (every sentence becomes its own chunk). If it is too high, we under-segment (the whole document becomes one chunk). We need to tune breakpoint_threshold empirically on a sample of actual documents.
Overlap Strategies and Boundary Handling
Information gets lost at chunk boundaries. The sentence that starts chunk N + 1 might be incomprehensible without the sentence that ended chunk N. Overlap is the mitigation.
Overlap means the tail of each chunk is repeated as the head of the next.
Document: [A][B][C][D][E][F][G][H]
Chunk 1: [A][B][C][D][E]
Chunk 2: [C][D][E][F][G] ← C,D,E repeated
Chunk 3: [E][F][G][H] ← E,F,G repeatedThis ensures that if a relevant sentence falls near a boundary, it appears in two chunks. Retrieval has two choices to find it.
How much overlap?
A practical band is 10-20% of chunk size. For a 1000-character chunk, 100-200 characters of overlap. Beyond 20%, we are inflating index size significantly without proportional retrieval gains as the duplicate content starts showing up as near duplicate results in the similarity search.
Sentence-boundary overlap is better than character-boundary overlap. If we are going to repeat content, it’s better to repeat sentences rather than fragments:
def sentence_aware_overlap(
sentences: list[str],
chunk_size: int,
overlap_sentences: int = 2
):
chunks = []
i = 0
while i < len(sentences):
chunk_sentences = []
total_len = 0
j = i
while j < len(sentences) and total_len + len(sentences[j]) < chunk_size:
chunk_sentences.append(sentences[j])
total_len += len(sentences[j])
j += 1
chunks.append(" ".join(chunk_sentences))
i = j - overlap_sentences # Step back by overlap_sentences before next chunk
return chunksDocument Structure Preservation
For structured documents like PDFs with headers, markdown files, HTML pages, the structure itself is a semantic signal. Losing it degrades retrieval quality in ways that are hard to diagnose.
Hierarchical chunking preserves document structure by chunking within structural boundaries:
import re
def hierarchical_chunks(
markdown_text: str,
max_chunk_size: int = 1000
) -> list[dict]:
# Split on headers first, preserving hierarchy
sections = re.split(r'(#{1,3} .+)', markdown_text)
chunks = []
current_header = ""
for section in sections:
if re.match(r'#{1,3} ', section):
current_header = section.strip()
else:
# Further split large sections
if len(section) > max_chunk_size:
sub_chunks = recursive_split(section, max_chunk_size)
for sub in sub_chunks:
chunks.append({
"content": sub,
"section": current_header,
})
elif section.strip():
chunks.append({
"content": section.strip(),
"section": current_header,
})
return chunksThis approach gives us chunks that are always within a coherent section. A chunk from ## Installation will never contain content from ## Configuration even if they are adjacent.
Code blocks deserve special treatment. We should never split a code block mid-block. Embed the entire code block as one chunk (even if it exceeds usual chunk size), paired with its surrounding explanation:
def extract_code_aware_chunks(markdown: str) -> list[str]:
code_block_pattern = re.compile(r'```[\s\S]*?```', re.MULTILINE)
chunks = []
last_end = 0
for match in code_block_pattern.finditer(markdown):
# Chunk the text before this code block
preceding = markdown[last_end:match.start()]
if preceding.strip():
chunks.extend(recursive_split(preceding))
# Keep code block intact, paired with its preceding sentence for context
context_start = max(0, match.start() - 200) # ~2 sentences of context
chunks.append(markdown[context_start:match.end()])
last_end = match.end()
# Chunk remaining text
remaining = markdown[last_end:]
if remaining.strip():
chunks.extend(recursive_split(remaining))
return chunksChunk Size Tradeoffs
There is a direct trade-off between retrieval precision and context coverage.
Smaller chunks (256–512 tokens):
Higher retrieval precision: the chunk is more likely to be entirely relevant to the query
Lower semantic coherence: a 256-token chunk often lacks sufficient context to be self-explanatory
Better for factual Q&A: where we need a specific fact, not an explanation
Larger chunks (1,024–2,048 tokens):
Better semantic coherence: more context per chunk
Lower retrieval precision: the chunk contains both relevant and irrelevant content, diluting the embedding
Better for summarization tasks or open-ended questions that need more context
The practical calibration heuristic:
We should match chunk size to the typical query scope. If users ask narrow factual questions ("What is the default timeout for X?"), use smaller chunks. If users ask conceptual questions ("How does X work?"), use larger chunks.
Small-to-big retrieval
This is a pattern that gets the best of both worlds: we should index small chunks for precision, but retrieve the parent chunk (or surrounding context window) for response generation.
LangChain’s ParentDocumentRetriever implements this pattern. An important thing to notice is: the unit of retrieval doesn’t have to be the unit of generation.
Metadata Enrichment
Metadata is often the most important improvement after we have a working chunking baseline.
Every chunk should carry structured metadata beyond its content vector. Metadata enables filtered retrieval → constraining the search space before similarity matching - and post-retrieval ranking based on signals that embeddings don’t capture.
High-value metadata fields:
chunk = {
"content": "...",
"metadata": {
# Provenance
"source_document": "user-guide-v2.pdf",
"source_url": "https://docs.example.com/guide",
"document_type": "documentation",
# Structure
"section_header": "Installation",
"page_number": 14,
"chunk_index": 3,
# Temporal
"created_at": "2025-11-01",
"updated_at": "2026-01-15",
# Domain-specific
"product_version": "3.2",
"audience": "developer",
"language": "en",
}
}Document level summary injection is an underused technique. Prepend a brief document summary to each chunk before embedding. This gives the embedding model signal about the overall document context, improving retrieval for queries that match document’s topic but not a specific chunk’s literal text.
def enrich_with_summary(
chunk_text: str,
doc_summary: str
) -> str:
return f"Document context: {doc_summary}\n\n{chunk_text}"This adds ~50 tokens per chunk but materially improves recall for broad queries. The tradeoff is worth it for most use cases.
Evaluating Chunk Quality
Usually we evaluate RAG by looking at final answer quality. This confuses chunking quality, retrieval quality, and generation quality into a single signal. When answers are bad, we don’t know which layer to fix.
We must evaluate chunking independently.
Recall@K
This means for a set of test queries with known relevant passages, what fraction of relevant passages appear in the top K retrieved chunks. Low recall means good information exists in the corpus but is not being retrieved. This is almost always a chunking or embedding issue.
def recall_at_k(
queries,
relevant_passages,
retriever, k=5
):
recalls = []
for query, relevant in zip(queries, relevant_passages):
retrieved = retriever.retrieve(query, k=k)
retrieved_contents = {c.content for c in retrieved}
hit = any(passage in content for passage in relevant
for content in retrieved_contents)
recalls.append(1.0 if hit else 0.0)
return sum(recalls) / len(recalls)Chunk Coherence Check
We sample 50 random chunks and read them cold. Would these make sense to someone without document context? If more than 20% are confusing in isolation, the boundaries are wrong
Boundary Audit
We find chunks where the first sentence starts with a pronoun (“It”, “This”, “They”) or a transitional word (“However”, “Therefore”, “Additionally”). These chunks likely start mid-thought - symbols of missed boundaries.
import re
def boundary_audit(chunks: list[str]) -> float:
boundary_words = re.compile(
r'^(It|This|These|They|He|She|However|Therefore|Additionally|Furthermore|Moreover)\b'
)
bad_starts = sum(1 for c in chunks if boundary_words.match(c.strip()))
return bad_starts / len(chunks) # Target: < 0.10A boundary audit score more than 10% is a red flag. We need to adjust our separators and overlap until it drops.
Decision Framework
We should start with recursive splitting, then measure recall@k. If it is below the target, we move up the complexity ladder → semantic chunking → hierarchical splitting → metadata enrichment. We should add complexity only where the metrics justify it.
The best chunking strategy is the simplest one that meets our recall target on real queries from real users. We must build that benchmark first. Everything else is optimization.
Mental Model
Embeddings encode meaning in vectors (covered in the Understanding Embeddings series). But embeddings can only encode meaning that is present in the text they are given. If the chunk contains half a thought, the embedding encodes half a thought. If the chunk mixes three topics, the embedding is a blurry average of three topics.
Chunking is the art of giving the embedding model coherent, self-contained units of meaning to work with. Every split we make is a decision about what constitutes a retrievable unit of knowledge in your corpus.
Get that decision right, and our retrieval is precise and reliable. Get it wrong, and we are explaining to stakeholders why the RAG system can't find the answer that's clearly in the documents.
Would love to know your thoughts on this!
Namaste!
— Anirudh
