Chunking Strategies for RAG Pipelines

Why Chunking Matters So Much

The core tension in chunking is between granularity and context. Small chunks (100 to 256 tokens) are more precisely retrievable because each chunk represents a narrow topic, so a query about that topic will produce a strong similarity match. But small chunks lose surrounding context: a chunk that says "the latency was reduced by 40%" is less useful than one that also explains what was measured, what the baseline was, and what change produced the improvement. Large chunks (1024 to 2048 tokens) preserve that context but are less precisely retrievable because they blend multiple topics into a single embedding, diluting the signal for any one of them.

This tension cannot be eliminated, only managed. The goal is to produce chunks where each chunk contains exactly one coherent idea with enough surrounding context to be self-explanatory, but not so much that the embedding represents a blend of unrelated ideas. Different document types have different natural unit sizes, which is why there is no single optimal chunk size, and why the choice of chunking strategy matters more than the choice of chunk size within a strategy.

The impact is quantifiable. In controlled experiments, switching from fixed-size 512-token chunks to semantic chunking on the same document set typically improves retrieval recall@10 by 8% to 15% and answer faithfulness by 5% to 12%, with no changes to any other component. These numbers vary by domain and document type, but the direction is consistent: better chunking produces better retrieval produces better answers.

Fixed-Size Chunking

Fixed-size chunking splits text into segments of a uniform token count, typically 256 to 1024 tokens, with overlap between adjacent chunks. It is the simplest strategy and the best starting point for any new RAG pipeline because it requires no document analysis, works with any text, and provides a baseline against which more sophisticated strategies can be measured.

The implementation is straightforward: tokenize the text (using the same tokenizer as the embedding model), split at every N tokens, and copy the last M tokens of each chunk to the beginning of the next chunk as overlap. The overlap ensures that sentences or ideas that span a chunk boundary are fully contained in at least one chunk. Without overlap, a question about a concept that happens to fall at a boundary will find no single chunk containing the complete answer.

The practical parameters that work well for most text: 512 tokens per chunk with 100 tokens of overlap is the standard baseline. Smaller chunks (256 tokens) work better for FAQ-style content where each answer is short and self-contained. Larger chunks (1024 tokens) work better for narrative prose where ideas develop over multiple paragraphs. Overlap of 10% to 20% of chunk size provides adequate boundary coverage without excessive duplication.

The limitation of fixed-size chunking is that it ignores document structure entirely. It will split a table in the middle, separate a heading from the paragraph it introduces, and produce chunks that start mid-sentence and end mid-thought. For well-structured documents like technical documentation, knowledge base articles, and legal texts, this structural blindness is a significant source of retrieval quality loss.

Section-Based Chunking

Section-based chunking uses the document's structural signals, headings, paragraph breaks, list boundaries, and other formatting elements, to determine where to split. Instead of cutting at a fixed token count, it cuts at natural section boundaries, producing chunks that correspond to the document's own organizational structure.

For Markdown and HTML documents, this means splitting at heading tags (H2, H3) and treating each section as a candidate chunk. For PDFs with detectable section structure, it means using the parsed section hierarchy from a layout-aware parser. For plain text with paragraph breaks, it means treating each paragraph or group of related paragraphs as a chunk.

Section-based chunking produces higher quality chunks than fixed-size for structured documents because each chunk corresponds to a topic the document author explicitly defined. A chunk titled "Authentication" contains the document's full discussion of authentication, not a random 512-token slice that starts partway through authentication and ends partway through authorization.

The complication is size variance. Sections in real documents range from a single sentence to thousands of words. Very short sections (under 100 tokens) produce chunks too small for meaningful embedding, while very long sections (over 2000 tokens) produce chunks too large for precise retrieval. The standard solution is to merge adjacent short sections until they reach a minimum size threshold (200 to 300 tokens) and recursively split long sections at the next level of heading or at paragraph boundaries until they fall under a maximum size threshold (800 to 1200 tokens).

Section-based chunking also benefits from prepending the section hierarchy to each chunk as a contextual header. A chunk from the "Error Handling" subsection of the "API Reference" section benefits from having "API Reference > Error Handling" prepended, because the embedding captures this context and makes the chunk more precisely retrievable for queries about API error handling specifically rather than error handling in general.

Semantic Chunking

Semantic chunking detects topic boundaries within the text by measuring the embedding similarity between adjacent sentences or paragraphs. When the similarity drops below a threshold, the chunker places a boundary, producing chunks that each represent a single coherent topic as determined by the embedding model itself rather than by document structure.

The algorithm works as follows: embed each sentence in the document, compute the cosine similarity between consecutive sentence embeddings, and identify positions where the similarity drops sharply (below a percentile threshold, typically the 10th to 25th percentile of all pairwise similarities in the document). These drop points indicate topic shifts, and the text between two consecutive drop points forms a chunk. If a resulting chunk exceeds the maximum size limit, it is further split using a secondary strategy like paragraph-based splitting.

Semantic chunking is particularly effective for documents that lack explicit structure: meeting transcripts, email threads, conversation logs, and free-form reports. These documents have natural topic boundaries that humans would recognize, but no formatting signals like headings or section breaks that a structure-based chunker could use. The embedding model detects these boundaries from the content itself.

The trade-off is computational cost. Semantic chunking requires embedding every sentence in the document during ingestion, which is much more expensive than fixed-size or section-based chunking that only needs to tokenize and count. For a 100-page document with 3,000 sentences, semantic chunking generates 3,000 sentence embeddings just to decide where to split, before generating the chunk embeddings that will be stored. For large corpora, this cost multiplied across all documents becomes significant.

There is also a risk of over-segmentation. Short topic tangents that last one or two sentences, an aside, an example, a parenthetical explanation, produce similarity dips that look like topic boundaries but are actually minor digressions within a larger topic. Setting the threshold too aggressively produces fragmented chunks that each contain a single sentence or two, which defeats the purpose of chunking. Tuning the percentile threshold and setting a minimum chunk size (200 tokens) mitigates this.

Recursive Chunking

Recursive chunking, popularized by LangChain's RecursiveCharacterTextSplitter, tries to split at the largest natural boundary first and progressively falls back to smaller boundaries if the resulting chunk exceeds the size limit. The hierarchy of separators is typically: section breaks, paragraph breaks (double newlines), sentence boundaries (periods followed by whitespace), and finally arbitrary character positions as a last resort.

The algorithm attempts to split the entire document at section breaks first. If any resulting section is too large, it splits that section at paragraph breaks. If any paragraph is still too large, it splits at sentence boundaries. This produces chunks that respect the document's structure to the maximum extent possible while still conforming to the size limit.

Recursive chunking is a good general-purpose strategy that works well when you do not know the structure of your documents in advance or when your corpus contains a mix of document types. It naturally adapts to whatever structure exists: well-structured documents get split at section boundaries, unstructured prose gets split at paragraph or sentence boundaries. The main tuning parameters are the chunk size limit (512 to 1024 tokens), overlap (50 to 200 tokens), and the list of separators (which you can customize for your document type).

Document-Specific Strategies

Tables

Tables are the most common chunking failure in production RAG systems. A fixed-size chunker will split a table after a certain number of tokens, producing a chunk with the header row and the first few data rows and a second chunk with the remaining data rows but no header. The second chunk is nearly useless because the data values have no column labels, so neither the embedding model nor the LLM can interpret them.

The correct approach is to treat each table as an atomic unit. If the table fits within the chunk size limit, it becomes its own chunk with the table caption or preceding heading as context. If the table is too large, each row (or small group of rows) becomes a chunk with the header row and table caption prepended, so every row-chunk is self-explanatory. Some implementations also generate a natural language summary of the table and store it as an additional chunk, which improves retrieval for questions that reference the table's content conceptually rather than looking for specific cell values.

Code

Source code should be chunked at structural boundaries: function definitions, class definitions, module-level blocks. An AST-aware chunker parses the code into its abstract syntax tree and produces chunks that correspond to complete functions or classes, including their signatures, docstrings, and bodies. Splitting code at arbitrary line numbers produces chunks that contain half a function, which is useless for both retrieval and generation.

For code documentation that mixes prose and code blocks, the chunker should keep each code block with its surrounding explanation as a unit, even if this produces larger-than-average chunks. The alternative, splitting the explanation from the code block it references, produces chunks where neither the explanation nor the code is useful in isolation.

Conversational Data

Chat logs, support tickets, and meeting transcripts have a natural turn-based structure. Chunking by topic (using semantic chunking) or by conversation turn groups (every 5 to 10 turns) generally works better than fixed-size chunking. Each chunk should include enough conversation context for the exchange to make sense, which usually means including at least the initial question and the response that follows it as a unit.

Chunk Size Recommendations by Document Type

Based on benchmarks across multiple production RAG systems, these chunk sizes consistently perform well:

Technical documentation and knowledge bases: 512 to 768 tokens with section-based or recursive chunking. These documents are well-structured, and sections are naturally sized for retrieval.

Legal and regulatory documents: 768 to 1024 tokens with section-based chunking that preserves clause hierarchy. Legal language is dense and context-dependent, so larger chunks preserve the necessary context.

FAQ and Q&A content: 256 to 512 tokens with each Q&A pair as an atomic chunk. Questions and answers are self-contained units that should not be split.

Research papers and long-form articles: 512 to 768 tokens with section-based chunking. Split at section boundaries (Abstract, Introduction, Methods, Results) and sub-section boundaries within each section.

Chat and conversation logs: 512 tokens with semantic chunking or turn-group chunking. Include enough turns for context but not so many that topics blur together.

Code repositories: Function or class-level chunking regardless of token count, with a maximum of 1500 tokens per chunk. Very large functions should be split at logical sub-blocks.

Measuring Chunking Quality

The only reliable way to evaluate a chunking strategy is to measure its impact on retrieval quality using a labeled evaluation set. Create a set of 50 to 100 question-answer pairs where you know which documents (and which specific passages within those documents) contain the answer. Run the retrieval step with your current chunking strategy and measure recall@5 and recall@10: what fraction of the relevant passages appear in the top 5 and top 10 results. Then change the chunking strategy and measure again. The strategy that produces higher recall without dramatically increasing the number of chunks (which increases storage and search costs) is the better strategy.

You can also measure chunk coherence directly by sampling 50 random chunks and rating each on a 1 to 5 scale: does this chunk contain a single coherent idea (5) or is it a fragment of multiple unrelated ideas (1)? Average coherence scores above 4 indicate good chunking. Scores below 3 indicate that chunks are being split at wrong boundaries and the strategy needs adjustment.