How to Summarize Documents That Exceed the Context Window
The core challenge is not just fitting text into a window, it is preserving cross-document relationships and global themes that span many pages. A conclusion that references findings from the introduction requires the summarization system to somehow connect information from the first and last chunks despite processing them independently. The strategies below each handle this challenge differently.
Step 1: Measure the Document and Choose Your Strategy
Before chunking, count the document's total tokens using a tokenizer appropriate for your model (tiktoken for OpenAI models, Anthropic's tokenizer for Claude). Compare this against your available context budget: total context window minus system prompt tokens minus expected output tokens minus safety margin (typically 500-1,000 tokens).
If the document fits within your available budget, skip chunking entirely and summarize directly. This produces the highest quality summary because the model sees all context simultaneously. Claude's 200K window handles documents up to approximately 150,000 tokens of source (reserving space for prompt and output), which covers most single documents, reports, and papers.
If the document exceeds your budget by 2-5x, a single round of map-reduce works well. Chunk the document into 3-8 pieces, summarize each, and combine. For documents exceeding your budget by 5-20x, hierarchical map-reduce with multiple reduction layers produces better results. For documents exceeding 20x (book-length), consider whether a full summary is the right approach or whether topic-focused extraction serves the use case better.
Strategy selection also depends on your quality requirements. If the summary must preserve specific details (names, numbers, dates), use smaller chunks with more overlap. If you need only high-level themes, larger chunks with aggressive compression per chunk work fine.
Step 2: Chunk the Document Intelligently
Naive chunking splits at fixed token counts regardless of content boundaries. This frequently cuts sentences in half, separates a topic introduction from its explanation, or splits a data table from its caption. Intelligent chunking respects the document's structure.
Paragraph-boundary chunking: Split only at paragraph boundaries, accumulating paragraphs until reaching 80% of your target chunk size, then starting a new chunk. This ensures no sentence is split and each chunk contains complete thoughts. Target chunk size of 4,000-8,000 tokens balances between giving the model enough context per chunk and keeping individual summarization calls fast and cheap.
Section-boundary chunking: For structured documents with headings (reports, papers, documentation), split at section boundaries. Each section becomes its own chunk regardless of length, unless a section itself exceeds the context window (in which case, sub-split at paragraph boundaries within that section). This preserves the author's logical organization and produces chunk summaries that correspond to coherent topics.
Semantic chunking: Use embedding similarity to detect topic shifts. Compute embeddings for each paragraph, identify points where consecutive paragraph embeddings diverge significantly (cosine similarity drops below a threshold, typically 0.7-0.8), and split there. This finds natural topic boundaries even in unstructured text like transcripts or email threads that lack explicit headings.
Overlapping chunks: Add 10-20% overlap between adjacent chunks. If chunk 1 ends at token 4,000, chunk 2 starts at token 3,600 rather than 4,001. The overlapping region appears in both chunk summaries, giving the reduce step redundant information about boundary content. This mitigates the primary weakness of chunked summarization: information loss at boundaries where ideas span two chunks.
Step 3: Summarize Each Chunk Independently
Process each chunk through your summarization prompt. Use a consistent prompt across all chunks to ensure uniform output format and compression ratio. A typical chunk summarization prompt:
"Summarize the following passage in 150-300 words. Preserve all specific facts, names, numbers, and conclusions. Maintain the logical flow of ideas. This is one section of a larger document that will be combined with other section summaries."
The last sentence is important: it tells the model that the output will be further processed rather than serving as a standalone summary. This encourages the model to preserve details that might seem minor in isolation but matter for the full document summary.
For the map step, parallelization is straightforward because chunks are independent. Send all chunk summarization requests concurrently to maximize throughput. A 20-chunk document processed sequentially takes 40-100 seconds (2-5 seconds per chunk). Processed in parallel, it completes in 3-7 seconds total. Most LLM APIs support concurrent requests up to your rate limit.
Target compression ratio per chunk: 10-20% of chunk length. If your chunks are 5,000 tokens each, aim for 500-1,000 token summaries per chunk. This keeps chunk summaries detailed enough for the reduce step to work with while achieving meaningful compression. Over-compressing at the map step (targeting 50-word summaries of 5,000-token chunks) loses too much detail for the reduce step to produce a coherent final summary.
Step 4: Combine Chunk Summaries into the Final Output
The reduce step takes all chunk summaries and produces one coherent final summary. If your combined chunk summaries fit within the context window (they usually do, since each was compressed to 10-20% of original), a single reduce call works.
The reduce prompt should specify:
"The following are summaries of consecutive sections of a single document. Synthesize them into one coherent summary of [target length]. Eliminate redundancy where sections covered the same topic. Maintain chronological or logical order. Identify and highlight the document's main conclusions and key findings."
If chunk summaries combined still exceed your context window (happens with very long documents), apply hierarchical reduction. Group chunk summaries into batches of 5-8, reduce each batch into a meta-summary, then reduce the meta-summaries into the final output. Each reduction layer compresses by 70-80%, so two layers of reduction handle documents up to 100x your context window, and three layers handle effectively unlimited length.
The reduce step is where you specify the final output format. If you want bullet points, section headers, or a specific structure, instruct it at this stage rather than during the map step. The map step should preserve information; the reduce step should organize and format it.
Step 5: Verify and Validate the Result
Long-document summarization has higher hallucination risk than single-context summarization because the reduce step works from chunk summaries rather than source text. Verify the final output by spot-checking key claims against the source document.
Automated verification approaches: run an entailment check between the final summary and a random sample of source chunks (does the source support each claim in the summary?). Track coverage by checking whether the final summary mentions the main topic of each chunk (each chunk should contribute at least one point to the final summary). Flag summaries where coverage drops below 70% of chunks represented.
For critical documents, add a verification pass: present the final summary alongside a sample of source chunks to an LLM and ask "Does this summary contain any claims not supported by the source material?" This catch-all verification adds one API call but catches hallucinations that systematic checks miss.
Length validation ensures the output meets your target. If you specified 500 words and got 1,200, the model ignored your constraint. Re-run with stronger length enforcement or post-process by asking the model to compress the over-long summary to the target length. Under-length summaries (200 words when you asked for 500) usually indicate that the reduce step over-compressed, which means your chunk summaries may have been too sparse.
Choosing Between Map-Reduce and Refine
The refine pattern is an alternative to map-reduce that processes chunks sequentially, updating a running summary with each new chunk. It starts with a summary of chunk 1, then presents chunk 2 alongside the current summary and asks for an updated summary, repeating until all chunks are processed.
Refine produces better continuity because each step sees the accumulated context from all previous chunks. Cross-chunk references survive because the running summary carries them forward. However, refine cannot be parallelized (each step depends on the previous output), making it 5-10x slower than parallel map-reduce for the same document. Refine also suffers from recency bias: later chunks disproportionately influence the final summary because the model processes them most recently and has the most "room" to adjust the summary.
Use map-reduce when: latency matters, the document has independent sections (each chapter stands alone), or the document is very long (more than 10 chunks). Use refine when: the document has a strong narrative thread that builds across sections, cross-references between early and late content are critical, and the total chunk count is small enough (under 8) that sequential processing remains acceptable.
Handling Common Edge Cases
Tables and structured data: Tables compress poorly because every cell may contain important information. Pre-process tables into natural language descriptions ("Q1 revenue was $4.2M, up 12% from Q4") before chunking. Alternatively, extract tables separately and include them as a dedicated chunk that the reduce step can reference.
Documents with appendices: Appendices often contain supporting data referenced in the main text. Decide upfront whether appendices should be included in summarization or excluded. If included, process them as separate chunks with a flag indicating they are supplementary material. The reduce prompt can then give them lower priority.
Multi-language documents: Documents mixing languages (common in international reports) should be chunked by language when possible, summarized in the original language per chunk, then combined with a reduce prompt specifying the desired output language. Cross-language summarization in a single step works but produces slightly lower quality than same-language summarization followed by translation.
For documents exceeding the context window, split at natural boundaries with 10-20% overlap, summarize chunks in parallel targeting 10-20% compression each, then synthesize chunk summaries in a reduce step. This map-reduce pattern handles documents of arbitrary length while keeping each processing step within token limits.