Incremental Summarization: Updating Summaries Without Reprocessing

Step 1: Detect What Changed

Before updating a summary, determine whether an update is needed and what the new content contributes that is not already captured.

Novelty check: Compare the new content against the existing summary. Send both to the model: "Does the following new content contain any information that is NOT already captured in the existing summary? Answer YES or NO. If YES, list the new information." If the answer is NO, skip the update entirely. This prevents unnecessary API calls when new content repeats previously summarized information (common in customer support where users restate their issue across interactions).

Significance threshold: Not every piece of new information warrants a summary update. Minor details ("user confirmed their email address") do not belong in a high-level interaction summary. Set a significance threshold: "Only flag information as requiring a summary update if it would change a reader's understanding of the customer's situation, preferences, or history." This prevents summary bloat from trivial additions.

Contradiction detection: New content may contradict the existing summary. If the summary says "customer prefers email communication" and the new interaction says "please stop emailing me, I want SMS only," the summary needs correction, not just addition. Instruct the change detection step to flag contradictions explicitly: "Does the new content contradict or update any claim in the existing summary? If so, list the contradictions."

Automated trigger vs continuous: You can run change detection on every new content addition (continuous) or batch new content and run detection periodically (scheduled). Continuous detection costs more but keeps summaries perpetually fresh. Scheduled detection (every 24 hours, every 10 new items) is cheaper and works well for applications where real-time freshness is not critical.

Step 2: Generate a Delta Summary

Once you have identified new information that warrants an update, summarize only the new content. This is the "delta" that will be merged into the existing summary.

Delta prompt: "The following is new content that has been added since the last summary update. Summarize the new information in 2-5 sentences, focusing on: (1) New facts or events not previously captured. (2) Changes to previously recorded information. (3) New preferences, decisions, or outcomes expressed. Do not repeat information already covered in the existing summary."

Context-aware delta: Provide the existing summary as context when generating the delta. This helps the model understand what is already captured and produce a delta that complements rather than duplicates: "Existing summary: [current summary]. New content: [new text]. Generate a summary of ONLY the new information that is not already in the existing summary."

Delta length: Keep delta summaries short, typically 50-200 words depending on the volume of new content. The delta should capture the essence of what changed, not provide a comprehensive summary of the new content in isolation. Longer deltas make the merge step harder and more prone to redundancy.

Step 3: Merge Delta into Existing Summary

The merge step combines the existing summary with the delta to produce an updated summary. This is the most nuanced step because it must resolve contradictions, eliminate redundancy, and maintain coherence.

Merge prompt: "You have an existing summary and a delta containing new information. Produce an updated summary that: (1) Incorporates all new information from the delta. (2) Replaces any information in the existing summary that the delta contradicts or updates. (3) Removes redundancy where the delta overlaps with existing content. (4) Maintains the same approximate length as the existing summary (you may grow by up to 20% if the new information is substantial). (5) Reads as a coherent, self-contained summary, not as an original plus amendments."

Length control: Without explicit length control, incremental updates cause summaries to grow monotonically. Each merge adds new information but rarely removes old information that has become less relevant. Set a target length and instruct the merge step to compress older, less relevant information to make room for new additions. "Target length: 300 words. If including the new information would exceed this length, compress or remove the least important details from the existing content to stay within budget."

Versioning: Keep the previous summary version before overwriting with the merged version. If the merge produces a poor result (coherence drop, hallucination introduction), you can roll back to the prior version and retry. A simple version history (current, previous, two-versions-ago) costs negligible storage and provides essential safety.

Step 4: Schedule Periodic Full Re-Summarization

Incremental updates accumulate small errors over time. Each merge step makes subtle decisions about what to keep and compress, and after 10-20 merges, the summary may have drifted from what a fresh full-pass summarization would produce. The original source's emphases may have shifted, early content may have been over-compressed, and the narrative structure may have become disjointed from repeated patch-and-merge operations.

Full refresh schedule: After every 10-20 incremental updates, perform a full re-summarization from all source content. This resets the summary to a clean, coherent baseline. Compare the full re-summarization against the incrementally maintained version to measure drift: if they diverge by more than 20% (measured by cosine similarity of their embeddings or by LLM-as-judge comparison), the incremental updates are introducing too much drift and the refresh frequency should increase.

Hybrid schedule: The optimal pattern for most applications is: incremental updates for every new content addition (keeping the summary fresh in real time) with a full refresh once per week or once per 15 updates (resetting accumulated drift). The full refresh catches information that was incorrectly compressed, fixes narrative coherence, and re-balances emphasis across the entire source.

Cost comparison: For a customer with 4 interactions per week and a 20-interaction history, incremental summarization costs roughly $0.01 per week (4 delta + merge operations at $0.0025 each). A weekly full refresh costs roughly $0.05 (processing all 20 interactions). Combined cost: $0.06 per customer per week. Full re-summarization on every interaction would cost $0.20 per week (4 full passes). The hybrid approach costs 70% less while maintaining summary quality.