Real-Time Summarization for Streaming and Live Conversations

How Real-Time Differs from Batch Summarization

Batch summarization sees the complete input before generating output. It knows the full context: how the story ends, what the conclusion is, which early details turn out to matter and which do not. Real-time summarization operates under uncertainty. When you summarize the first 10 minutes of a meeting, you do not know what the next 50 minutes will contain. A topic mentioned briefly early may become the central focus later, or may never be mentioned again. A preliminary decision stated at minute 8 may be reversed at minute 45.

This uncertainty creates specific challenges that batch summarization never faces. The summary must be simultaneously useful right now (reflecting current understanding) and updateable later (able to incorporate new information without complete regeneration). It must handle retraction gracefully (removing or correcting information that was superseded by later content). And it must maintain coherence through dozens of incremental updates without degrading into a disjointed collection of fragments.

Latency requirements compound the challenge. In a live meeting scenario, participants want updated notes within seconds of a topic concluding, not minutes later after batch processing. This means the summarization system must process new content in near-real-time, which constrains model choice and prompt complexity. You cannot run a 30-second summarization call every time a new sentence arrives, so the system must batch incoming content into appropriate update intervals.

The Incremental Update Pattern

The foundation of real-time summarization is the incremental update: maintain a running summary and periodically merge new content into it. This pattern appears across all real-time summarization applications with minor variations.

The cycle works as follows: (1) New content accumulates in a buffer since the last update. (2) When the buffer reaches a trigger threshold (time-based, token-based, or event-based), the system initiates an update cycle. (3) The update call receives the current running summary plus the buffered new content. (4) The model produces an updated summary that integrates the new information. (5) The updated summary replaces the previous version. (6) The buffer clears, and the cycle repeats.

The trigger threshold determines the update frequency, which directly trades off between freshness and cost. A 30-second trigger in a live meeting produces nearly-instant notes but generates 120 API calls per hour. A 5-minute trigger reduces calls to 12 per hour but means notes lag behind reality by up to 5 minutes. Most production systems use adaptive triggers: update frequently during high-activity periods (when content is arriving rapidly) and less frequently during low-activity periods (pauses, small talk, repetitive content).

The update prompt must handle three operations simultaneously: adding new information from the buffer, maintaining all relevant information from the existing summary, and removing or updating information from the existing summary that the new content supersedes. A prompt template: "Below is the current summary of this conversation, followed by new content since the last update. Produce an updated summary that: (1) preserves all important information from the existing summary that remains valid, (2) integrates new facts, decisions, and context from the recent content, (3) updates or removes any information from the existing summary that the new content corrects or supersedes. Keep the updated summary under [token limit]."

Architectures for Different Real-Time Scenarios

Live Meeting Notes

The live meeting use case requires: transcription streaming in near-real-time (via Whisper, AssemblyAI, or similar), summarization that produces readable notes within 15-30 seconds of content being spoken, output visible to participants during the meeting for reference and correction, and a final polished summary at meeting end that incorporates any corrections made during the meeting.

Architecture: speech-to-text streams chunks of transcript (typically 15-30 seconds of audio at a time) into a processing queue. A summarization worker watches the queue and triggers an update when either 60 seconds of new transcript accumulates or a topic change is detected (significant silence followed by a new speaker, or explicit topic transition phrases like "moving on to"). The update merges new transcript content with the running structured notes (decisions, action items, discussion points).

The output format for live notes differs from post-meeting summaries. During the meeting, participants need to see the latest topic being discussed at the top (reverse chronological within the current meeting). After the meeting, the same content reorganizes into a structured format (decisions first, then action items, then topic summaries). The real-time system maintains both views: an append-only live feed and a periodically restructured summary.

Correction handling: when a participant corrects the notes during the meeting ("that is not what I said, the deadline is Friday, not Thursday"), the system must propagate the correction into the running summary immediately. Store corrections separately and inject them into the next update cycle with high priority: "The following correction was explicitly made by a participant and must override any conflicting information in the current summary: [correction]."

Streaming Chat Compression

AI assistants with long conversations need to compress their own history in real-time to stay within context window limits. Unlike meeting notes (where the summary is an output for humans), chat compression produces a summary that feeds back into the system's own context, making it an internal infrastructure component.

Architecture: the system maintains a conversation buffer containing recent messages in full detail and a compressed summary of older messages. When the buffer grows past a threshold (typically 60-70% of the allocated conversation history budget), the oldest portion of the buffer gets compressed into the running summary. The most recent messages always remain in full detail to preserve immediate conversational context.

The compression boundary moves forward over time: at the start of a conversation, everything is in the full-detail buffer. After 20 messages, the first 12 messages get compressed into a summary while the most recent 8 remain in full detail. After 40 messages, the first 32 have been compressed (through multiple incremental updates to the summary) while the most recent 8 remain in full detail. The user experiences seamless continuity because the assistant can reference both the detailed recent messages and the summarized earlier context.

Quality requirement: the compression must preserve information the assistant needs to maintain coherent conversation. This means aggressively preserving user preferences, stated constraints, technical details, and decisions while aggressively compressing pleasantries, acknowledgments, and the assistant's own explanations. The prompt should explicitly instruct: "Preserve what the user said and decided. Compress what the assistant explained."

News and Event Monitoring

Monitoring systems track developing stories across multiple sources, producing evolving summaries that update as new articles, social media posts, or data points arrive. Unlike meeting summarization where content is a single stream, monitoring aggregates multiple concurrent streams about the same topic.

Architecture: a topic detection layer identifies incoming content related to tracked stories. A deduplication layer filters content that adds no new information beyond what the current summary already captures (often 60-80% of incoming content about a popular story is redundant). Content that passes deduplication triggers a summary update. The update incorporates the new information, adjusts any facts that the new source corrects, and flags conflicting reports from different sources.

Timeline awareness is critical for monitoring: the summary must reflect not just what is known but when it became known. "At 2:00 PM, initial reports indicated X. By 3:30 PM, officials confirmed Y, correcting earlier reports." This temporal structure helps readers understand how the story developed rather than just its current state. The update prompt should instruct: "Maintain chronological structure. When new information corrects earlier reports, note both the correction and when it was established."

Managing Summary Drift and Degradation

After many incremental updates, running summaries tend to drift in predictable ways. Understanding these drift patterns lets you design countermeasures.

Information decay: Facts mentioned early in the stream gradually lose representation in the summary as newer content pushes them out. Each update slightly favors new information because the prompt focuses on "integrating new content," and the model interprets this as permission to compress older content more aggressively. After 20-30 updates, facts from the first few updates may have vanished entirely.

Countermeasure: periodically anchor important early facts. Every 10 updates, append a "preserved facts" section to the update prompt: "The following facts must remain in the summary regardless of what new content arrives: [list of permanently important facts extracted from early in the stream]." This list grows slowly over time as new permanently-important facts emerge, but it prevents the most critical early information from decaying.

Length creep: Despite length constraints in the prompt, summaries tend to grow over many updates because each cycle adds new information while removing old information imperfectly. After 30 updates, a summary targeted at 500 tokens may have crept to 800. The model adds complete new points but only partially compresses old ones.

Countermeasure: enforce length strictly with a two-pass approach. After generating the updated summary, check its token count. If it exceeds the budget by more than 10%, run a compression pass: "The following summary exceeds its target length of [N] tokens. Compress it to fit within [N] tokens while preserving the most important information." This adds latency (two API calls instead of one) but should only trigger occasionally, perhaps once every 5-10 updates.

Coherence degradation: After many incremental patches, the summary reads like it was assembled from fragments rather than written as a coherent whole. Transitions between topics are abrupt, information is organized by when it arrived rather than by logical grouping, and the writing style varies between sections (reflecting different update cycles).

Countermeasure: periodically regenerate the summary from scratch rather than incrementally updating it. Every 15-20 update cycles, take the current summary and the most recent buffer content and ask the model to "rewrite this summary from scratch as a coherent, well-organized document" rather than "update this summary with new content." This regeneration pass reorganizes the content logically, smooths transitions, and normalizes writing style. It costs one extra API call but dramatically improves readability.

Latency Optimization

Real-time summarization has a latency budget: the time between content arriving and the updated summary being available. For live meeting notes, this budget is 5-15 seconds. For chat compression, it can be longer (30-60 seconds is fine because the system can use the old summary while the new one generates). For monitoring, minutes are acceptable.

Model selection for latency: Use the fastest model that produces acceptable quality. For incremental updates (where the model merges a short buffer into an existing summary), the task is relatively simple and does not require frontier model intelligence. Claude Haiku, GPT-4o-mini, or Gemini Flash handle updates well at 3-5x lower latency than their larger siblings. Reserve frontier models for the periodic regeneration passes where quality matters more than speed.

Streaming output: When the summary is displayed to users in real-time (live meeting notes), stream the model's output token-by-token to the display. Users see the updated summary building in real-time rather than waiting for the complete generation to finish before anything appears. This reduces perceived latency by 60-80% even though actual completion time is unchanged.

Speculative pre-processing: In predictable scenarios (scheduled meetings that start at known times, news monitoring of anticipated events), pre-warm the system by loading the existing summary into a pre-built prompt template. When new content arrives, the system only needs to append the new content and send the already-constructed prompt, eliminating prompt assembly latency.

Parallel update paths: For systems with multiple summary types (a detailed summary plus a brief headline summary, or a full summary plus extracted action items), run update calls in parallel. Both receive the same new buffer content and the existing version of their respective outputs. Parallel calls complete in the time of the slowest call rather than the sum of both.

Handling Incomplete Context Gracefully

Real-time summarization always works with incomplete information. The meeting is not over. The story is still developing. The conversation is ongoing. The summary must communicate this incompleteness to readers without undermining its usefulness.

Use explicit temporal framing: "As of [timestamp], the meeting has covered..." or "Based on discussion so far..." These phrases signal that the summary reflects current state, not final conclusions. Avoid definitive language ("The team decided to...") for topics still under active discussion. Instead use progressive language ("The team is leaning toward..." or "Discussion is ongoing about...").

Track confidence levels for different pieces of information. A decision explicitly stated ("we are going with option B") has high confidence and can be stated definitively. A preference implied from discussion tone ("most comments favored option B but no formal decision was made") should be stated tentatively. The update prompt can include: "Mark information as definitive only when it was explicitly stated as a conclusion or decision. Mark information as tentative when it is inferred from discussion but not formally concluded."

Plan for retraction. In developing stories and ongoing meetings, early information frequently gets revised. Design the summary format to accommodate updates gracefully. Avoid embedding provisional facts deeply into the narrative structure where they become hard to extract. Instead, present facts as discrete items that can be individually updated. "Key decisions: (1) Launch date: March 15 [confirmed at 2:30 PM]. (2) Budget: $500K [preliminary, pending finance review]." This structure lets individual items be updated without rewriting surrounding context.