How to Summarize Conversation History for AI Memory
Without conversation summarization, AI assistants face a hard choice when the context window fills: either drop old messages entirely (causing the assistant to "forget" earlier context) or stop accepting new messages. Neither is acceptable for production applications. Summarization provides the third option: compress older history into a fraction of its original token count while preserving the information the model needs to continue the conversation coherently.
Step 1: Define Your Summarization Trigger
The trigger determines when the system generates a new summary. Three common approaches:
Token threshold trigger: Summarize when total conversation history exceeds a set token count. For example, when history reaches 8,000 tokens, summarize the oldest 6,000 tokens into approximately 800 tokens, reducing total history back to roughly 2,800 tokens. This is the most common approach because it directly manages the resource you are constrained by (tokens). Set the threshold based on your context budget: if your total context window is 128,000 tokens and you reserve 100,000 for system prompt and retrieved documents, your conversation history budget is 28,000 tokens. Trigger summarization when history reaches 70-80% of that budget.
Message count trigger: Summarize every N messages regardless of token count. This is simpler to implement but less precise because message length varies wildly. A trigger every 20 messages works for typical consumer chatbots where messages average 50-100 tokens each. For developer tools where messages may include code blocks (500-2,000 tokens each), a lower threshold of 8-10 messages is more appropriate.
Session boundary trigger: Summarize at the end of each conversation session. When a user returns after a gap (typically 30+ minutes of inactivity), the previous session's raw messages get summarized and stored as long-term memory. The new session starts with only the summary of previous sessions plus any persistent user profile data. This naturally separates "working memory" (current session, full detail) from "long-term memory" (past sessions, summarized).
Step 2: Classify Message Importance
Not all messages carry equal information value. "Thanks!" and "Sounds good" contain zero information that a summary needs to preserve. "I use PostgreSQL 16 with pgvector on AWS RDS" contains a technical preference that matters for every future recommendation. Importance classification determines what the summary emphasizes and what it discards.
High-importance signals: user states a preference, constraint, or requirement. User provides personal or professional context (role, company, tech stack). User corrects the assistant. User makes a decision between options the assistant presented. User provides feedback on quality ("that is not what I meant"). The assistant provides information the user requested (facts the user may reference later).
Low-importance signals: greetings and closings. Acknowledgments without new information ("ok," "got it," "thanks"). The assistant asking clarifying questions (the answer matters, the question itself does not). Repeated information already captured in previous summaries. Formatting requests ("make it shorter," "add bullet points") that do not affect future interactions.
Implementation: you can classify importance with a fast LLM call (Haiku-class model, prompt: "Rate each message 1-5 for information density relevant to future conversations") or with heuristics (messages under 10 tokens are low-importance, messages containing personal pronouns with factual verbs are high-importance). The classification does not need to be perfect, it just needs to bias the summary toward preserving the right content.
Step 3: Generate the Rolling Summary
The rolling summary pattern maintains a single, continuously updated summary document that grows in quality (not necessarily in length) with each summarization cycle. When new messages accumulate past your trigger threshold, you pass the current summary plus the new messages to the LLM and ask for an updated summary.
The prompt structure for rolling summarization:
"Below is the existing conversation summary, followed by new messages since the last summary update. Generate an updated summary that incorporates important new information from the recent messages. Preserve all user preferences, technical details, decisions made, and unresolved questions from the existing summary. Compress or remove information about greetings, pleasantries, and resolved logistical details. Keep the updated summary under [target length] tokens."
Then provide: [existing summary] followed by [new messages since last summary].
Critical implementation detail: always include the existing summary in the summarization call. This ensures continuity. Without it, each summarization cycle only sees the most recent messages and loses accumulated context from earlier in the conversation. The existing summary carries forward everything important from the entire conversation history, and the new messages add to or update that accumulated knowledge.
Target summary length depends on your token budget and conversation complexity. For simple customer support conversations: 200-400 tokens captures the issue, resolution attempts, and user details. For complex multi-session assistants: 500-1,500 tokens preserves the full user profile, project context, and ongoing task state. For developer tools with technical context: 800-2,000 tokens retains code preferences, architecture decisions, and active debugging context.
Step 4: Store and Retrieve Summaries
Summaries need persistent storage associated with the user or conversation they belong to. The storage design depends on your memory architecture:
Single-summary model: One rolling summary per user, updated with each trigger. Simplest to implement. Works well for applications where conversations are largely independent and context from any past conversation is equally relevant. Store in a key-value store keyed by user ID.
Session-based model: One summary per conversation session, accumulating over time. When the user starts a new session, retrieve the most recent 3-5 session summaries to include in context. This preserves temporal structure, letting you show the model "last week you discussed X, yesterday you discussed Y." Store in a table with user ID and timestamp as keys.
Topic-based model: Multiple summaries per user, one per major topic or project. A developer assistant might maintain separate summaries for "Authentication Project," "Database Migration," and "CI/CD Pipeline." When the current conversation touches a topic, retrieve the relevant topic summary. This requires topic detection (which summary to update and which to retrieve) but provides the most relevant context injection. Store summaries with user ID and topic tag as composite keys.
Regardless of model, store metadata alongside each summary: creation timestamp, message count covered, session IDs included, and a list of key entities mentioned (for retrieval filtering). This metadata enables intelligent retrieval when the user has hundreds of stored summaries.
Step 5: Inject Summaries into Context
How you inject the summary into the model's context affects how the model uses it. Three injection strategies:
System prompt injection: Place the summary in the system prompt with a header like "Conversation context from previous interactions:" This positions the summary as background knowledge the model should be aware of but not explicitly reference unless relevant. Good for persistent user profiles and general preferences.
Conversation history injection: Insert the summary as a special message at the beginning of the conversation history, formatted as "Summary of earlier conversation: [summary]." The model treats this as prior conversational context and can reference it naturally ("As you mentioned earlier..."). Good for continuity within extended conversations.
Retrieval-augmented injection: Only inject summaries that are relevant to the current query, identified via embedding similarity between the current message and stored summaries. This prevents irrelevant historical context from consuming tokens. Good for users with extensive history across many topics where only a fraction is relevant at any time.
In practice, many systems combine approaches: a short user profile in the system prompt (always present), recent session summaries in the conversation history (always present), and topic-specific summaries retrieved on demand (present only when relevant).
Handling the Summarization Transition Gracefully
Users should never notice when summarization occurs. The conversation should feel continuous. Common failure modes to avoid:
Context discontinuity: The assistant suddenly cannot recall something discussed 10 messages ago because it was over-compressed in the summary. Mitigation: keep the most recent 5-8 messages in raw form even when the rest is summarized. This creates a "working memory" buffer where recent context is always fully available.
Summary staleness: The summary says "User is working on a Python project" but 3 messages ago the user switched to discussing a Rust project. This happens when summarization triggers are too infrequent. Mitigation: trigger summarization frequently enough that the summary never lags more than 5-8 messages behind reality. For fast-moving conversations, trigger every 10 messages rather than waiting for a token threshold.
Repetitive acknowledgment: The assistant keeps re-introducing information from the summary ("As I recall from our earlier conversation...") in every response. This feels unnatural. Mitigation: instruct the model to use summary context implicitly rather than explicitly referencing it. The model should know the user prefers Python without saying "Based on our earlier conversation where you mentioned preferring Python."
Measuring Conversation Summarization Quality
Track these metrics to ensure your summarization maintains conversation quality:
Context retrieval accuracy: When a user references something from earlier in the conversation, can the model respond correctly? Test by asking questions about information from before the summarization boundary. Target: 90%+ correct responses about summarized content.
User repetition rate: How often do users re-explain something the assistant should already know? High repetition indicates the summary is not preserving important context. Measure by detecting user messages that contain information already present in the conversation history.
Summary token efficiency: How much useful information does the summary pack per token? Measure by counting distinct facts in the summary and dividing by summary length. A 200-token summary containing 15 distinct user facts is more efficient than a 500-token summary containing 10 facts.
Conversation summarization is a rolling compression process that should fire frequently (every 10-20 messages or when history exceeds 70% of your token budget), always merge new messages with the existing summary for continuity, and preserve user preferences and decisions with much higher priority than greetings and logistics.