How to Build Conversational RAG with Chat History

The Follow-Up Problem

Standard RAG pipelines are stateless. Each query arrives, gets embedded, retrieves relevant chunks, generates an answer, and the pipeline forgets everything. This works for isolated questions but fails immediately in dialogue. Consider this conversation:

User: "What authentication methods does the API support?"
System: "The API supports OAuth 2.0, API keys, and JWT tokens..."
User: "Which one is best for mobile apps?"
System: ???

The follow-up "which one is best for mobile apps?" is ambiguous without the prior turn. If this query is sent to vector search as-is, the retrieval step searches for chunks about mobile apps and "which one," which is too vague to retrieve anything useful. The system needs to understand that "which one" refers to "authentication methods" and that "best for mobile apps" means the user wants a recommendation among OAuth 2.0, API keys, and JWT tokens in a mobile context.

This is not an edge case. Studies of conversational search systems show that 40% to 60% of user queries in multi-turn sessions contain pronouns, ellipsis, or implicit references that require prior context to resolve. Users naturally speak in context because that is how human conversation works. A RAG system that forces users to repeat themselves with fully specified questions in every turn delivers a frustrating experience.

Step 1: Add Query Rewriting

Query rewriting is the most impactful addition to a conversational RAG pipeline. A lightweight LLM takes the conversation history and the latest follow-up query as input and produces a standalone query that captures the full intent without requiring any prior context to understand.

The rewriting prompt looks like this: "Given the conversation history below, rewrite the user's latest question as a standalone query that can be understood without any prior context. Include all relevant details from the conversation that the question refers to. If the question is already standalone, return it unchanged."

For the example above, the rewriter transforms "Which one is best for mobile apps?" into "Which authentication method (OAuth 2.0, API keys, or JWT tokens) is best for mobile applications?" This standalone query retrieves relevant chunks about mobile authentication patterns, which is exactly what the user needs.

The rewriting step adds 100 to 300 milliseconds using a fast model (GPT-4o mini or Claude 3.5 Haiku) and costs $0.0001 to $0.001 per query. This is negligible compared to the generation step and produces a dramatic improvement in retrieval quality for follow-up questions. Without rewriting, follow-up retrieval recall drops by 30% to 50% compared to standalone queries. With rewriting, the gap closes to under 5%.

The rewriter should also handle query expansion when the conversation establishes domain context. If the user spent three turns discussing Kubernetes networking, a follow-up "what about load balancing?" should be rewritten as "How does load balancing work in Kubernetes networking?" rather than a generic load balancing query. The conversation history provides the domain scope that the rewriter should preserve.

Step 2: Manage Conversation History in the Prompt

The LLM that generates the final answer needs to see both the retrieved chunks and the conversation history. The conversation history provides continuity, so the model can reference its prior answers, avoid repeating information, and maintain a coherent dialogue thread. The prompt structure for conversational RAG is: system instruction, retrieved context with source labels, conversation history (last N turns), and the current user question.

The critical decision is how many turns of history to include. Too few turns and the model loses context for ongoing discussions. Too many turns and the conversation history consumes context window tokens that could be used for retrieved chunks, which are the actual knowledge source. The practical sweet spot for most applications is the last 3 to 5 turns (6 to 10 messages, counting both user and assistant messages). This covers the immediate conversational context without overwhelming the prompt.

Token budgeting becomes essential. A typical conversational RAG prompt has four components competing for context window space: system instruction (200 to 500 tokens, fixed), retrieved chunks (1,500 to 3,000 tokens, variable), conversation history (500 to 2,000 tokens, growing), and the current question plus generated response (500 to 1,500 tokens). On a model with a 128K context window, there is plenty of room. On a model with an 8K or 16K window, or when cost optimization requires keeping prompts small, the conversation history must be aggressively trimmed.

Three strategies for managing history length:

Sliding window: Keep the last N turns and discard everything older. Simple and predictable, but loses important context from the beginning of long conversations. Works well for short sessions (under 10 turns).

Summarization: When the conversation exceeds a threshold (8 to 10 turns), summarize the older turns into a 2 to 3 sentence summary and keep only the last 3 to 4 turns in full. The summary preserves key context (what the user asked about, what was discussed, what conclusions were reached) without consuming proportional tokens. This is the most common approach in production systems.

Memory-augmented: Extract key facts and preferences from the conversation and store them in a memory system that persists across sessions. The prompt includes a compact memory context (50 to 100 tokens of key facts) instead of raw conversation history. This is the most token-efficient approach and also enables cross-session continuity, where the system remembers what the user discussed yesterday.

Step 3: Handle Topic Switches

Not every follow-up question continues the current topic. Users switch topics mid-conversation: "Thanks for explaining the API authentication. By the way, what are your pricing tiers?" Carrying the authentication context forward into a pricing query wastes retrieval bandwidth and can confuse the LLM if authentication chunks appear alongside pricing chunks in the prompt.

Topic switch detection uses one of two approaches. The simple approach compares the embedding similarity between the current query and the previous query (or the rewritten version). A similarity score below a threshold (typically 0.3 to 0.4 on cosine similarity) signals a topic switch. When detected, the retrieval step searches without the conversational context, treating the new query as standalone. The conversation history is still included in the prompt for continuity, but the rewriter does not incorporate prior topics into the new query.

The LLM-based approach adds a classification step to the rewriting prompt: "If the user's latest question is about a new topic unrelated to the prior conversation, indicate this as a topic switch and rewrite it as a standalone query without incorporating prior context." This is more accurate than similarity thresholds because the LLM understands when "by the way" signals a digression and when "what about" is a follow-up to the current topic.

When a topic switch is detected, some systems reset the conversation history window, starting fresh for the new topic while archiving the old conversation thread. Others maintain the full history but tag the switch point so the LLM knows the context shifted. The right choice depends on whether cross-topic references are common in your use case. In customer support, users frequently jump between topics and then return ("going back to the authentication issue..."), so maintaining full history with topic tags works better. In research assistants, topics tend to be more sequential, so resetting on switch is cleaner.

Retrieval Strategy for Conversations

Beyond query rewriting, there are structural decisions about how retrieval interacts with conversation state.

Re-retrieve on every turn: The simplest approach runs a full retrieval step on every user query (after rewriting). This is the default and works well for most applications. Each turn gets fresh, relevant context from the knowledge base.

Incremental retrieval: Instead of replacing the retrieved context on every turn, add new chunks to a growing context that accumulates over the conversation. This preserves relevant chunks from prior turns that remain relevant to the current discussion. The risk is context accumulation: after 10 turns, the accumulated chunks may consume most of the context window, and older chunks may no longer be relevant. Capping the accumulated context at a token limit and evicting the oldest chunks mitigates this.

Cached retrieval: If the rewriter determines that the current query does not need new information (for example, a clarification question like "can you explain that in simpler terms?"), skip the retrieval step entirely and reuse the chunks from the previous turn. This reduces latency and cost for turns that are about reformulating the answer rather than finding new information.

The best approach for most systems is re-retrieve on every turn with cached retrieval for non-informational follow-ups. The rewriter already transforms follow-up queries into effective standalone queries, so fresh retrieval on each turn produces good results. Caching for clarification questions saves the retrieval latency (50 to 200 milliseconds) on turns where new context is not needed.

Architecture for Multi-User Conversations

In production, the conversational RAG system serves many users simultaneously, each with their own conversation state. The conversation history must be stored per-session and retrieved when the user sends a new message.

Use a session store (Redis, DynamoDB, or PostgreSQL) keyed by a session ID that maps to the conversation history. Each message exchange appends to the session's history. The session has a TTL (typically 30 minutes to 24 hours) after which inactive conversations are archived or deleted. For systems with long conversation history requirements, the session store holds the raw history while the memory system extracts and persists the key facts.

The processing pipeline for each incoming message: load the session's conversation history from the store, run the query rewriter with history context, run retrieval with the rewritten query, assemble the prompt (system instruction + retrieved chunks + history + current query), generate the response, append both the user message and the assistant response to the session history, and save the updated session. Each step is synchronous because the output of one step feeds the next.

For high-concurrency deployments, the session store must handle concurrent reads and writes without losing messages. Redis handles this naturally with atomic operations. DynamoDB requires conditional writes to prevent race conditions when the same user sends messages in rapid succession (which happens when they hit enter before the previous response finishes streaming).