How to Debug a RAG Pipeline That Returns Wrong Answers

The Three Failure Modes

Every wrong RAG answer falls into one of three categories, and correctly classifying the failure is the single most important debugging step. Getting this wrong means fixing the wrong component, which wastes time and leaves the actual problem untouched.

Retrieval failure: The knowledge base contains the correct information, but the retrieval step did not return it. The LLM generated an answer from irrelevant or partially relevant context because that was all it received. This accounts for roughly 60% of RAG failures in production systems and is the most common root cause.

Generation failure: The retrieval step returned the correct documents, but the LLM misinterpreted them, ignored them, or blended them with parametric knowledge to produce an incorrect answer. This accounts for roughly 25% of RAG failures and is caused by prompt design issues, context length problems, or model limitations.

Knowledge gap: The answer simply does not exist in the knowledge base. No amount of retrieval or prompt engineering can fix a missing document. This accounts for roughly 15% of RAG failures and is solved by expanding the knowledge base, not by pipeline changes.

Step 1: Classify the Failure

Start every debugging session by running the failing query and inspecting the intermediate outputs rather than just the final answer.

Log the retrieved chunks. Every production RAG pipeline should log (or make inspectable) the chunks that retrieval returned for each query, along with their similarity scores, source documents, and rank positions. If your pipeline does not log this, add it immediately. Without retrieval logs, you are debugging blind.

Check if the correct chunk was retrieved. Read through the retrieved chunks and determine whether the correct information appears in any of them. If the answer exists in the knowledge base but none of the retrieved chunks contain it, you have a retrieval failure. If one or more retrieved chunks contain the correct information but the final answer is still wrong, you have a generation failure. If you search the entire knowledge base and the information simply is not there, you have a knowledge gap.

Automate the classification. For recurring issues, build a test harness that runs a set of known query-answer pairs through the pipeline and classifies each failure automatically. For each test case, record: (1) whether the correct chunk appears in the top 5, top 10, and top 20 results, (2) whether the generated answer matches the expected answer, and (3) the retrieval similarity scores. This lets you track retrieval quality and generation quality separately over time and catch regressions early.

Step 2: Debug Retrieval Failures

When the correct chunk exists in the knowledge base but was not retrieved, the problem is in the chain from document to embedding to search. Work backwards through the chain.

Check the chunk exists and is well-formed

Verify that the chunk you expect to be retrieved actually exists in the vector database with the content you expect. Parsing and chunking errors during ingestion can produce corrupted, truncated, or incorrectly split chunks. Search the vector database by metadata (source document, page number) rather than by embedding to confirm the chunk is there. Read the chunk text: does it contain the complete answer, or was the answer split across two chunks by a bad chunking boundary?

Common findings: the answer was split between two chunks and neither chunk is self-sufficient. A table was flattened into incomprehensible text during parsing. The section containing the answer was skipped by the parser because of a formatting issue. Headers or footers were not cleaned, diluting the chunk's embedding. The fix is to re-parse and re-chunk the affected documents with better parsing and chunking strategies.

Check the embedding quality

Compute the cosine similarity between the query embedding and the expected chunk embedding. If the similarity is low (below 0.5), the embedding model is not placing them close together in vector space. This happens when the query uses different terminology than the chunk (the user asks about "cancellation policy" but the document says "termination of service"), when the query is very short and the embedding lacks enough signal, or when the embedding model was not trained on your domain's vocabulary.

Solutions: try a different embedding model that better captures your domain semantics. Add hybrid search (BM25 alongside vector search) to catch keyword matches that semantic search misses. Preprocess queries with expansion (adding synonyms and related terms) to bridge vocabulary gaps.

Check metadata filtering

If you use metadata filtering, verify that the filter is not excluding the correct chunk. A common bug: the chunk has metadata {department: "engineering"} but the query filter specifies {department: "Engineering"} (case mismatch). Or the chunk has {version: "3.2"} but the filter defaults to {version: "4.0"} because the user's profile says they are on the latest version. Metadata filter bugs are silent killers because the retrieval appears to work (it returns results) but the correct result was excluded before search even started.

Check the search parameters

Top-k too small: if you retrieve only 3 candidates and the correct chunk ranks 5th, it will never reach the LLM. Increasing top-k (especially when combined with reranking) is the simplest fix for marginal retrieval failures. HNSW ef_search too low: the approximate search might miss the correct chunk if the search beam is too narrow. Increasing ef_search reduces speed but improves recall for hard queries.

Step 3: Debug Generation Failures

When the correct chunk was retrieved but the answer is still wrong, the problem is in the generation step. Test this by passing the correct chunk directly to the LLM with the query (bypassing retrieval entirely) and checking if it produces the right answer.

The LLM ignores the context

If the correct chunk is in the prompt but the LLM generates an answer from its own knowledge instead, the system prompt is not sufficiently directive. Strengthen the instruction: "Answer ONLY based on the provided context. If the context does not contain the answer, say you do not have enough information. Do not use your own knowledge." Test with different phrasings of this instruction. Some models respond better to "You are a document Q&A assistant that only references the provided documents" than to "Do not use external knowledge."

The correct chunk is buried in noise

If the prompt contains 10 chunks but only 1 is relevant, the LLM may be distracted by the 9 irrelevant chunks, especially if they contain plausible-sounding but incorrect information. The "lost in the middle" phenomenon, where models pay less attention to information in the middle of a long context, makes this worse. The fix is better retrieval precision: add reranking, reduce the number of chunks passed to the LLM (3 to 5 is better than 10), or apply a relevance score threshold to filter out low-quality results before prompt assembly.

The chunk is ambiguous or incomplete

Sometimes the correct chunk contains the answer but in a form that the LLM misinterprets. A table where the answer depends on reading the correct row-column intersection. A paragraph where the answer requires inferring from multiple sentences. A chunk where the answer is conditional ("if you are on the enterprise plan...") but the condition is in a different chunk. The fix is to improve chunk quality during ingestion: add contextual headers, ensure tables are self-explanatory, and include enough surrounding context in each chunk for it to be interpretable in isolation.

Context window overflow

If the total prompt (system instruction + retrieved chunks + conversation history + query) exceeds the model's context window, the prompt gets truncated, potentially cutting off the chunk that contains the answer. This is a silent failure because most LLM APIs truncate without error. The fix is to track prompt token counts and alert when they approach the context window limit. Reduce the number of retrieved chunks, shorten conversation history, or compress the system instruction to stay within limits.

Step 4: Fix Knowledge Gaps

When the answer simply is not in the knowledge base, the correct behavior is for the RAG system to say it does not know. If the system instead fabricates an answer, that is a generation failure (the prompt does not instruct the model to decline when context is insufficient) layered on top of a knowledge gap.

Track knowledge gap queries as a product signal. Queries that consistently find no relevant chunks reveal topics your knowledge base does not cover. Aggregate these into a report: "These 50 queries found no relevant documents in the last 30 days." This report tells the content team exactly what documentation to create, prioritized by query frequency.

For the immediate fix, ensure the system prompt instructs the model to decline gracefully: "If the provided context does not contain information relevant to the question, respond with: I don't have information about that topic in my knowledge base. Do not attempt to answer from your own knowledge." This prevents hallucinated answers for topics outside the knowledge base and makes knowledge gaps visible rather than hidden behind fabricated responses.

Building a Debugging Dashboard

A production RAG system needs observability. The minimum debugging infrastructure includes:

Query log: Every query with the rewritten version (if using query rewriting), the retrieved chunk IDs and scores, the assembled prompt, the generated response, and the timestamp. Store for at least 30 days.

Retrieval quality metrics: Track average similarity scores, the number of queries with zero results, and the distribution of top-1 similarity scores. A drop in average similarity signals an embedding or index problem.

User feedback signals: Thumbs up/down, explicit corrections, follow-up questions that rephrase the same query (a signal that the first answer was wrong). Correlate negative feedback with retrieval logs to identify systematic retrieval failures.

Component-level latency: Track the time spent in each pipeline stage (embedding, retrieval, reranking, generation) separately. A latency spike in one component is easier to diagnose than a spike in end-to-end latency.

Tools like LangSmith, Arize Phoenix, Ragas, and Braintrust provide pre-built observability for RAG pipelines. If you are building custom, the minimum viable implementation is a structured log entry per query with the fields above, stored in a searchable system (Elasticsearch, PostgreSQL, or even a JSON file for early-stage systems). The existing evaluation pillar covers the broader monitoring landscape.