Self-RAG and Corrective RAG Explained

The Problem with Blind Retrieval

Standard RAG has a structural weakness: it always retrieves and always generates, regardless of whether retrieval is necessary or whether the retrieved documents are any good. This produces three specific failure modes that Self-RAG and Corrective RAG address.

Unnecessary retrieval: Some queries do not need retrieval at all. "What is 2 + 2?" and "Write a Python function to sort a list" are tasks the LLM can handle from its parametric knowledge. Running these through a RAG pipeline wastes compute on retrieval and risks contaminating the answer with irrelevant retrieved context. If the vector search returns chunks about basic arithmetic or sorting algorithms, the model may produce a worse answer than it would have without retrieval, because it tries to incorporate context that adds nothing.

Irrelevant retrieval: When the knowledge base does not contain information relevant to the query, vector search still returns the top-k most similar chunks, even if their similarity scores are low and the content is only tangentially related. The model receives this marginal context and treats it as relevant, producing answers that blend hallucinated content with vaguely related retrieved text. The user gets a confident-sounding but wrong answer grounded in irrelevant documents.

Unfaithful generation: Even when the correct documents are retrieved, the LLM may generate claims that are not supported by the context. This happens when the model blends retrieved information with its parametric knowledge, when the prompt does not sufficiently constrain the model to the context, or when the answer requires synthesizing across multiple chunks and the model introduces unsupported inferences. Standard RAG has no mechanism to check whether the generated answer is actually supported by the retrieved context.

How Self-RAG Works

Self-RAG, introduced in a 2023 paper by Asai et al., trains the LLM to generate special reflection tokens that evaluate its own behavior at each step of the RAG process. The model learns to produce four types of reflection tokens:

Retrieve token: Before generating, the model outputs a token indicating whether retrieval is needed for the current query. If the model determines it can answer from parametric knowledge alone, retrieval is skipped entirely. This is not a separate classifier, it is a learned behavior embedded in the model's generation process.

IsRel token: After retrieval, the model evaluates each retrieved document and outputs a relevance judgment (relevant or irrelevant) for each one. Irrelevant documents are discarded before generation, so the model only works with context it has judged to be useful.

IsSup token: During generation, the model evaluates whether each claim in its response is supported by the retrieved context. Claims flagged as unsupported can be removed or regenerated with a stricter prompt. This is the self-monitoring mechanism that catches unfaithful generation.

IsUse token: After generation, the model evaluates the overall utility of its response for the query. Low-utility responses trigger a retry with different retrieval parameters or a different generation approach.

The key insight of Self-RAG is that these reflection capabilities are trained into the model using special tokens in the training data, not added as external classifiers or prompting tricks. The model genuinely learns to evaluate its own retrieval and generation quality as part of its forward pass. This makes the reflection fast (no additional API calls) and integrated (the model considers reflection as part of generation, not as an afterthought).

Implementing Self-RAG in Practice

The original Self-RAG requires fine-tuning a model with reflection tokens, which is impractical for most teams using API-based models. The practical alternative is to implement Self-RAG's logic using prompting and multi-step generation with standard models.

Retrieval decision: Before retrieval, send the query to a fast LLM with a prompt: "Does this question require searching a knowledge base to answer accurately, or can it be answered from general knowledge alone? Respond with RETRIEVE or NO_RETRIEVE." If the model says NO_RETRIEVE, skip the retrieval step and generate directly. This saves 50 to 300 milliseconds of retrieval latency for queries that do not need it and prevents irrelevant context from degrading the answer.

Relevance filtering: After retrieval, send each retrieved chunk with the query to a fast LLM: "Is this document relevant to answering the question? Respond with RELEVANT or IRRELEVANT." Filter out irrelevant chunks before prompt assembly. This is functionally equivalent to the IsRel token but implemented as a separate classification call. Using a fast model (GPT-4o mini or Claude 3.5 Haiku) keeps the cost under $0.001 per query for 5 to 10 chunks.

Faithfulness verification: After generation, send the response, the retrieved context, and the query to a verifier: "For each claim in the response, indicate whether it is supported by the provided context, not supported by the context, or contradicted by the context." Claims flagged as unsupported can be removed, flagged with a disclaimer, or the entire response can be regenerated with a stricter system prompt. This is the most expensive step (a full LLM call on the entire prompt plus response) and is typically reserved for high-stakes applications where hallucination has real consequences.

The trade-off is clear: each reflection step adds one LLM call and 100 to 500 milliseconds of latency. A full Self-RAG implementation with all three reflection steps adds 300 to 1,500 milliseconds to the pipeline. For interactive chat applications, this may be too slow. For batch processing, report generation, or high-stakes Q&A (legal, medical, financial), the accuracy improvement justifies the latency cost.

How Corrective RAG Works

Corrective RAG (CRAG), introduced by Yan et al. in 2024, takes a different approach. Instead of having the model reflect on its own outputs, CRAG adds an external evaluator that checks retrieval quality and triggers corrective actions when the retrieval is poor.

The CRAG pipeline has three stages after initial retrieval:

Retrieval evaluation: A lightweight evaluator (a fine-tuned classifier or a prompted LLM) scores the relevance of the retrieved documents. Each document gets a confidence score, and the scores are aggregated into an overall retrieval quality assessment: correct (high confidence that the retrieved documents contain the answer), ambiguous (mixed confidence, some documents may be relevant), or incorrect (low confidence, the retrieved documents are unlikely to contain the answer).

Corrective action: Based on the assessment, CRAG takes different actions. If the retrieval is correct, it proceeds with standard generation using the retrieved context. If the retrieval is ambiguous, it refines the retrieved documents by extracting only the relevant knowledge strips (specific sentences or passages within each document that are most relevant) and discarding the rest. If the retrieval is incorrect, it triggers a fallback: web search, a different knowledge base, or a query rewrite followed by a second retrieval attempt.

Knowledge refinement: For both correct and ambiguous assessments, CRAG decomposes each retrieved document into fine-grained knowledge strips (individual sentences or small passages) and re-scores each strip against the query. Only strips that score above the relevance threshold make it into the final prompt. This filtering at the sentence level removes noise within otherwise relevant documents, producing a cleaner, more focused context for generation.

The difference between CRAG and Self-RAG is architectural: CRAG uses an external evaluator and explicit corrective actions, while Self-RAG embeds reflection into the generation model itself. CRAG is easier to implement with off-the-shelf models because it does not require model fine-tuning, just a classification step and fallback logic.

When to Use Each Pattern

Use standard RAG when: your knowledge base is comprehensive for the queries it receives, the retrieval quality is consistently high (recall@5 above 85%), and the latency budget is tight. Standard RAG is simpler, faster, and cheaper. Most production RAG systems work well enough with standard RAG plus reranking.

Use Corrective RAG when: your knowledge base has gaps and you need fallback sources, retrieval quality is inconsistent (some queries retrieve well, others poorly), and you want to improve reliability without fine-tuning a model. CRAG's explicit corrective actions (web search fallback, query rewrite) handle the cases where standard RAG fails without adding overhead to the cases where it succeeds.

Use Self-RAG when: hallucination is unacceptable (medical, legal, financial applications), you need the model to handle a mix of knowledge-base questions and general questions without always retrieving, and you can afford the latency and cost of reflection steps. Self-RAG's per-claim faithfulness checking is the strongest defense against unfaithful generation available in a RAG architecture.

Combine both when: you need both retrieval correction (CRAG) and generation verification (Self-RAG). The pipeline becomes: retrieve, evaluate retrieval quality (CRAG), apply corrective action if needed, generate with the refined context, verify faithfulness (Self-RAG), and return the verified response. This is the most reliable configuration but also the most expensive, suitable for systems where every answer must be auditable and defensible.

Practical Trade-Offs

The core trade-off for both patterns is the same: more LLM calls for better accuracy. Each evaluation and reflection step adds $0.001 to $0.01 per query (depending on the model) and 100 to 500 milliseconds of latency. At 100,000 queries per month, the additional cost ranges from $100 to $1,000 per month. At 1,000,000 queries per month, it ranges from $1,000 to $10,000.

A cost-effective implementation applies the reflection selectively rather than on every query. Use a confidence-based trigger: run the retrieval evaluator on every query (it is cheap), but only run the faithfulness verifier when the retrieval confidence is low or the query involves a high-stakes topic. This applies the expensive steps only when they are most likely to make a difference, reducing average cost by 50% to 70% while preserving most of the accuracy benefit.

For teams starting with advanced RAG, begin with Corrective RAG's retrieval evaluator because it provides the highest value-to-cost ratio. If the retrieved documents score below a relevance threshold, trigger a web search fallback or query rewrite. This single addition catches the most impactful failure mode (retrieval returning irrelevant documents) without the complexity of full Self-RAG reflection.