RAG Pipelines: Building Retrieval-Augmented Generation Systems

What RAG Actually Is

RAG is not a product, a framework, or a specific technology. It is a design pattern. The pattern has three steps that happen on every user request: retrieve relevant documents from a knowledge base, inject those documents into the LLM's context window alongside the user's question, and generate a response that uses the retrieved information as its source material. The LLM does not need to memorize the facts because they are handed to it at request time, fresh from the source.

The term was introduced in a 2020 paper by Patrick Lewis and colleagues at Meta AI, where they demonstrated that combining a neural retriever with a seq2seq generator outperformed pure generative models on knowledge-intensive tasks. But the idea is older than the paper: search engines augmenting responses with retrieved snippets, question-answering systems pulling from document stores, and even the simple pattern of pasting relevant context into a prompt all follow the same principle. What changed in 2023 and accelerated through 2026 is that LLMs became capable enough to synthesize retrieved information into coherent, nuanced answers rather than just extracting spans, and vector databases matured enough to make semantic retrieval practical at scale.

The reason RAG dominates production AI architectures is economic and practical. Fine-tuning teaches a model new behaviors but cannot give it access to information that changes weekly. Long context windows can hold more text but cost proportionally more per token and degrade in quality as the context grows. A knowledge graph stores structured relationships but requires expensive curation. RAG provides a middle path: your knowledge base is a collection of documents that can be updated at any time by adding, removing, or modifying files, and the retrieval system finds the right ones for each question without requiring the model to have seen them during training. Updates are immediate, costs scale with storage rather than training compute, and the retrieved documents serve as built-in citations that make the system's reasoning auditable.

The practical reality of RAG in 2026 is that every major LLM application uses some form of it. Customer support bots retrieve product documentation. Legal AI systems retrieve case law and statutes. Code assistants retrieve repository context. Medical AI retrieves clinical guidelines. Enterprise search platforms retrieve internal documents. The architecture varies in sophistication, from a simple vector similarity lookup to multi-hop agentic retrieval chains, but the core pattern is the same: find the right context, put it in the prompt, and let the model do what it does best, which is language generation, not memorization.

The RAG Pipeline Architecture

A RAG pipeline has two distinct phases that run at different times. The offline phase, called ingestion, processes your source documents into a searchable index before any user query arrives. The online phase, called retrieval and generation, runs on every user request and has millisecond-level latency requirements. Understanding this separation is critical because the engineering constraints are completely different: ingestion can be slow, batch-oriented, and compute-intensive, while retrieval must be fast, concurrent, and reliable.

The ingestion phase takes raw documents (PDFs, web pages, database records, Markdown files, Slack messages, code repositories) and transforms them into a format the retrieval system can search. This involves parsing the documents into clean text, splitting the text into chunks of appropriate size, computing a vector embedding for each chunk using an embedding model, and storing those embeddings alongside the original text and metadata in a vector database. The quality of every downstream step depends on how well this phase is executed, which is why chunking strategies are the most studied and debated component of RAG engineering.

The online phase begins when a user submits a query. The query is embedded using the same embedding model that was used during ingestion, ensuring the vectors are in the same semantic space. The vector database performs a similarity search, comparing the query embedding against all stored chunk embeddings, and returns the top-k most similar chunks. These chunks are then formatted and injected into the LLM's prompt as context, typically after a system instruction and before the user's question. The LLM generates a response using both the retrieved context and its own parametric knowledge, ideally citing the retrieved documents and declining to answer when the context does not contain relevant information.

Between retrieval and generation, most production pipelines add a reranking step. The initial vector search returns candidates quickly but with imperfect relevance ranking, because embedding similarity is a rough proxy for relevance. A reranker, typically a cross-encoder model that scores query-document pairs jointly, re-orders the candidates by true relevance before they go into the prompt. This step adds 50 to 200 milliseconds of latency but measurably improves answer quality, especially on queries where the correct answer is semantically adjacent to several incorrect results. The existing guide on choosing a reranker covers the available models and their trade-offs.

The full architecture also includes metadata filtering, query preprocessing (expansion, decomposition, routing), and post-generation verification, but the core loop is always the same: embed the query, search the index, rerank the results, build the prompt, and generate the answer. Every advanced pattern in RAG is a refinement of one or more of these steps, not a replacement of the architecture itself.

The Ingestion Pipeline

The ingestion pipeline is the foundation that determines the ceiling on your RAG system's quality. No amount of clever retrieval or prompt engineering can compensate for poorly ingested data, because the system can only retrieve what it has indexed, and if the indexed chunks are noisy, incomplete, or poorly segmented, the retrieved context will be noisy, incomplete, or poorly segmented.

Document parsing is the first step and the one most teams underestimate. Plain text and Markdown files are straightforward, but PDFs, which account for a majority of enterprise knowledge, are structurally complex. A PDF might contain multi-column layouts, headers and footers that repeat on every page, tables where cell boundaries are implicit, images with embedded text, and nested sections with inconsistent formatting. Naive PDF-to-text extraction (using libraries like PyPDF2 or pdfminer) often produces garbled text with broken paragraphs, merged columns, and lost table structure. Production pipelines use layout-aware parsers like Unstructured.io, Docling, or Amazon Textract that understand the visual structure of the page and extract text in reading order while preserving table boundaries and section hierarchy.

After parsing, the text needs cleaning. This includes removing repeated headers and footers, normalizing whitespace and encoding, handling special characters, and stripping boilerplate like copyright notices and page numbers. The goal is a clean, continuous text that represents the document's actual content without noise that would confuse both the embedding model and the LLM. Cleaning is tedious and domain-specific: legal documents have different noise patterns than medical records, which have different patterns than software documentation.

Metadata extraction happens alongside text cleaning and is often overlooked but critically important. Every chunk should carry metadata that includes at minimum the source document title, the section heading it came from, the page number or location within the document, and the document's creation or modification date. This metadata enables filtered retrieval (search only documents from the last 6 months, search only the user manual, exclude drafts) and provides context that helps the LLM understand where the information comes from. Without metadata, retrieved chunks are anonymous text fragments that the model cannot attribute or contextualize.

The final ingestion step is embedding, where each chunk is converted to a high-dimensional vector representation using an embedding model. The choice of embedding model matters: OpenAI's text-embedding-3-large, Cohere's embed-v4, Voyage's voyage-3, and open-source models like BGE and GTE each have different strengths in terms of retrieval quality, multilingual support, context length, and cost. The key requirement is consistency: you must use the same embedding model for ingestion and query-time embedding, or the vectors will be in different semantic spaces and similarity search will fail. The existing guide on embedding models compared covers the current landscape.

Chunking: Where Most Pipelines Succeed or Fail

Chunking is the process of splitting a parsed document into segments that will be individually embedded and retrieved. It is the single most impactful parameter in a RAG pipeline, and getting it wrong degrades every downstream component. The core tension is between granularity and context: small chunks are more precisely retrievable but lose surrounding context, while large chunks preserve context but are less precisely retrievable and consume more of the LLM's context window.

The simplest approach is fixed-size chunking, where you split the text into segments of a uniform token count (typically 256 to 1024 tokens) with overlap between adjacent chunks (typically 10% to 20%) to avoid cutting sentences or ideas in half. This works reasonably well for homogeneous text like prose articles or conversation logs, and its simplicity makes it a good baseline. But it fails on structured documents because it splits tables in the middle, separates headings from their content, and produces chunks that start and end at arbitrary points rather than at natural topic boundaries.

Semantic chunking addresses this by using the document's structure to guide the splits. Section headers, paragraph breaks, and formatting signals define natural boundaries, and the chunker respects those boundaries rather than splitting at a fixed token count. A more sophisticated variant uses embedding similarity between adjacent sentences to detect topic shifts: when the similarity between consecutive sentence embeddings drops below a threshold, the chunker places a boundary there. This produces chunks that correspond to coherent topics rather than arbitrary text spans, which improves retrieval precision because the embedding of each chunk represents a single concept rather than a blend of unrelated sentences.

In practice, the best approach depends on the document type. Technical documentation and knowledge bases respond well to section-based chunking because they are already structured. Conversational data and meeting transcripts benefit from topic-shift detection because they have natural topic boundaries but no formal structure. Legal contracts and regulatory filings need hierarchical chunking that preserves the nesting of clauses and subclauses. Code repositories need AST-aware chunking that respects function, class, and module boundaries rather than splitting code at arbitrary line numbers.

The overlap strategy also matters more than most teams realize. A 10% overlap means the last 10% of chunk N is duplicated as the first 10% of chunk N+1. This ensures that sentences or ideas that span a chunk boundary are fully represented in at least one chunk. Without overlap, a query about a concept that happens to fall at a boundary will retrieve two chunks that each contain half the answer but neither contains the whole thing. The trade-off is storage: 20% overlap means 20% more chunks to embed, store, and search. Most production systems use 50 to 200 tokens of overlap regardless of chunk size, which provides adequate boundary coverage without excessive duplication. The deep dive on chunking strategies covers every approach with benchmarks.

Retrieval Strategies

Retrieval is the step that determines which chunks the LLM sees, and there are fundamentally two approaches: dense retrieval using vector embeddings and sparse retrieval using keyword matching, plus the hybrid combination that outperforms either alone.

Dense retrieval works by computing the cosine similarity (or dot product) between the query embedding and every chunk embedding in the index, then returning the top-k most similar chunks. It excels at semantic understanding: a query about "reducing customer churn" will retrieve chunks about "retention strategies" and "preventing subscriber loss" even though those phrases share no keywords. This semantic matching is the core advantage of dense retrieval and the reason vector search became the default retrieval method for RAG. The limitation is that dense retrieval can miss exact matches: a query for "error code E-4217" might not retrieve a chunk that mentions that exact code if the embedding model maps it to a general "error handling" region of the vector space.

Sparse retrieval, using algorithms like BM25 or TF-IDF, works by matching query terms against document terms and scoring based on term frequency and document rarity. It excels at exact matching: the query "error code E-4217" will reliably retrieve any chunk containing that code. Sparse retrieval also handles rare terminology, proper nouns, and domain-specific jargon better than dense retrieval because it does not depend on the embedding model having seen those terms during training. The limitation is that sparse retrieval misses semantic connections: "reducing churn" and "retention strategies" share no terms and would not match.

Hybrid retrieval combines both approaches and consistently outperforms either one alone across benchmarks. The typical implementation runs dense and sparse retrieval in parallel, produces two ranked lists, and merges them using a fusion algorithm like Reciprocal Rank Fusion (RRF). RRF assigns scores based on rank position rather than raw similarity scores, which avoids the calibration problem of combining scores from different systems. The result captures both semantic relevance and keyword precision, covering the failure modes of each individual approach. The existing guide on reciprocal rank fusion explains the algorithm, and the dedicated page on hybrid search for RAG covers implementation patterns.

Beyond the retrieval algorithm itself, query preprocessing significantly impacts what gets retrieved. Query expansion adds related terms to the query to improve recall. Query decomposition breaks a complex question into sub-questions that are each retrieved independently. HyDE (Hypothetical Document Embeddings) generates a hypothetical answer to the query and embeds that answer rather than the query itself, which can improve retrieval because the hypothetical answer is closer in embedding space to actual document chunks than the question is. These techniques add latency and complexity but measurably improve retrieval quality for complex queries.

The Generation Step

Once retrieval delivers the relevant chunks, the generation step constructs a prompt that integrates the retrieved context with the user's question and sends it to the LLM. This step is where the quality of the final answer is determined, and the prompt structure matters more than most teams expect.

The standard prompt structure for RAG has four components: a system instruction that tells the model it should answer based on the provided context and decline to answer when the context is insufficient, the retrieved documents formatted with clear source markers, the user's question, and optionally the conversation history for multi-turn interactions. The system instruction is critical for preventing the model from hallucinating beyond the context. Without it, the model will freely blend retrieved information with its parametric knowledge, which defeats the purpose of RAG because you cannot distinguish grounded facts from hallucinated ones.

Context formatting affects quality in measurable ways. Each retrieved chunk should be clearly delineated with source markers (document title, section heading, source number) so the LLM can attribute information and the user can verify claims against specific sources. Placing the most relevant chunks closest to the user's question takes advantage of the LLM's recency bias in attention, improving the likelihood that the most important context influences the answer. Limiting the total context to what is genuinely relevant, rather than filling the context window to capacity, reduces noise and improves precision: 3 highly relevant chunks typically produce better answers than 10 chunks of mixed relevance.

Citation generation is the mechanism that makes RAG answers verifiable. The system instruction should tell the model to cite its sources using the source markers provided in the context, and the output should be parseable so your application can render citations as clickable links to the original documents. This is not just a nice feature, it is a trust mechanism. Users of RAG systems quickly learn to check citations, and a system that consistently provides accurate citations earns trust, while one that provides wrong or missing citations destroys it. Some implementations verify citations programmatically after generation, checking that each cited source actually contains the information attributed to it, and flagging or removing unverifiable claims.

For multi-turn conversations, the generation step needs to integrate conversation history with the current retrieval results. The standard approach is to include the last N turns of conversation before the user's current question, so the model understands the context of follow-up questions like "what about the pricing?" that only make sense in the context of a prior exchange. This interacts with retrieval: a follow-up question should ideally trigger a new retrieval using the contextualized query rather than the ambiguous follow-up text alone, which is where query rewriting and conversation-aware retrieval come in.

Evaluating RAG Quality

RAG evaluation is more complex than evaluating a standalone LLM because you need to measure both retrieval quality (did the system find the right documents?) and generation quality (did the model produce a correct answer from those documents?) independently. A wrong answer could be caused by a retrieval failure (the right document was not returned) or a generation failure (the right document was returned but the model misinterpreted it), and fixing the two problems requires different interventions.

Retrieval quality is measured with information retrieval metrics: recall@k (what fraction of relevant documents appear in the top k results), precision@k (what fraction of the top k results are relevant), Mean Reciprocal Rank (how high the first relevant document ranks), and Normalized Discounted Cumulative Gain (NDCG, which accounts for the position of every relevant document, not just the first). These metrics require a labeled evaluation set where you know which documents are relevant for each query. Building this set is the most time-consuming part of RAG evaluation, but without it you are guessing at retrieval quality rather than measuring it.

Generation quality is measured along three dimensions: faithfulness (does the answer contain only claims supported by the retrieved context?), answer relevance (does the answer actually address the question?), and completeness (does the answer cover all the information in the retrieved context that is relevant to the question?). Automated frameworks like RAGAS, DeepEval, and Trulens compute these metrics using LLM-as-judge approaches, where a separate LLM evaluates the answer against the context and question. These are not perfect substitutes for human evaluation, but they provide a scalable signal that correlates well with human ratings when properly calibrated. The full evaluation methodology is covered in how to evaluate RAG pipeline quality.

The most important metric for production RAG systems is the hallucination rate: what fraction of answers contain claims not supported by the retrieved context. This is distinct from factual accuracy because a RAG system can be factually correct (the claim happens to be true based on the model's parametric knowledge) while still being unfaithful (the claim is not supported by the retrieved context). In a RAG system, unfaithful answers are failures even when they are factually correct, because the system is supposed to be grounded in its knowledge base, and an answer that ignores the retrieved context to draw on parametric knowledge cannot be audited against the knowledge base. The existing pillar on hallucinations covers the broader problem, while the RAG evaluation guide focuses specifically on grounding and faithfulness metrics.

Advanced RAG Patterns

The basic RAG pattern, where a single retrieval step feeds a single generation step, works for straightforward factual questions but struggles with complex queries that require reasoning across multiple documents, synthesizing information from different sources, or determining the right retrieval strategy dynamically. Advanced patterns address these limitations by adding intelligence to the retrieval process itself.

Agentic RAG replaces the fixed retrieve-then-generate pipeline with an autonomous agent that decides how to retrieve information based on the query. Instead of always running the same vector search, the agent can choose between different retrieval tools (vector search, keyword search, SQL queries, API calls), decompose complex questions into sub-queries, evaluate whether the retrieved results are sufficient, and iterate by retrieving additional context if the first round was incomplete. This is particularly powerful for multi-hop questions like "Compare the pricing models of the top 3 vendors mentioned in last quarter's procurement reports" where no single retrieval step can get all the needed information. The dedicated page on agentic RAG covers the architecture and implementation patterns.

GraphRAG combines knowledge graphs with vector retrieval to capture both semantic similarity and structural relationships. A standard vector search finds chunks that are similar to the query but has no concept of how those chunks relate to each other or to the broader document structure. GraphRAG constructs a knowledge graph from the source documents, where entities (people, products, concepts) are nodes and their relationships are edges, and uses graph traversal alongside vector search to retrieve not just relevant chunks but the contextual network around them. This produces answers that are better at multi-entity questions, comparison queries, and questions about relationships between concepts. The page on GraphRAG covers how to build one, and the existing knowledge graphs pillar provides the foundational concepts.

Corrective RAG adds a verification loop after retrieval: before sending the retrieved chunks to the LLM for generation, a lightweight model evaluates whether the chunks are actually relevant to the query and filters out irrelevant results. If the verification determines that the retrieved context is insufficient, the system can fall back to web search, query rewriting, or asking the user for clarification rather than generating an answer from poor context. This pattern significantly reduces hallucination by preventing the model from being forced to generate an answer when the knowledge base simply does not contain the relevant information.

Self-RAG takes this further by having the LLM itself decide whether it needs retrieval at all, and if so, evaluate the retrieved documents before using them. The model generates special tokens that indicate whether retrieval is needed, whether each retrieved document is relevant, and whether its own generated response is supported by the retrieved context. This creates a self-monitoring loop that is more precise than external verification because the model can evaluate relevance in the context of its own generation process rather than as an independent step.

Cost and Performance

The cost of a RAG pipeline breaks down into four components: embedding computation during ingestion, vector database storage, retrieval compute on each query, and LLM inference for generation. Understanding where the money goes is essential for making informed architecture decisions.

Embedding costs during ingestion are a one-time expense for each document, plus re-embedding when documents are updated. Using OpenAI's text-embedding-3-small at $0.02 per million tokens, embedding a million tokens of documents costs $0.02. Even at the scale of 100 million tokens (roughly 300,000 pages of text), the embedding cost is $2. Embedding is not the expensive part of RAG. Alternatively, open-source embedding models like BGE or GTE can be self-hosted on a GPU and run at near-zero marginal cost after the infrastructure is set up, though the fixed cost of the GPU matters for smaller deployments.

Vector database costs depend on scale and performance requirements. Managed services like Pinecone charge $0.096 per GB per month, which translates to roughly $8 per month per million 1536-dimensional vectors. Qdrant Cloud, Weaviate Cloud, and Zope offer similar pricing. Self-hosted options like pgvector (a PostgreSQL extension) have no per-vector cost but require you to provision and manage the database server. For most applications with under 10 million vectors, pgvector on a modest instance ($50 to $200 per month) provides adequate performance without the premium of a managed service. The existing comparison of vector databases covers the trade-offs in detail.

Retrieval compute per query is dominated by the embedding of the query (one embedding model call) and the vector similarity search. The query embedding cost is negligible (a single embedding costs fractions of a cent). The vector search latency depends on the database and index type: approximate nearest neighbor (ANN) indexes like HNSW serve queries in 1 to 10 milliseconds even at millions of vectors, while exact search scales linearly with collection size. If you add a reranker, the cross-encoder inference adds 50 to 200 milliseconds and costs roughly $0.001 to $0.01 per query depending on the model and the number of candidates being reranked.

LLM inference for generation is the largest ongoing cost by far. The cost scales with the total tokens in the prompt (system instruction + retrieved context + conversation history + user question) plus the generated response tokens. If your prompt averages 3,000 input tokens and the response averages 500 output tokens, and you use GPT-4o at $2.50 per million input tokens and $10 per million output tokens, each query costs roughly $0.0125. At 100,000 queries per month, that is $1,250 per month, dwarfing the embedding and database costs. Using a smaller model like GPT-4o mini ($0.15 input / $0.60 output per million tokens) drops the same workload to about $75 per month. The RAG cost guide provides detailed calculations and optimization strategies.

RAG vs Fine-Tuning vs Memory

RAG, fine-tuning, and memory solve different problems and are complementary rather than competing approaches. The choice depends on what your application needs, and most production systems use more than one.

RAG provides factual grounding from a knowledge base that can be updated independently of the model. It is the right choice when your application needs access to specific documents, when the information changes over time, when citations and auditability matter, and when the knowledge base is large enough that the model could not memorize it even if you tried. The trade-off is that RAG adds latency (the retrieval step), requires infrastructure (the vector database and ingestion pipeline), and is limited by retrieval quality, meaning the system can only use information it successfully retrieves.

Fine-tuning changes the model's behavior, style, and capabilities at the weight level. It is the right choice when you need the model to follow a specific format or tone without being told in every prompt, when you need to teach it patterns not well-represented in pretraining data, or when prompt length is a cost or latency constraint. Fine-tuning does not solve the knowledge freshness problem because the knowledge it embeds is static. The existing comparison on RAG alternatives and the fine-tuning pillar cover this decision in depth.

Memory, as implemented in systems like Adaptive Recall, stores per-user or per-session information that the system learns from interactions and retrieves when relevant. It solves the personalization and continuity problem: remembering a user's preferences, past questions, ongoing projects, and corrections across sessions. Memory is not a replacement for a document knowledge base (that is RAG's job) or for behavioral customization (that is fine-tuning's job). It fills the gap between them by providing contextual awareness that is dynamic, personal, and interaction-driven.

The production pattern that works best for most applications is: start with RAG for the knowledge base, add memory for user context, and fine-tune only if the base model's behavior needs persistent adjustment that cannot be achieved with system prompts. This combination gives you grounded answers (RAG), personalized context (memory), and optimized behavior (fine-tuning) without the downsides of relying on any single approach.

Foundations

Core Concepts

Advanced Patterns

Implementation Guides