RAG Architecture: Components and Data Flow

The Two-Phase Architecture

The fundamental insight of RAG architecture is the separation of offline and online processing. The offline phase (ingestion) can be slow, batch-oriented, and compute-intensive because it runs ahead of time, not in the critical path of a user request. The online phase (retrieval and generation) must be fast, concurrent, and reliable because users are waiting for the response. These different constraints drive fundamentally different engineering decisions for each phase.

The ingestion phase takes raw documents and produces a searchable vector index. The online phase takes a user query and produces a grounded answer. Between them sits the vector database, which is the shared state that connects the two phases. Everything upstream of the database is ingestion, everything downstream is retrieval and generation. This separation means you can upgrade, retune, or completely replace either phase independently, which is why well-designed RAG systems are modular rather than monolithic.

Ingestion Pipeline Components

Document Loader

The document loader is the entry point that reads raw files from their storage location and converts them into a common internal format. In practice this means handling PDFs, Word documents, HTML pages, Markdown files, CSV/JSON data, email archives, Confluence pages, Notion exports, Google Docs, and whatever other formats your knowledge base contains. Each format requires a different parser, and the quality of parsing directly limits the quality of everything downstream.

PDF parsing deserves special attention because PDFs are the dominant format in enterprise knowledge bases and the hardest to parse correctly. A PDF is a visual format, not a semantic one: it describes where to place characters on a page, not how those characters form paragraphs, sections, or tables. Naive text extraction (PyPDF2, pdfminer) reads characters in the order they appear in the file, which may not match reading order, especially in multi-column layouts. Layout-aware parsers like Unstructured.io, Docling, and Amazon Textract analyze the visual layout to reconstruct reading order, table structure, and section hierarchy. The difference in downstream quality between naive and layout-aware parsing is substantial, particularly for documents with tables, sidebars, and complex formatting.

The output of the document loader is a stream of document objects, each containing clean text, structural metadata (title, sections, page numbers), and source metadata (file path, author, date, version). This metadata travels with the text through every subsequent stage and becomes part of the stored record in the vector database.

Text Preprocessor

The preprocessor cleans and normalizes the text output from the document loader. This includes removing artifacts from the parsing process (page numbers, headers and footers that repeat on every page, table of contents entries, watermarks), normalizing Unicode and whitespace, handling encoding issues, and stripping boilerplate content like copyright notices and disclaimers that would add noise without adding information.

Preprocessing is domain-specific. Legal documents need clause numbering preserved because it is semantically important. Medical documents need abbreviations expanded or annotated because a chunk containing "pt presented w/ SOB and JVD" is less useful for retrieval than "patient presented with shortness of breath and jugular venous distension." Code documentation needs code blocks preserved as-is while surrounding prose is cleaned. There is no universal preprocessing pipeline, and the preprocessing rules for one knowledge base may be wrong for another.

Chunker

The chunker splits preprocessed text into segments that will be individually embedded and stored. Chunk size, chunking strategy, and overlap between chunks are the highest-leverage parameters in the entire RAG pipeline. Too small and chunks lose context, making them harder to retrieve and less useful when retrieved. Too large and chunks become imprecise, matching broadly rather than specifically, and consuming more of the context window when injected into the prompt.

The main strategies are fixed-size chunking (split at a uniform token count with overlap), section-based chunking (split at structural boundaries like headers and paragraphs), semantic chunking (split where the topic changes, detected by drops in embedding similarity between adjacent sentences), and recursive chunking (try to split at the largest structural boundary, then progressively split at smaller boundaries until chunks meet the size constraint). The dedicated guide on chunking strategies covers each approach with benchmarks and recommendations by document type.

Most production systems use chunks of 256 to 1024 tokens with 50 to 200 tokens of overlap. These numbers are not arbitrary: they reflect the sweet spot where chunks are large enough to contain a coherent idea but small enough to be precisely retrievable and to fit several of them into the LLM's context window alongside the system instruction and conversation history.

Embedding Model

The embedding model converts each text chunk into a dense vector, typically 768 to 3072 dimensions, that captures its semantic meaning as a point in high-dimensional space. Similar chunks produce similar vectors, which is what makes vector similarity search work. The choice of embedding model affects retrieval quality, latency, storage costs, and multilingual support.

The current landscape (mid-2026) includes OpenAI's text-embedding-3-small (1536 dimensions, $0.02/M tokens) and text-embedding-3-large (3072 dimensions, $0.13/M tokens), Cohere's embed-v4 (1024 dimensions, strong multilingual), Voyage AI's voyage-3 (1024 dimensions, best-in-class on MTEB benchmarks for code and technical text), and open-source models like BGE-M3 and GTE-Qwen2 that can be self-hosted at no marginal cost. The existing comparison of embedding models provides detailed benchmarks.

The critical requirement is consistency: the same model must be used for embedding documents during ingestion and for embedding queries at retrieval time. If you switch embedding models, you must re-embed your entire document collection, which is computationally trivial (embedding is cheap) but operationally important to remember.

Vector Database

The vector database stores the embedding vectors alongside the original text and metadata, and provides efficient similarity search. It is the bridge between the ingestion and retrieval phases. The main options are managed cloud services (Pinecone, Qdrant Cloud, Weaviate Cloud), self-hosted vector databases (Qdrant, Weaviate, Milvus), and vector extensions for existing databases (pgvector for PostgreSQL, the vector type in Redis, Elasticsearch's dense vector support).

For most RAG applications, the choice between these options is driven more by operational considerations (managed vs self-hosted, integration with existing infrastructure) than by retrieval quality, because the similarity search algorithms (HNSW, IVF, Product Quantization) are largely the same across implementations. The existing guide on vector databases compared covers the trade-offs. The key technical decision is the index type: HNSW provides the best quality-latency trade-off for collections under 100 million vectors, which covers the vast majority of RAG deployments.

Online Pipeline Components

Query Preprocessor

The query preprocessor transforms the user's raw input into one or more queries optimized for retrieval. At minimum, this involves embedding the query using the same model used during ingestion. More sophisticated systems add query expansion (adding synonyms or related terms to improve recall), query decomposition (breaking a complex question into simpler sub-questions that are each retrieved independently), and conversational context integration (rewriting a follow-up question like "what about the pricing?" into a standalone query like "what is the pricing for the authentication service we discussed?" using the conversation history).

HyDE (Hypothetical Document Embeddings) is a query preprocessing technique that generates a hypothetical answer to the question using the LLM and embeds that answer rather than the original query. The hypothesis is that a generated answer, even if factually incorrect, will be closer in embedding space to the actual relevant documents than the question itself, because the answer uses the same vocabulary and framing as the documents while the question uses question-oriented phrasing. HyDE adds one LLM call of latency but can significantly improve retrieval quality for questions whose phrasing is very different from the language used in the documents.

Retriever

The retriever executes the similarity search against the vector database and returns the top-k most relevant chunks. Dense retrieval uses vector cosine similarity, sparse retrieval uses keyword-based scoring (BM25), and hybrid retrieval runs both in parallel and merges the results. Hybrid consistently outperforms either approach alone because it captures both semantic relevance (dense) and exact keyword matching (sparse), covering each other's failure modes. The dedicated page on hybrid search for RAG covers the implementation.

The retriever may also apply metadata filters before or during search. Filtering by document type, date range, access permissions, or source collection narrows the search space to only relevant documents, improving both precision and performance. For example, a query about current pricing should filter to documents from the last 6 months, and a query from a specific user should filter to documents they have permission to access.

Reranker

The reranker takes the top-k results from the retriever and re-scores them using a more computationally expensive model, typically a cross-encoder that evaluates each query-chunk pair jointly. The retriever's vector similarity scores are fast to compute (milliseconds) but are a rough proxy for relevance because the query and document are embedded independently. The cross-encoder reads the query and chunk together, attending to the relationships between them, which produces a more accurate relevance score at the cost of more computation (50 to 200ms for 20 to 50 candidates).

Reranking typically reorders the top 20 to 50 retriever results and selects the top 3 to 10 for inclusion in the prompt. The improvement is most pronounced when the retriever returns a mix of relevant and marginally relevant chunks, which is common for ambiguous or multi-topic queries. Popular reranker models include Cohere Rerank, the BGE reranker series, and Jina Reranker. The existing guide on choosing a reranker benchmarks the options.

Prompt Builder

The prompt builder assembles the final prompt that goes to the LLM. It combines the system instruction (which tells the model to answer based on the provided context and cite sources), the reranked context chunks (formatted with source markers), optional conversation history (for multi-turn interactions), and the user's question. The order and formatting of these components affect answer quality: placing the most relevant chunks closest to the question exploits the LLM's attention patterns, and clear source markers enable the model to generate accurate citations.

The prompt builder also enforces the context window budget. If the reranked results contain more tokens than the budget allows (accounting for the system instruction, conversation history, question, and reserved space for the response), the builder must truncate by dropping the least relevant chunks or trimming long chunks. Production systems typically reserve 25% to 40% of the context window for the response and distribute the remainder across context chunks, system instructions, and history.

Generator (LLM)

The generator is the LLM that receives the assembled prompt and produces the final answer. In a RAG system, the LLM's role is synthesis, not recall: it reads the provided context, understands the question, and generates a coherent answer that draws from the context while citing its sources. The model's parametric knowledge is secondary, and the system instruction should explicitly tell the model to rely on the provided context and decline to answer when the context is insufficient.

The choice of generator model involves a cost-quality trade-off. GPT-4o and Claude Opus 4 produce the highest quality synthesis and the most reliable citation behavior, but at $2.50 to $15 per million input tokens. GPT-4o mini and Claude Haiku produce good quality at 5x to 20x lower cost, which matters at high query volumes. For latency-sensitive applications, streaming the response (displaying tokens as they are generated) masks the generation time and provides a better user experience.

Data Flow Summary

The complete data flow through a RAG pipeline follows this path:

Ingestion: Raw documents are loaded by the document loader, cleaned by the preprocessor, split by the chunker, embedded by the embedding model, and stored in the vector database with their original text and metadata.

Retrieval-Generation: The user's query is preprocessed (optionally expanded, decomposed, or rewritten), embedded by the same embedding model, used to search the vector database, reranked by the cross-encoder, assembled into a prompt by the prompt builder, and sent to the LLM for generation. The LLM's response, including citations to the retrieved sources, is returned to the user.

Each component is independently replaceable. You can swap embedding models (and re-embed), change vector databases, switch rerankers, or change the generator LLM without redesigning the pipeline. This modularity is one of RAG's architectural strengths and the reason it is practical to start simple and improve incrementally.