How to Build a RAG Pipeline from Scratch

Step 1: Set Up the Vector Database

The vector database is the persistent store that connects your ingestion pipeline to your retrieval pipeline. For most projects, start with one of these three options based on your constraints.

pgvector (PostgreSQL extension) is the best choice if you already run PostgreSQL or want the simplest deployment. Install the extension, create a table with a vector column, and create an HNSW index. pgvector handles millions of vectors on a single instance and requires no additional infrastructure. A db.r6g.large RDS instance (~$150/month) comfortably handles 5 million vectors with sub-10ms query latency.

Qdrant (self-hosted or cloud) is the best choice if you need advanced filtering, multi-tenancy, or plan to scale beyond 10 million vectors. Qdrant's HNSW implementation is highly optimized, and its payload filtering (metadata queries combined with vector search) is the strongest in the field. The Docker image runs locally for development, and Qdrant Cloud handles production deployment.

Pinecone is the best choice if you want a fully managed service with zero operational burden. You create an index via API, push vectors, and query. Pinecone handles scaling, replication, and failover. The trade-off is vendor lock-in and higher per-vector cost compared to self-hosted options.

Whichever you choose, configure the index with these parameters: vector dimension matching your embedding model (1536 for OpenAI text-embedding-3-small, 1024 for Voyage or Cohere), cosine similarity as the distance metric, and HNSW as the index type with ef_construction of 200 and M of 16 (these are good defaults for quality-speed balance). Create the metadata schema to store at minimum: chunk text, source document title, section heading, page number, document date, and any domain-specific fields you will want to filter on.

Step 2: Build the Ingestion Pipeline

The ingestion pipeline transforms raw documents into searchable vector records. Build it as a standalone script or service that can be run on new documents as they arrive.

Document loading: Use Unstructured.io (open source or hosted API) for PDFs and complex document formats. It handles layout detection, table extraction, and section hierarchy better than simpler parsers. For Markdown, HTML, and plain text, direct parsing is sufficient. For data from APIs (Confluence, Notion, Slack), use the platform's API to extract content and convert to a common format.

Text cleaning: Remove repeated page headers/footers, normalize whitespace, fix encoding issues, and strip boilerplate (copyright notices, "page X of Y" markers). Keep this simple and document-type-specific. Do not over-clean: removing content that might be useful for retrieval is worse than leaving in some noise.

Chunking: Start with recursive chunking at 512 tokens with 100 tokens of overlap. This is the best general-purpose baseline. If your documents are well-structured with clear headings, switch to section-based chunking. If your corpus is conversational (support tickets, meeting transcripts), consider semantic chunking. The detailed guide on chunking strategies covers when to use each approach.

Metadata attachment: Each chunk must carry metadata: the source document's title, the section it came from, the page number or position, the document's date, and any tags or categories. This metadata enables filtered retrieval later (search only recent documents, search only user manuals, exclude drafts) and helps the LLM attribute information in its response.

Embedding: Embed each chunk using your chosen embedding model. For most projects, OpenAI's text-embedding-3-small ($0.02 per million tokens) provides the best cost-quality trade-off. For code-heavy content, Voyage AI's voyage-code-3 is stronger. For multilingual content, Cohere's embed-v4 excels. Batch the embedding calls (most APIs accept batches of 100+ texts) to maximize throughput and minimize latency.

Storage: Write each chunk's embedding, text, and metadata to the vector database. Include a unique ID that combines the document ID and chunk position, which allows you to update or delete specific chunks when documents are modified. After inserting all chunks, verify the index is built (for pgvector, run ANALYZE on the table, for Qdrant, check the collection info endpoint).

Deduplication: If your knowledge base has overlapping documents (multiple versions of the same guide, Confluence pages that quote each other), add a deduplication step that detects near-duplicate chunks by embedding similarity and keeps only one copy. A cosine similarity threshold of 0.95 catches exact and near-exact duplicates without removing legitimately similar but distinct content.

Step 3: Implement Retrieval

The retrieval layer takes a user query and returns the most relevant chunks from the vector database. Start with dense retrieval (vector similarity) and add sparse retrieval (BM25) later if evaluation shows gaps in keyword matching.

Query embedding: Embed the user's question using the same embedding model used during ingestion. This is a single API call that returns in 50 to 100 milliseconds.

Vector search: Query the vector database for the top-k most similar chunks. Start with k=20 (retrieve 20 candidates for reranking in the next step, which will select the final 3 to 5). Apply any metadata filters at this stage: if the user is asking about a specific product, filter to that product's documentation. If the user's question is time-sensitive, filter to documents from the last N months.

Hybrid search (recommended upgrade): After your dense retrieval baseline is working, add BM25 keyword search in parallel. Run both dense and sparse searches, then merge the results using Reciprocal Rank Fusion (RRF). The formula assigns each result a score of 1/(k + rank) where k is a constant (typically 60), then sums scores across the two result lists and sorts by combined score. This consistently outperforms either approach alone. Some vector databases (Qdrant, Weaviate, Elasticsearch) support hybrid search natively. For pgvector, you can run a separate BM25 query against the text column using PostgreSQL's full-text search and merge the results in application code. The hybrid search guide covers the implementation details.

Step 4: Add Reranking

Reranking re-scores the retrieval candidates using a cross-encoder model that evaluates each query-chunk pair jointly, producing a more accurate relevance score than the embedding similarity used in the initial retrieval. This step is optional for prototypes but consistently improves answer quality in production.

Take the top 20 results from the retrieval step and pass each one through the reranker alongside the query. The reranker scores each pair on a 0-to-1 relevance scale. Sort by the reranker's scores and keep the top 3 to 5 results for inclusion in the prompt.

Model choices: Cohere Rerank v3.5 is the strongest managed API option ($1 per 1,000 search queries). For self-hosted, BGE-reranker-v2-m3 and Jina Reranker v2 are the strongest open-source options, runnable on a single GPU or even CPU (with higher latency). The existing guide on choosing a reranker benchmarks the options.

Reranking adds 50 to 200 milliseconds of latency per query. For latency-critical applications, you can reduce this by reranking fewer candidates (top 10 instead of top 20) or by using a faster, smaller reranker model at some cost to accuracy.

Step 5: Build the Generation Prompt

The prompt builder assembles the final prompt from four components: the system instruction, the retrieved context, optional conversation history, and the user's question.

System instruction: Tell the model its role and constraints. A production system instruction should include: "Answer the user's question based on the provided context documents. If the context does not contain enough information to answer, say so rather than guessing. Cite your sources using the document numbers provided. Do not use information from outside the provided context."

Context formatting: Present each retrieved chunk with a source marker. Use a format like "[Source 1: Document Title, Section Name]" followed by the chunk text. This gives the LLM clear handles to cite in its response and helps it distinguish between multiple sources. Order the chunks by relevance score (most relevant last, closest to the question) to exploit the LLM's recency bias in attention.

Conversation history: For multi-turn interactions, include the last 3 to 5 turns of conversation before the current question. This provides context for follow-up questions. For the first turn, this section is empty.

User question: Place the user's question after the context, clearly labeled. The complete prompt structure is: system instruction, then "Context Documents:" section with the chunks, then "Conversation History:" section with prior turns, then "User Question:" with the current question.

Context window budget: Reserve at least 25% of the context window for the response. For a 128k-token model, this means up to 96k tokens for input, but in practice you should aim for 3,000 to 8,000 tokens of context (3 to 5 chunks of 500 to 1500 tokens each) to keep the model focused on the most relevant material. More context is not always better: experiments consistently show that 3 to 5 highly relevant chunks outperform 15 to 20 mixed-relevance chunks.

Step 6: Add Evaluation and Monitoring

Before deploying to users, build the evaluation infrastructure that will tell you whether the system is working and when it stops working.

Evaluation dataset: Create 50 to 100 question-answer pairs where you know the correct answer and which documents contain it. This is manual work but essential. Include questions of varying difficulty: simple factual lookups, questions requiring synthesis across multiple documents, questions with ambiguous phrasing, and questions where the answer is not in the knowledge base (to test the system's ability to decline).

Retrieval metrics: Measure recall@5 (do the top 5 results include the correct document?) and MRR (how high does the first correct result rank?). These tell you whether retrieval is finding the right documents. If recall is low, the problem is in chunking, embedding, or the retrieval algorithm.

Generation metrics: Use an LLM-as-judge framework (RAGAS, DeepEval) to measure faithfulness (does the answer only contain claims supported by the context?), answer relevance (does the answer address the question?), and completeness (does it use all relevant information from the context?). The RAG evaluation guide covers these frameworks in detail.

Production monitoring: Log every query, the retrieved chunks, the generated answer, and the user's feedback (thumbs up/down, follow-up questions, or explicit corrections). Monitor retrieval latency, generation latency, and token usage. Set alerts for latency spikes, increased hallucination rates, and declining user satisfaction scores.

Feedback loop: Use the logged queries and user feedback to continuously improve the system. Queries that produce poor answers reveal gaps in the knowledge base (missing documents), chunking failures (the relevant information is present but not in a retrievable chunk), or generation failures (the context was correct but the model misinterpreted it). Each failure type requires a different intervention.