How to Add Reranking to a RAG Pipeline

Why Vector Search Alone Is Not Enough

Vector search works by comparing the embedding of the query against the embeddings of stored chunks using cosine similarity or dot product. This comparison is fast (milliseconds at million-scale indexes using HNSW) but fundamentally limited because the query and each chunk are embedded independently. The embedding model compresses each piece of text into a single vector, losing the fine-grained token-level interactions that determine whether a specific chunk actually answers a specific question.

Consider a query: "What is the maximum file size for uploads in the enterprise plan?" A vector search might return chunks about file uploads (relevant topic), the enterprise plan features (relevant topic), and generic storage limits (related topic). All three are semantically close to the query, so they get similar similarity scores. But only one chunk, the one that specifically states "Enterprise plan uploads are limited to 500 MB per file," actually answers the question. The vector search cannot distinguish this chunk from the others because the distinction requires understanding the relationship between the query's specific requirements and each chunk's specific content, which requires joint processing of both texts.

This is exactly what a cross-encoder reranker does. Instead of embedding the query and chunk separately and comparing their vectors, a cross-encoder takes the concatenated query-chunk pair as input and processes both texts through a transformer together, allowing full attention between query tokens and chunk tokens. The output is a single relevance score that reflects how well this specific chunk answers this specific query. The cross-encoder sees that "Enterprise plan uploads are limited to 500 MB per file" directly addresses the question about maximum file size on the enterprise plan, while "Our platform supports file uploads up to various limits" does not.

The Retrieve-Then-Rerank Pattern

Reranking is always a two-stage process because cross-encoders are too slow to score every chunk in the index. The first stage uses vector search to retrieve a broad set of candidates, typically 20 to 50 chunks, which takes 5 to 20 milliseconds even at large scale. The second stage uses the reranker to score and re-order those 20 to 50 candidates, which takes 50 to 200 milliseconds depending on the model and the number of candidates. The top 3 to 5 results after reranking are passed to the LLM for generation.

The key insight is that the first stage casts a wide net with high recall but imperfect precision, and the second stage refines the ranking with high precision. Retrieving 50 candidates instead of 5 ensures that the correct chunk is almost certainly in the candidate set (recall@50 is typically 90%+ even for mediocre embeddings), and the reranker ensures it ends up at the top of the list (precision@5 after reranking is typically 15% to 25% higher than precision@5 from vector search alone).

The number of candidates to retrieve for reranking is a tunable parameter. Too few (under 10) and the correct chunk might not be in the candidate set, making reranking pointless. Too many (over 100) and the reranking step becomes a latency bottleneck because cross-encoders process each candidate sequentially (or in small batches). The sweet spot for most systems is 20 to 50 candidates: enough to ensure high recall, few enough to keep reranking under 200 milliseconds.

Cross-Encoder Rerankers

Cross-encoder rerankers are transformer models fine-tuned to predict relevance scores for query-document pairs. The most widely used models in production are:

Cohere Rerank (rerank-v3.5): An API-based reranker that scores up to 1,000 documents per call. Pricing is $2 per 1,000 search queries (each query can rerank up to 100 documents). It consistently ranks among the top rerankers on the BEIR and MTEB benchmarks and supports multilingual queries. The API adds 100 to 300 milliseconds of latency depending on the number of candidates. For most teams, this is the fastest path to production-quality reranking without managing model infrastructure.

Jina Reranker (jina-reranker-v2-base-multilingual): Available both as an API and as an open model for self-hosting. The self-hosted version runs on a single GPU and processes 50 candidates in about 80 milliseconds. It handles 8,192-token inputs, which means it can rerank longer chunks without truncation.

BGE Reranker (bge-reranker-v2-m3): A fully open-source cross-encoder from BAAI that runs locally with no API dependency. It scores well on multilingual benchmarks and handles chunks up to 8,192 tokens. Self-hosting on a T4 GPU costs about $0.25 per hour and can process 100 to 200 reranking queries per second, making it cost-effective at scale.

ms-marco-MiniLM-L-12-v2: A smaller, faster cross-encoder trained on the MS MARCO passage ranking dataset. It processes 50 candidates in about 30 milliseconds on a CPU, making it suitable for deployments where GPU infrastructure is unavailable. Accuracy is 5% to 10% lower than the larger models, but the speed advantage makes it a reasonable trade-off for latency-sensitive applications.

The choice between API-based and self-hosted rerankers depends on query volume. At under 10,000 queries per day, API-based rerankers (Cohere, Jina) are cheaper and simpler because you avoid GPU infrastructure costs. At over 50,000 queries per day, self-hosting becomes cost-effective because the per-query cost drops below the API pricing. The detailed model comparison is in the reranker comparison guide.

ColBERT: Late Interaction Reranking

ColBERT (Contextualized Late Interaction over BERT) is a hybrid between bi-encoder (fast, imprecise) and cross-encoder (slow, precise) architectures. It produces per-token embeddings for both the query and the document, then computes relevance by matching each query token to its most similar document token using a MaxSim operation. This is more precise than single-vector similarity (because it operates at the token level) but faster than a full cross-encoder (because the document embeddings can be precomputed and stored).

The practical advantage of ColBERT is that it can serve as both the initial retriever and the reranker in a single model, eliminating the two-stage architecture. ColBERTv2 and its successors (PLAID, RAGatouille) precompute and compress the per-token document embeddings during ingestion, enabling fast retrieval with reranker-quality precision. The trade-off is higher storage requirements: per-token embeddings are 10 to 50 times larger than single-vector embeddings, which increases database costs.

For RAG applications where both retrieval precision and latency are critical, ColBERT offers the best balance. It is particularly effective for technical documentation and legal text where term-level precision matters, because the per-token matching catches exact term matches that single-vector embeddings miss while still understanding semantic similarity.

LLM-Based Reranking

Instead of using a specialized reranker model, some systems use a general-purpose LLM to score relevance. The prompt provides the query and a list of candidate chunks, and the LLM outputs a relevance score or ranking for each candidate. GPT-4o mini and Claude 3.5 Haiku are cost-effective choices for this approach, adding $0.001 to $0.01 per reranking call.

LLM-based reranking has the advantage of reasoning about relevance rather than just measuring it. The model can handle complex relevance judgments that require understanding context, intent, and nuance beyond what a cross-encoder captures. It can also explain its ranking decisions, which is useful for debugging retrieval quality.

The disadvantage is latency and cost at scale. An LLM call adds 500 to 1,500 milliseconds, which is 3 to 10 times slower than a cross-encoder. At high query volumes, the cost accumulates quickly. LLM-based reranking is best suited for low-volume, high-value applications (legal research, medical Q&A) where accuracy is worth the latency and cost premium, or as a fallback for queries where the cross-encoder reranker produces low-confidence results.

Step-by-Step Integration

Adding reranking to an existing RAG pipeline requires minimal architectural changes because it slots in between two existing steps.

Step 1: Increase the candidate count. Change your vector search to return 20 to 50 candidates instead of 3 to 5. This is typically a single parameter change in the vector database query (top_k=50 instead of top_k=5). The additional retrieval time is negligible because HNSW search is sublinear.

Step 2: Add the reranker call. After vector search returns the candidates, pass each candidate with the query to the reranker model. The reranker returns a relevance score for each candidate. Sort the candidates by reranker score in descending order.

Step 3: Apply a score threshold. Discard candidates with reranker scores below a minimum threshold (the exact value depends on the model, but a common starting point is 0.3 on a 0-to-1 scale). This prevents low-relevance chunks from reaching the LLM even if they are in the top 5 by score, which happens when the knowledge base simply does not contain information relevant to the query.

Step 4: Take the top results. Pass the top 3 to 5 candidates (after reranking and thresholding) to the LLM for generation. This is the same prompt assembly step you already have, with no changes needed.

Measuring the Impact

The value of reranking should be measured, not assumed. Build an evaluation set with 50 to 100 query-passage pairs where you know the correct answer and which chunk contains it. Run the evaluation with and without reranking and compare:

Retrieval precision@5: What fraction of the top 5 results are relevant. Reranking typically improves this by 15% to 25%.

Mean Reciprocal Rank (MRR): The average position of the first relevant result. Reranking pushes relevant results to position 1 more consistently.

Answer faithfulness: Whether the LLM's answer is supported by the retrieved context. Better retrieval leads to better generation because the LLM receives more relevant context and less noise.

Latency: The end-to-end query time with reranking versus without. The reranking step should add no more than 200 milliseconds for cross-encoder models. If it adds more, reduce the candidate count or switch to a faster model.

If reranking does not improve precision@5 by at least 10% on your evaluation set, the vector search is already good enough for your content type and the reranker is adding latency without proportional quality improvement. This occasionally happens with small, focused knowledge bases where the embedding model is well-suited to the content and the number of semantically similar but irrelevant results is low.