How to Reduce RAG Latency Below 500 Milliseconds

Where the Time Goes

A typical RAG pipeline has five sequential stages, each contributing to the total latency. Before optimizing, instrument each stage with timing so you know where the actual bottleneck is rather than guessing.

Query embedding (10 to 100ms): The user's query is sent to an embedding model (API call or local inference) to produce a vector. API-based embedding (OpenAI, Cohere) adds network round-trip time. Local embedding on a GPU takes 5 to 15 milliseconds for a single query.

Vector search (1 to 50ms): The vector database performs approximate nearest neighbor search to find the top-k most similar chunks. HNSW-based search on million-scale indexes typically completes in 1 to 10 milliseconds. Adding metadata pre-filters can increase this to 10 to 50 milliseconds depending on filter selectivity and database implementation.

Reranking (50 to 300ms): If using a cross-encoder reranker, each candidate chunk is scored against the query. This is the slowest retrieval stage because cross-encoders process each candidate sequentially. API-based rerankers (Cohere) add network latency. Self-hosted rerankers on GPU are faster but still the retrieval bottleneck.

Prompt assembly (1 to 5ms): String formatting and template rendering. Negligible in isolation but can spike if the assembly involves additional database lookups for metadata or conversation history.

LLM generation (200 to 2000ms to first token): The LLM processes the assembled prompt and begins generating. Time to first token depends on prompt size (more input tokens = longer prefill), model size, and provider load. This is typically the largest component and the one you have least control over when using API-based models.

The total latency is the sum of all five stages because they run sequentially. A typical unoptimized pipeline: 80ms embedding + 15ms search + 150ms reranking + 2ms assembly + 600ms generation = 847ms total. The target is to reduce this to under 500ms to first token, or under 300ms for the retrieval portion so that streaming generation can start quickly.

Optimize Query Embedding

Self-host the embedding model. API-based embedding adds 50 to 100 milliseconds of network round-trip time that self-hosting eliminates. A model like BGE-small or GTE-base running on a T4 GPU produces query embeddings in 3 to 8 milliseconds. The GPU cost ($0.25 to $0.50 per hour) is offset by eliminating per-query API charges and network latency at any meaningful volume.

Cache frequent query embeddings. If your application sees repeated or similar queries (which is common in customer support and internal knowledge bases), cache the embedding results. A simple LRU cache keyed by query text avoids recomputing embeddings for identical queries. For semantically similar queries, you can cache at a normalized form (lowercase, stripped punctuation, sorted terms) to increase cache hit rates. A 70% cache hit rate reduces average embedding time to under 5 milliseconds.

Use a smaller, faster model for queries. Query text is short (10 to 30 tokens), so a smaller embedding model processes it faster than a large one with minimal quality loss. OpenAI's text-embedding-3-small is faster than text-embedding-3-large for query embedding, and the retrieval quality difference is measurable but often within acceptable margins. The key is to use the same model for both ingestion and query embedding, so if you switch to a smaller query model, you must re-embed the entire corpus with the same model.

Optimize Vector Search

Tune HNSW parameters. The ef_search parameter controls the search beam width: higher values improve recall but increase latency. Most vector databases default to conservative values (ef_search=128 or higher) that provide excellent recall but are slower than necessary. For many knowledge bases, reducing ef_search to 64 or even 32 maintains 95%+ recall while cutting search time by 40% to 60%. Measure recall on your evaluation set at different ef_search values to find the sweet spot.

Use quantized vectors. Scalar quantization (SQ8) reduces each vector dimension from 32-bit float to 8-bit integer, cutting memory usage and search time by roughly 4x with minimal accuracy loss (typically under 1% recall reduction). Product quantization (PQ) compresses even further but with larger accuracy trade-offs. Qdrant, Milvus, and pgvector all support scalar quantization. For indexes over 1 million vectors, quantization is the single highest-impact search optimization.

Partition the index. If your knowledge base has natural partitions (by product, by department, by language), split the index into partitions and search only the relevant partition based on query context. Searching a 100,000-vector partition is faster than searching a 1,000,000-vector index, and the partition selection can be done by the query router at near-zero additional cost.

Keep the index in memory. Vector search performance degrades dramatically when the index does not fit in RAM and the database must read from disk. Ensure your vector database instance has enough memory to hold the entire index. For pgvector, this means setting shared_buffers and effective_cache_size appropriately. For managed services, choose an instance tier with sufficient memory for your vector count and dimensionality.

Optimize Reranking

Reduce the candidate count. Reranking 50 candidates takes 2.5x longer than reranking 20 candidates. If your evaluation shows that recall@20 is within 2% of recall@50, reduce the candidate count from 50 to 20 and save 60% of reranking time.

Use a faster reranker model. The ms-marco-MiniLM-L-6-v2 cross-encoder processes 20 candidates in about 15 milliseconds on a CPU, compared to 80 milliseconds for the larger bge-reranker-v2-m3 on the same hardware. The accuracy difference is 3% to 5% on standard benchmarks. For latency-sensitive applications, the faster model is often the right trade-off.

Batch the reranker inference. Process all query-candidate pairs in a single batch rather than sequentially. GPU-based rerankers benefit significantly from batching because the parallel execution units are utilized more efficiently. A batch of 20 candidates on a T4 GPU takes roughly the same time as processing 3 to 5 candidates sequentially.

Skip reranking for high-confidence results. If the top vector search result has a similarity score significantly higher than the second result (a gap of 0.1 or more), the ranking is likely correct and reranking will not change it. Skip the reranker for these clear-winner queries and only apply it when the top results have similar scores (indicating uncertainty in the ranking). This reduces average reranking cost by 30% to 50% depending on your content.

Optimize Generation

Reduce prompt size. Time to first token scales with input token count because the model must process the entire prompt before generating. Passing 3 highly relevant chunks instead of 8 mixed-quality chunks reduces input tokens by 50% to 60% and proportionally reduces time to first token. Better retrieval precision (via reranking and metadata filtering) enables passing fewer, higher-quality chunks without sacrificing answer quality.

Use prompt caching. Anthropic's prompt caching and OpenAI's equivalent let you cache the static portions of the prompt (system instruction, few-shot examples) so the model only processes the dynamic portions (retrieved chunks and user query) on each request. This reduces time to first token by 30% to 50% for prompts where the static portion is large. The cache persists for 5 minutes (Anthropic) or longer, so repeated queries within a session benefit from the cached prefix.

Stream the response. Streaming does not reduce total generation time, but it delivers the first token to the user 200 to 500 milliseconds sooner, making the system feel responsive even while generation continues. In chat interfaces, streaming is the single most impactful perceived-latency optimization. Every major LLM API supports streaming (server-sent events or WebSocket), and most client frameworks (LangChain, LlamaIndex, Vercel AI SDK) handle stream rendering out of the box.

Choose the right model size. GPT-4o mini generates the first token in 150 to 300 milliseconds compared to 300 to 600 milliseconds for GPT-4o. Claude 3.5 Haiku is faster than Claude 3.5 Sonnet. For RAG applications where the answer is largely determined by the retrieved context rather than the model's reasoning ability, a smaller, faster model often produces answers of comparable quality. Test on your evaluation set: if the smaller model matches the larger model's faithfulness score, use the smaller model.

Parallelize Where Possible

Some pipeline stages can run in parallel rather than sequentially, cutting total latency.

Embed the query while loading conversation history. The query embedding and the conversation history database lookup are independent operations. Running them in parallel saves the duration of whichever is shorter (typically 10 to 50 milliseconds).

Run vector search and keyword search in parallel. In hybrid search, the vector search and BM25 search query different indexes and do not depend on each other. Running them concurrently and merging results after both complete reduces hybrid search latency to the duration of the slower search rather than the sum of both.

Prefetch chunks for likely follow-up queries. In conversational RAG, predict the likely next query based on the current answer and pre-retrieve chunks for it. When the user sends the actual follow-up, the chunks are already in cache. This speculative prefetching adds background load but eliminates retrieval latency for predicted queries. Accuracy depends on how predictable your users' follow-up patterns are.

Latency Budget Template

For a target of 300ms time to first token (allowing streaming to deliver the rest):

Query embedding: 5ms (self-hosted, cached)
Vector search: 5ms (tuned HNSW, quantized vectors)
Reranking: 40ms (fast model, 20 candidates, batched)
Prompt assembly: 2ms
Generation prefill: 200ms (small model, 2000 input tokens, streaming)
Total: 252ms

This is achievable with self-hosted components. For API-based pipelines, add 50 to 100ms for each network round-trip (embedding API, reranker API, LLM API), pushing the budget to 400 to 550ms, which is still acceptable for most interactive applications.