How to Build a Semantic Cache for LLM Responses
A semantic cache intercepts LLM API calls, checks whether a similar query has been answered before, and returns the cached response when a match exceeds a similarity threshold. The core loop is simple, but building a system that works reliably in production requires attention to storage design, threshold calibration, and cache lifecycle management.
Step 1: Choose Your Embedding Model
The embedding model converts text queries into numerical vectors. Your choice affects cache accuracy, latency, and per-query cost. For most applications, start with a cloud-hosted model and consider self-hosting later if latency or cost becomes a concern.
For general-purpose applications: OpenAI text-embedding-3-small (1536 dimensions, $0.02/million tokens) provides reliable similarity scores across domains. It handles customer support queries, documentation questions, and general knowledge queries well. Embedding a typical user query (20-50 tokens) costs roughly $0.000001.
For technical and code-related applications: Voyage AI voyage-3 or voyage-code-3 produces better similarity scores for programming-related queries where standard models sometimes conflate different technical concepts.
For latency-critical applications: Self-host a smaller model like BGE-small-en-v1.5 (384 dimensions) on a GPU. Embedding latency drops from 15-20ms (API call) to 2-3ms (local inference). The accuracy tradeoff is minor for cache matching, where you need to distinguish "same question" from "different question," not subtle semantic differences.
One important design decision: embed only the user query, not the full prompt including system instructions. The system prompt is typically identical across requests, so including it inflates the embedding with shared content that does not help distinguish between different user questions. If your application uses different system prompts for different contexts, include a context identifier as a separate cache key component rather than embedding the full system prompt.
Step 2: Set Up the Vector Store
The vector store needs to support three operations efficiently: insert a new embedding with associated metadata, search for the nearest neighbor to a query embedding, and delete or expire entries based on age or manual invalidation.
Redis with Redis Stack (recommended for most cases): Redis Stack includes vector similarity search built into the same instance you likely already use for exact match caching. Store embeddings as Redis hashes with vector fields. Create an HNSW index on the vector field for fast approximate nearest neighbor search. Advantages: single infrastructure dependency, sub-5ms search latency, familiar Redis operations for TTL and deletion. Limitation: all data lives in RAM, so storage costs scale linearly with cache size.
Qdrant (recommended for large caches): A purpose-built vector database that stores data on disk with in-memory indexing. Handles millions of vectors efficiently. Provides built-in filtering (search only within a specific tenant or time range), payload storage for the cached response, and automatic index optimization. Run it as a Docker container or use Qdrant Cloud. Best when your cache exceeds what fits comfortably in Redis RAM.
PostgreSQL with pgvector (recommended for simplicity): If your application already uses PostgreSQL, pgvector adds vector storage and search to your existing database. No new infrastructure required. Performance is adequate for caches under 500,000 entries with proper indexing (ivfflat or hnsw). Beyond that, dedicated vector stores outperform significantly.
Whichever store you choose, create an HNSW index from the start. HNSW provides sub-5ms search at millions of entries. Without an index, search degrades to linear scan, which becomes unusable beyond 10,000 entries.
Step 3: Build the Cache Middleware
The middleware sits between your application code and the LLM API client. It intercepts outgoing requests, checks the cache, and either returns a cached response or passes through to the API.
The lookup flow works in four stages. First, extract the user query from the request. Second, generate the embedding for the query. Third, search the vector store for the nearest cached embedding. Fourth, if the nearest match exceeds the similarity threshold, return the associated cached response.
The store flow activates on cache misses. After the LLM generates a response, store three things: the query embedding (for future similarity searches), the original query text (for debugging and monitoring), and the complete LLM response (to return on future hits).
Design the middleware as a wrapper around your LLM client. The application calls the wrapper exactly as it would call the LLM client directly. The wrapper transparently handles cache logic. This design principle is important: if you ever need to disable caching (for debugging, A/B testing, or cache maintenance), you simply bypass the wrapper without changing any application code.
Include a bypass mechanism from the start. Some requests should never hit cache: requests with temperature > 0 where the user expects varied output, requests that include user-specific context that makes the response unique, and requests during testing or evaluation where you need fresh LLM output. Implement this as a flag (cache_enabled=false) on the request or as a list of query patterns that skip cache.
Step 4: Configure the Similarity Threshold
The threshold determines the minimum cosine similarity score for a cache hit. This single parameter controls the tradeoff between hit rate (lower threshold = more hits) and accuracy (higher threshold = fewer false positives).
Initial setting: 0.95. This is conservative enough to avoid most false positives while still catching obvious paraphrases. "What is your return policy?" (similarity 0.96 with "How do I return an item?") hits. "What is your return policy?" (similarity 0.89 with "What is your shipping policy?") misses. This is the right starting point for any application.
Tuning process: After one week of production data, review all cache hits with similarity scores between 0.90 and 0.96. For each hit, evaluate whether the cached response correctly answered the query. Calculate the false positive rate at each 0.01 threshold increment. Find the threshold where false positive rate stays below your tolerance (typically 2-5%). Adjust and monitor.
Per-category thresholds: If your application handles multiple query types, different types may need different thresholds. FAQ-style questions tolerate lower thresholds (0.92) because the answer space is small and well-defined. Technical troubleshooting questions need higher thresholds (0.96) because subtle differences in the problem description require different solutions. Implement threshold routing by classifying the incoming query type before cache lookup.
Log every cache decision with the similarity score regardless of hit or miss. This data is essential for threshold tuning and debugging. Include: the incoming query text, the matched cached query text (if any), the similarity score, whether it was classified as hit or miss, and the threshold used.
Step 5: Add Cache Management
A semantic cache without management grows unboundedly, serves stale answers, and becomes impossible to debug. Implement these four management mechanisms before going to production.
TTL (time-to-live): Set a default TTL on every cache entry. For most applications, 7-14 days is a reasonable default for factual content. For rapidly changing information, use hours or minutes. Implement TTL at the storage level: Redis has native key expiration, Qdrant supports payload-based filtering by timestamp, pgvector can filter by a created_at column.
Size limits: Cap the total number of cache entries. When the limit is reached, evict the least recently used (LRU) or least frequently hit entries. A cache with 100,000 well-chosen entries typically performs as well as one with 1,000,000 entries because the distribution of queries follows a power law: a small number of common queries account for most of the traffic.
Invalidation hooks: Build API endpoints or event listeners that invalidate specific cache entries when underlying data changes. If your product documentation updates, invalidate all cache entries tagged with that documentation source. If a policy changes, invalidate entries related to that policy. Tag each cache entry with metadata about its source to enable targeted invalidation.
Quality gates: Not every LLM response should enter the cache. Implement checks before storing: verify the response is not an error message, check that the response length exceeds a minimum (very short responses are often error or refusal messages), and optionally require the response to pass a relevance check against the original query.
Step 6: Warm the Cache
Starting with an empty cache means 0% hit rate on day one. Cache warming pre-populates entries for your most common queries so you achieve meaningful hit rates immediately.
Gather your top 200-500 most frequently asked queries from historical logs, analytics, or support ticket data. Generate LLM responses for each query. Embed each query and store the query-response pairs in the cache.
If you do not have historical query data, use your FAQ page, documentation table of contents, and common user flows to generate a seed set of queries. Have team members brainstorm the 50-100 most likely questions a user would ask. Even a modest warming set produces noticeable day-one improvement.
After warming, verify the cache works correctly by testing a sample of paraphrased versions of your seeded queries. "What is your pricing?" should hit the cache if you seeded "How much does it cost?" Adjust your threshold if expected matches are missing.
Common Implementation Mistakes
Caching the full prompt instead of just the user query. When the system prompt is identical across all requests (which it usually is), including it in the cache key or embedding wastes storage and reduces cache utility. Only embed and match on the variable part of the request.
No cache bypass for testing. Without a way to skip the cache, you cannot evaluate whether the LLM's answers have improved after prompt changes. Always include a bypass flag for development and A/B testing.
Ignoring cache stampedes. When your cache is cold or an entry expires, multiple identical requests may arrive simultaneously, each triggering its own LLM call. Implement request coalescing: the first request triggers the LLM call, and subsequent identical requests wait for that single response.
No monitoring from day one. If you do not measure hit rate, accuracy, and latency from launch, you cannot tune the system. Add metrics before writing the first line of cache logic. Instrument everything: hits, misses, similarity scores, latencies, cache size.
Build your semantic cache in layers: embedding model, vector store, middleware wrapper, threshold configuration, management policies, and cache warming. Start with a 0.95 similarity threshold and tune from production data. The most common mistakes are not monitoring from day one and caching the full prompt instead of just the user query.