Semantic Caching: Match by Meaning Not Exact Text

Why Exact Match Caching Falls Short

Exact match caching only works when two prompts are character-for-character identical. In reality, users express the same intent in dozens of different ways. A customer support system might receive these variations of the same question in a single day:

"What is your return policy?"
"How do I return something?"
"Can I send back an item I bought?"
"Returns policy please"
"I want to return my order"
"What are the rules for returns?"

All six queries need the same answer. Exact match caching treats each as a unique request, sending all six to the LLM. Semantic caching recognizes they share the same meaning and serves the cached response for all six after generating it once.

In production deployments, exact match caching typically achieves 5-15% hit rates for conversational applications. Semantic caching on the same workload achieves 30-70%. That difference represents thousands of dollars in monthly savings and a much better user experience.

The Embedding-Based Matching Process

Semantic caching works by converting text into numerical vectors (embeddings) that represent meaning. Queries with similar meanings produce vectors that are geometrically close together in the embedding space. The process has five stages.

Stage 1: Embed the incoming query. When a new request arrives, the system sends the query text to an embedding model (like OpenAI text-embedding-3-small, Voyage AI voyage-3, or Cohere embed-v4). The model returns a dense vector, typically 256-1536 dimensions, representing the semantic content of the query.

Stage 2: Search the cache index. The system performs an approximate nearest neighbor (ANN) search against all previously stored query embeddings. This search finds the cached entries whose embeddings are most similar to the incoming query embedding. With proper indexing (HNSW, IVF), this search completes in under 5 milliseconds even with millions of cached entries.

Stage 3: Apply the similarity threshold. The nearest match returns with a similarity score (cosine similarity between 0 and 1). If this score exceeds the configured threshold (typically 0.92-0.97), the system considers it a cache hit. If the score falls below the threshold, the system treats it as a cache miss.

Stage 4: Return or generate. On a hit, the cached response returns to the caller immediately. On a miss, the query proceeds to the LLM API for fresh generation.

Stage 5: Store new entries. After a cache miss produces a response, the system stores both the query embedding and the response in the cache for future matching.

Choosing the Right Embedding Model

The embedding model determines how well your semantic cache distinguishes between truly similar queries (that deserve the same answer) and merely related queries (that need different answers). Three factors matter: accuracy, dimensions, and cost.

OpenAI text-embedding-3-small (1536 dimensions): Good general-purpose accuracy, widely used, $0.02 per million tokens. Produces reliable similarity scores across diverse domains. The default choice for most applications starting with semantic caching.

OpenAI text-embedding-3-large (3072 dimensions): Higher accuracy for subtle distinctions, $0.13 per million tokens. Worth it when your queries contain nuanced differences that smaller models conflate. Requires more storage per entry.

Voyage AI voyage-3 (1024 dimensions): Strong performance on technical and code-related content. $0.06 per million tokens. Good choice for developer-facing applications and documentation assistants.

Cohere embed-v4 (1024 dimensions): Competitive accuracy with built-in support for different input types (search query vs document). $0.10 per million tokens. Its input-type awareness helps distinguish short queries from longer content.

Open source models (BGE, E5, GTE): Free to run but require self-hosted infrastructure. Latency depends entirely on your hardware. Best for organizations with GPU infrastructure already in place and strict data privacy requirements.

For caching purposes, smaller embedding dimensions (512-1024) usually suffice. Cache matching needs to determine "same question, different words" not "subtly different nuance in a legal document." The simpler distinction requires less embedding precision.

The Similarity Threshold Problem

The threshold setting determines your cache's tradeoff between hit rate and accuracy. This is the single most impactful parameter in any semantic cache implementation.

Threshold too high (0.98-0.99): Only matches near-identical phrasings. "What is your return policy?" matches "What's your return policy?" but not "How do I return something?" Hit rate barely exceeds exact match caching. You get high accuracy but minimal benefit.

Threshold too low (0.80-0.88): Matches broadly related queries that need different answers. "What is your return policy?" might match "What is your shipping policy?" because both are about company policies. You get high hit rate but unacceptable false positive rates. Users receive wrong answers.

The sweet spot (0.92-0.96): Most production systems operate in this range. At 0.94, "How do I return an item?" matches "I want to send something back" (same intent, different words) but does not match "How do I exchange an item?" (related but different intent). The exact optimal value depends on your embedding model and domain.

Start at 0.95 and tune based on production data. Monitor user feedback signals (thumbs down, immediate rephrasing, escalation to human) as indicators of false positives. Lower the threshold by 0.01 increments until you reach your acceptable error rate, typically 2-5% false positive rate for non-critical applications.

Handling Edge Cases in Semantic Caching

Pure semantic similarity can fail in predictable ways. Robust implementations handle these edge cases explicitly.

Context-dependent queries: "What is the status of my order?" means something different for every user. The text is semantically identical across users, but the correct response depends on who is asking. Solution: include user ID or session context in the cache key, or exclude personalized queries from semantic caching entirely.

Negation sensitivity: "How do I enable notifications?" and "How do I disable notifications?" are semantically very similar (cosine similarity often 0.93+) but require opposite answers. Solution: implement a negation detection layer that checks for opposing intent before serving cached responses, or raise the threshold specifically for how-to queries.

Temporal queries: "What was announced today?" has the same embedding regardless of when it is asked, but the correct answer changes daily. Solution: include the date in the cache key or exclude time-sensitive queries from caching.

Multi-part queries: "What is your return policy and do you offer free shipping?" combines two questions. It might partially match a cached response for just the return policy. Solution: implement query decomposition that splits compound queries before cache lookup, or only cache when similarity exceeds a higher threshold (0.97+) for longer queries.

Performance Characteristics

Semantic cache lookup adds latency compared to exact match caching, but the total is still dramatically faster than an uncached LLM call.

Embedding generation: 5-20ms for API-based models, 2-5ms for local models on GPU. This runs on every incoming request regardless of hit or miss.

Vector similarity search: 1-10ms depending on index size and algorithm. HNSW indexes maintain sub-5ms search even at millions of entries. Flat indexes become slow beyond 100,000 entries.

Total cache hit latency: 10-30ms for API-based embeddings, 5-15ms for local embeddings. Compare to 1,000-5,000ms for an uncached LLM API call.

Cache miss overhead: On a miss, you pay the embedding cost (5-20ms and $0.00002) plus the normal LLM call. This overhead is negligible compared to the LLM call itself.

For applications where even 10-30ms of additional latency matters, run the embedding model locally on a small GPU. A BGE-small model on an NVIDIA T4 embeds queries in under 3ms, reducing total cache hit latency to under 8ms.

Semantic Caching Architecture Patterns

Three architectural patterns dominate production semantic cache deployments.

Pattern 1: Inline middleware. The cache sits directly in the request path between your application and the LLM API. Every request passes through the cache layer. This is the simplest pattern, works well for single-service architectures, and adds minimal latency. Downside: the cache becomes a single point of failure if not properly redundant.

Pattern 2: Sidecar cache. The cache runs as a separate service that your application queries before making LLM calls. Your application explicitly calls cache.lookup() and cache.store(). This gives you more control over caching logic, allows different caching strategies for different query types, and isolates cache failures from your main application. Downside: more application code to maintain.

Pattern 3: Proxy cache. A transparent proxy intercepts all traffic to the LLM API, handles caching logic independently, and your application connects to the proxy instead of directly to the LLM API. This requires zero application code changes. GPTCache and similar tools operate in this mode. Downside: less fine-grained control over caching decisions.