LLM Caching: Reduce Latency and Cost in AI Applications
In This Guide
Why LLM Caching Matters
Every LLM API call costs money and takes time. A single GPT-4o request costs $2.50-$10.00 per million input tokens and takes 1-5 seconds to complete. Claude Sonnet processes requests at $3 per million input tokens and $15 per million output tokens. When your application handles thousands of users asking overlapping questions, you pay repeatedly for the same computation. LLM caching eliminates this waste by intercepting repeated queries before they reach the model.
Consider a customer support chatbot that handles 10,000 conversations per day. Roughly 30-60% of those conversations contain questions that are either identical or semantically equivalent to previous queries. Without caching, every single one triggers a full API call. With caching, only the first instance of each unique question hits the LLM, and subsequent matches resolve from cache in under 50 milliseconds. Over a month, that translates to 90,000-180,000 fewer API calls.
The cost math is straightforward. If your monthly LLM spend is $5,000 and your cache hit rate reaches 50%, you save $2,500 per month. At 70% hit rate, common for FAQ-heavy applications, monthly savings reach $3,500. The cache infrastructure itself typically costs $50-200 per month for Redis or a managed vector database, making the ROI immediate. For enterprise deployments spending $50,000-$200,000 monthly on LLM APIs, caching becomes a strategic necessity rather than an optimization.
Latency improvements matter even more for user experience. Users perceive responses under 200ms as instant. A cached response returns in 20-50ms. An uncached LLM response takes 1,000-5,000ms depending on output length and model choice. That difference determines whether your AI product feels responsive or sluggish. In e-commerce applications, every 100ms of latency correlates with measurable drops in conversion rate, making cache latency improvements directly revenue-impacting.
Rate limiting adds another dimension. Every major LLM provider enforces tokens-per-minute and requests-per-minute limits. At scale, these limits throttle your application during peak traffic. Caching reduces the number of requests that reach the provider, effectively multiplying your rate limit capacity by the inverse of your miss rate. A 60% hit rate means your effective rate limit is 2.5x higher than your actual allocation.
How LLM Caching Differs from Traditional Caching
Web developers are familiar with HTTP caching, database query caching, and CDN caching. LLM caching shares some principles with these systems but introduces unique challenges that require different thinking.
Traditional caches use exact key matching. A database query cache hashes the SQL statement and parameters, then checks for a matching hash. If the hash matches, the cached result is correct by definition because the query is identical. LLM caching with semantic matching abandons this certainty. Two queries with different text can produce the same answer, but determining whether they will requires judgment, not hash comparison. This introduces a false positive risk that does not exist in traditional caching.
Cache invalidation in traditional systems follows clear rules. When a database row updates, you invalidate all cached queries that depend on that row. The dependency graph is explicit and deterministic. LLM cache invalidation is murkier. If a company changes its return policy, which cached responses need clearing? Any query that touches return policies, but there is no schema that maps questions to knowledge domains automatically. You must build this mapping yourself through tagging, TTL policies, or event-driven invalidation hooks.
Response variability is another difference. Traditional caches assume the same input always produces the same output. LLM responses vary by temperature setting, model version, and context window contents. Two identical prompts sent one minute apart might produce different phrasings of the same answer. This means caching locks in one specific phrasing of a response, which is usually acceptable for factual queries but problematic for creative or varied-output applications.
Storage size per entry is larger in LLM caching. A typical cached web response might be 50-200 KB. A cached LLM response includes the full prompt (often 1-10 KB), the response text (500 bytes to 10 KB), the prompt embedding vector (1-6 KB), and metadata. At millions of entries, storage costs and vector index performance become significant planning factors.
Types of LLM Caches
LLM caching operates at multiple levels, each with different tradeoffs between hit rate, accuracy, and implementation complexity.
Exact Match Caching
The simplest approach stores responses keyed by the exact input prompt. If the same prompt arrives again, byte for byte, the cached response returns immediately. Implementation takes a few lines of code: hash the prompt, check your key-value store, return on hit or call the LLM on miss. This works well for structured queries, API tool calls, and applications where users submit requests through fixed templates or form interfaces. The hit rate depends entirely on how often your users submit perfectly identical prompts, which in most conversational applications is only 5-15% but can reach 40-60% in structured search interfaces where queries come from predefined patterns.
Semantic Caching
Semantic caching uses vector embeddings to find cached responses for queries that mean the same thing but use different words. "What is the return policy?" matches against "How do I return an item?" because both questions map to nearby points in embedding space. This dramatically increases hit rates to 30-70% for most applications, but introduces complexity around similarity thresholds and edge cases where semantically close embeddings produce wrong answers. The embedding step adds 10-30ms of latency to every request (hit or miss), and you need a vector database to store and search embeddings efficiently. Despite this overhead, semantic caching delivers the highest ROI for most conversational AI applications.
Prompt Caching (Provider-Level)
Anthropic, OpenAI, and Google offer server-side prompt caching that reduces costs for repeated system prompts and context prefixes. This is not response caching in the traditional sense. You still make a full API call and pay for output token generation, but input tokens for cached prefixes cost 75-90% less. Anthropic charges 10% of the normal input price for cache-hit tokens. OpenAI offers 50% discount on cached prompt tokens. This is most valuable when you send long system prompts, extensive tool definitions, or context documents with every request. A 4,000-token system prompt that repeats across all requests saves $2.70 per million requests with Claude at the discounted rate.
KV Cache (Inference-Level)
For self-hosted models running on your own GPUs, the key-value cache stores intermediate attention computations during inference. When a new request shares a common prefix with a previous request (same system prompt, same few-shot examples), the model skips recomputing attention for those shared tokens. This is automatic in most inference frameworks like vLLM and TensorRT-LLM. It reduces latency and GPU utilization but only applies to self-hosted deployments, not API calls to providers.
Hybrid Approaches
Production systems typically combine multiple caching layers. The first layer checks exact match (fastest, highest confidence, lowest hit rate). The second layer checks semantic similarity (moderate speed, high hit rate, requires threshold tuning). The third layer applies provider-level prompt caching to reduce costs on cache misses. Each layer catches requests that slip through the layer above. A well-tuned three-layer system can reduce effective LLM costs by 70-85% compared to no caching, with the combined hit rate exceeding what any single strategy achieves alone.
How Semantic Caching Works
Semantic caching converts each incoming query into a vector embedding, then searches a vector store for previously cached queries within a similarity threshold. When a match is found above the threshold (typically 0.92-0.97 cosine similarity), the system returns the cached response instead of calling the LLM.
The process follows five stages. First, the incoming query is embedded using a model like OpenAI text-embedding-3-small, Voyage AI voyage-3, or an open-source model like BGE or E5. Second, the cache backend performs a nearest-neighbor search against all previously stored query embeddings, typically returning the top 1-3 closest matches. Third, the system evaluates whether the closest match exceeds the configured similarity threshold. Fourth, if the threshold is met, the cached response returns to the caller with a cache-hit flag for logging. Fifth, if no match exceeds the threshold, the query proceeds to the LLM, and the new query-response pair stores in cache for future requests.
The similarity threshold is the most critical parameter in the entire system. Set it too low (0.85) and you return wrong answers for queries that are merely topically related but ask different questions. Set it too high (0.99) and you only match near-identical phrasings, reducing your effective hit rate to barely above exact match levels. Most production systems start at 0.95 and tune based on observed error rates over a 5-7 day evaluation period. Different query categories often need different thresholds: factual lookups tolerate 0.92 while nuanced advisory questions need 0.96 or higher.
Embedding model choice affects both quality and cost. Smaller models (384 dimensions) are faster and cheaper to store but less precise at distinguishing subtle meaning differences. Larger models (1536+ dimensions) catch nuances like negation and conditional phrasing but cost more per embedding call and require more storage and index memory. For caching purposes, medium-dimension models (768-1024) typically offer the best tradeoff between precision and efficiency. The embedding call itself takes 5-20ms for short queries, adding minimal latency to the request path.
Edge cases require special handling. Negation is the most common problem: "How do I enable dark mode?" and "How do I disable dark mode?" have very high embedding similarity but require opposite answers. Time-sensitive queries need metadata checks: "What is the weather today?" changes daily. Context-dependent queries where the answer depends on who is asking need tenant or session metadata as additional cache key components beyond just the semantic match.
Cache Architecture Decisions
Building an LLM cache requires decisions about storage backend, TTL policies, cache scope, invalidation strategy, and deployment topology.
Storage Backend Options
Redis with the vector similarity module (Redis Stack or Redis Enterprise) handles both exact match and semantic caching in a single system. It keeps everything in memory for sub-millisecond lookups and supports both hash-based exact match and HNSW-based vector similarity search. The limitation is RAM: at 10 million cached entries with 768-dimension embeddings, you need roughly 30-40 GB of RAM just for vectors, plus the response text storage. Redis is the best choice for applications under 5 million cached entries where latency requirements are strict.
Dedicated vector databases like Qdrant, Weaviate, or Milvus provide disk-backed storage with comparable query speed for semantic matching. They handle tens of millions of vectors efficiently, offer built-in filtering (match by tenant, timestamp, category), and scale horizontally. The tradeoff is added infrastructure complexity and slightly higher query latency (5-20ms vs 1-5ms for Redis). These are the right choice for large-scale multi-tenant applications.
PostgreSQL with pgvector works for smaller deployments where you want to avoid adding new infrastructure. If your application already uses Postgres, adding a vector column to a cache table keeps your stack simple. Performance is acceptable up to a few hundred thousand entries with proper indexing (IVFFlat or HNSW), but degrades at larger scales compared to purpose-built vector databases.
For exact-match-only caching, any key-value store works: Redis, Memcached, DynamoDB, or even an in-process dictionary for single-instance deployments. The simplicity of exact match means storage choice barely matters until you reach millions of entries.
TTL and Eviction Policies
Not all cached responses remain valid forever. Product prices change, documentation updates, real-time data shifts. Time-to-live (TTL) settings determine how long cached responses remain valid. Static knowledge like definitions, explanations, and how-to content can cache for days or weeks. Dynamic information like stock prices, weather, or inventory might need TTLs of minutes or hours. Most systems use tiered TTLs based on content classification: 7 days for stable knowledge, 24 hours for moderately dynamic content, 1 hour for rapidly changing data, and no-cache for real-time requirements.
Eviction strategies determine what leaves the cache when storage fills up. LRU (least recently used) eviction removes entries that haven't been accessed recently. LFU (least frequently used) keeps popular entries longer. For LLM caches, a weighted approach works best: entries with high access frequency and recent access time survive longest, while stale low-frequency entries evict first. Set a maximum cache size based on your storage budget and monitor eviction rates to determine if you need to scale up.
Cache Scope and Isolation
Multi-tenant applications must decide whether caches are shared across all users or isolated per tenant. Shared caches produce higher hit rates because all users contribute to and benefit from the same cache pool. If 500 different users ask "What does this product do?", one cached answer serves all of them. Isolated caches prevent information leaking between users but reduce hit rates proportionally, since each tenant builds their cache independently. The hybrid approach caches generic queries globally and tenant-specific queries in isolated pools, giving you both the hit rate benefits of sharing and the privacy guarantees of isolation.
Deployment Topology
The cache can sit in three positions relative to your application. Inline caching places the cache directly in the request path: the application checks cache before calling the LLM, with no external service involved. This is simplest but couples cache logic to application code. Sidecar caching runs the cache as a separate service that your application calls via HTTP or gRPC. This decouples cache logic and lets you scale cache infrastructure independently. Proxy caching intercepts all LLM API calls through a transparent proxy that handles caching invisibly to the application. This requires no application code changes but offers less control over caching behavior per request type.
Cache Warming
Starting with an empty cache means every request misses initially, creating a cold-start period where users see full LLM latency and you pay full cost. Cache warming pre-populates the cache with known frequent queries before launch. Analyze your query logs from the past 30 days, identify the top 200-500 most common questions, generate responses for each by calling the LLM in batch, and insert them into cache. This gives you day-one cache hit rates of 20-40% instead of starting from zero. Re-warm periodically (weekly or after model updates) to keep pre-cached responses fresh.
Caching Patterns by Application Type
Different application types exhibit different query repetition patterns, which determines which caching strategy works best and what hit rates to expect.
Customer Support Chatbots
Support bots have the highest caching potential because customers ask the same questions repeatedly. "Where is my order?", "What is your return policy?", and "How do I reset my password?" appear thousands of times per day with minor wording variations. Semantic caching with a 0.93 threshold typically achieves 50-70% hit rates. The primary challenge is ensuring cached answers include personalized elements (order status, account details) only for generic queries, not personalized ones. Separate your query classification into "generic" (cacheable) and "personalized" (must call LLM with user context) before checking cache.
Documentation Assistants
AI assistants that answer questions about product documentation have moderate repetition. Different users explore similar topics but often ask progressively more specific questions. Hit rates of 30-50% are typical. The invalidation challenge is significant: when documentation updates, every cached response about the changed topic needs clearing. Build a mapping between documentation sections and cached query embeddings so you can selectively invalidate when specific pages update.
Code Generation Tools
Code assistants like autocomplete and explanation tools show variable caching potential. Simple completions like boilerplate code, import statements, and common patterns repeat frequently and cache well (40-60% hit rate). Complex code generation requests are highly context-dependent, referencing the user's specific codebase, and rarely match previous queries (5-10% hit rate). A useful hybrid is caching at the function-signature level: "generate a Python function to parse CSV" maps to a cached template regardless of the surrounding context.
Enterprise Search and Q&A
Internal knowledge base assistants see moderate repetition within departments. Sales teams ask similar questions about product features, engineering teams ask similar questions about architecture decisions. Hit rates of 25-45% are common. The multi-tenant challenge is pronounced here: answers may differ by role, department, or permission level. Cache keys should incorporate the user's access context to prevent returning answers that reference documents the current user cannot access.
Creative and Content Generation
Applications generating marketing copy, social media posts, or creative writing benefit least from response caching because users expect varied output. However, prompt caching at the provider level still helps because system prompts (brand voice guidelines, format instructions, examples) repeat across every request. Focus on provider-level prompt caching rather than response caching for creative workloads, and use response caching only for the analytical parts of your pipeline (sentiment classification, topic extraction) where deterministic output is expected.
Measuring Cache Performance
Four metrics determine whether your LLM cache is working effectively and where to invest in improvements.
Hit rate measures what percentage of incoming queries resolve from cache. Track this daily, by hour-of-day, and by query category. A healthy semantic cache achieves 30-70% overall hit rate depending on workload repetitiveness. FAQ bots hit 60-80%. Creative writing assistants hit 5-15%. If your hit rate plateaus below expectations, investigate whether your similarity threshold is too strict, your embedding model is not capturing semantic similarity well enough, or your query distribution genuinely has low repetition.
Accuracy (precision) measures how often cached responses correctly answer the incoming query. With semantic caching, not every cache hit produces the right answer because similar questions sometimes have different answers. Monitor user feedback signals: thumbs-down clicks, follow-up rephrasing patterns, and session abandonment after cached responses. Calculate your false positive rate weekly. Keep accuracy above 95% by raising similarity thresholds for categories that show high error rates. A cache that serves wrong answers quickly is worse than no cache at all.
Latency distribution shows the full picture of response times. Track p50, p95, and p99 latencies separately for cache hits and cache misses. Your cache hit latency should be 10-50ms at p95. If it exceeds 200ms, your cache infrastructure is undersized, your embedding model is too slow for real-time use, or your vector index needs rebuilding. Cache miss latency includes the cache lookup overhead (10-30ms) plus the full LLM call, so it will always be slightly higher than direct LLM calls without caching. Keep this overhead under 50ms so cache misses do not feel slower than a non-cached system.
Cost per query combines all factors into a single metric. Calculate: (cache infrastructure monthly cost + LLM cost for cache misses) / total queries. Compare this against your pre-caching cost per query. This number should drop 40-80% after caching is fully deployed. Track it monthly to demonstrate ROI to stakeholders and to detect regressions when model pricing changes or hit rates drop.
Build a dashboard showing these four metrics with daily granularity. Alert on sudden drops in hit rate (may indicate a new traffic pattern or cache corruption), spikes in latency (infrastructure issue), or drops in accuracy (threshold may need tightening after a model update changes embedding similarity patterns).
Production Considerations
Moving LLM caching from prototype to production introduces challenges around consistency, observability, safety, and graceful degradation that do not appear in development environments.
Cache poisoning occurs when an incorrect or harmful response enters the cache and serves repeatedly to many users. A single hallucinated answer that gets cached can affect thousands of users before detection. Implement quality gates before caching: verify responses pass basic validation rules, check for known hallucination markers (fabricated URLs, invented statistics, confident assertions about uncertainty), and consider only caching responses that pass an automated quality check or receive positive user feedback. Some teams cache all responses but mark them as "unverified" until they accumulate a threshold of positive interactions.
Stale responses become problematic when underlying facts change but cached answers remain. A customer asks about pricing, gets a cached answer from last week, but prices changed yesterday. Build invalidation hooks that clear relevant cache entries when source data updates. For RAG applications, maintain a mapping between source documents and the cached queries those documents informed, then invalidate the relevant cache entries whenever a source document changes. For time-sensitive domains, combine aggressive TTLs (hours, not days) with event-driven invalidation as a safety net.
Cache stampedes happen when many identical requests arrive simultaneously for an uncached query, often after a cache invalidation event or during traffic spikes around trending topics. Without protection, all requests hit the LLM concurrently, potentially overwhelming your rate limits and multiplying costs. Implement request coalescing: the first request triggers the LLM call, and subsequent identical requests subscribe to that pending result rather than each triggering their own call. When the single LLM call completes, all waiting requests receive the same response simultaneously. This is also called "single-flight" or "request deduplication" in infrastructure terminology.
Model version drift creates subtle problems. When you upgrade your LLM from one model version to another, cached responses from the old model may differ in style, accuracy, or format from fresh responses from the new model. Users might see inconsistent behavior where some answers feel different from others. After model upgrades, consider a cache flush or gradual invalidation rather than serving stale responses from the previous model indefinitely. The same applies to embedding model upgrades: a new embedding model may produce different similarity scores, requiring threshold recalibration.
Observability requires logging every cache decision in a queryable format. For each request, record: whether it hit or missed, the similarity score if semantic, which cached entry it matched, the latency of cache lookup and embedding generation, the total end-to-end latency, and whether the user gave positive or negative feedback on the response. This data powers threshold tuning, capacity planning, and debugging when users report receiving wrong answers. Store at least 30 days of these logs for trend analysis.
Graceful degradation ensures your application still works if the cache becomes unavailable. Design the cache as an optimization layer, not a hard dependency. If cache lookup fails, if the vector database is down, or if lookup times out (set a 100ms timeout on cache operations), fall through to the LLM directly. Users experience higher latency and you pay more, but the application never returns errors due to cache infrastructure problems. Test this fallback path regularly by intentionally disabling the cache during off-peak hours.
Privacy and compliance require attention in regulated industries. Cached responses may contain personally identifiable information from the original query context. Ensure your cache respects data deletion requests (GDPR right to erasure), does not serve one user's personal data to another user (tenant isolation), and stores cached data in approved geographic regions (data residency). Build automated scanning that flags cached entries containing PII patterns and purges them according to your retention policy.
Implementation Roadmap
Implementing LLM caching works best as a phased rollout rather than a single large deployment. Each phase builds on the previous one and delivers measurable value independently.
Phase 1: Exact Match with Provider Caching
Start with the simplest approach. Add exact match caching using Redis or any key-value store, keyed by a hash of the full prompt. Simultaneously enable provider-level prompt caching (Anthropic cache_control headers, OpenAI automatic caching) for your system prompts. This phase takes minimal code changes, introduces no accuracy risk, and typically delivers 10-25% cost reduction immediately. Use this phase to build your cache logging infrastructure and establish baseline metrics for hit rate, latency, and cost per query.
Phase 2: Semantic Caching for High-Volume Categories
Identify your highest-volume query categories from Phase 1 logs. Add semantic caching for those specific categories only, not your entire traffic. This limits blast radius if thresholds are miscalibrated. Choose an embedding model, deploy a vector store, set initial thresholds at 0.95, and run in shadow mode for one week (compute semantic matches but still call the LLM, comparing cached vs fresh responses to measure accuracy before going live). After validation, enable semantic cache serving for those categories. Expected additional cost reduction: 20-40% on top of Phase 1 savings.
Phase 3: Full Deployment with Monitoring
Expand semantic caching to all query categories, implement per-category thresholds based on Phase 2 learnings, add cache warming for known high-frequency queries, build invalidation hooks connected to your data update pipelines, and set up alerting on accuracy drops. This phase is ongoing optimization rather than a one-time deployment. Continuously tune thresholds, expand warming queries, and refine invalidation rules as your application evolves. Target: 50-75% total cost reduction with sub-50ms cache hit latency at p95.