Cache Invalidation Strategies for AI Applications

Why LLM Cache Invalidation Is Harder Than Traditional Caching

In traditional web caching, invalidation is straightforward: when the underlying resource changes, clear the cached version. A product page updates, the CDN invalidates that URL. A database row changes, the query cache for that table clears. The relationship between source data and cached content is direct and traceable.

LLM caches break this simple relationship in three ways. First, the LLM's response synthesizes information from multiple sources, making it unclear which source changes should invalidate which cached responses. A cached answer about "best practices for deployment" might draw on documentation from five different sections, and any of those sections could update.

Second, semantic caching means a single cached response serves multiple different queries. Invalidating a cached response for one query affects all queries that match it semantically. You cannot invalidate for one phrasing without affecting all phrasings.

Third, the LLM itself changes. Model updates from providers can produce different, potentially better responses to the same prompts. Your cached responses from last month's model version may be worse than what the current version would generate. There is no change event you can listen for when the provider silently improves the model.

TTL-Based Expiration

Time-to-live expiration is the simplest and most common invalidation strategy. Every cache entry receives an expiration timestamp when stored. After that timestamp, the entry is no longer served and gets evicted.

The right TTL depends on how quickly the correct answer to a query changes. Use these guidelines as starting points:

Stable knowledge (TTL: 7-30 days): Definitions, explanations of concepts, historical facts, technical documentation for stable APIs. "What is machine learning?" does not change week to week. Long TTLs maximize cache value for these queries.

Moderately dynamic content (TTL: 1-7 days): Company policies, product features, pricing tiers, best practices. These change occasionally but not daily. A 3-day TTL ensures updates propagate within a few days while maintaining strong hit rates.

Frequently changing content (TTL: 1-24 hours): News summaries, trending topics, availability information, schedule data. Short TTLs prevent seriously stale responses while still providing caching benefit during peak traffic periods.

Real-time data (TTL: 0, do not cache): Stock prices, live scores, current weather, inventory counts. These change faster than any reasonable TTL. Exclude these query types from caching entirely rather than setting very short TTLs that provide negligible benefit.

Implement tiered TTLs by tagging cache entries with a content category at storage time. A simple keyword classifier can distinguish "What is your return policy?" (moderately dynamic, 3-day TTL) from "What is the weather today?" (real-time, do not cache). More sophisticated systems use the LLM itself to classify whether its response contains time-sensitive information.

Event-Driven Invalidation

Event-driven invalidation clears specific cache entries when the underlying data they depend on changes. This is more precise than TTL expiration because it invalidates exactly when needed, not on an arbitrary schedule.

Document update events: When a source document in your RAG pipeline updates, invalidate all cache entries that were generated using that document as context. This requires tagging each cache entry with the document IDs used in its generation. When document X updates, query the cache for all entries tagged with document X and remove them.

Configuration change events: When your company changes its return policy, pricing, or any factual information that cached responses reference, trigger a targeted invalidation. Maintain a mapping between business entities (policies, products, features) and the cache entries that reference them. A policy change event clears all related cache entries.

Feedback-driven invalidation: When users consistently mark a cached response as incorrect or unhelpful (thumbs down, "not helpful" clicks, immediate rephrasing), automatically invalidate that entry. Set a threshold: if a cache entry receives negative feedback from more than 10% of users served (with a minimum of 3 negative signals), invalidate it and flag the query for fresh generation.

Model version events: When you know the LLM provider has updated the model (Anthropic and OpenAI announce major model updates), consider invalidating your entire cache or at least the entries with the most traffic. The new model version may produce better responses that you want to capture.

Versioned Caching

Versioned caching adds a version number to the cache key. When your system prompt, RAG documents, or other context changes, you increment the version. All existing cache entries belong to the old version and effectively become invisible to lookups using the new version.

This approach is simple to implement: prepend a version string to your cache keys or include version in the metadata filter for vector searches. When version changes from "v7" to "v8," all v7 entries stop matching. You can keep old entries for rollback (if v8 performs worse, switch back to v7 immediately) or purge them asynchronously.

The downside is that version bumps invalidate the entire cache at once, even entries that would still be correct under the new version. Hit rate drops to zero after each version bump until the cache re-warms. For applications where you update system prompts frequently, this creates a sawtooth pattern: high hit rate that drops to zero repeatedly.

Mitigate this by combining versioning with selective re-validation. When version bumps, do not immediately purge old entries. Instead, for each cache hit against an old-version entry, generate a fresh response in the background and compare it to the cached response. If they match closely, promote the entry to the new version. If they differ, serve the fresh response and cache it as a new entry.

Partial Invalidation for RAG-Based Caches

When your LLM responses are grounded in retrieved documents (RAG), cache invalidation becomes more nuanced because a single cached response may depend on multiple source chunks.

Tag each cache entry with the chunk IDs or document IDs used in its generation. When any source chunk updates, identify all cache entries that used that chunk and invalidate them. This is more targeted than a full cache flush but requires consistent metadata tracking through the entire RAG pipeline.

Some teams implement a looser approach: when a document updates, invalidate all cache entries whose queries are semantically related to the document's topic, regardless of whether the specific entry used that document. This catches cases where the updated document should have been retrieved but was not included in the original generation due to ranking differences.

For large document collections with frequent updates (knowledge bases, wikis, changelog feeds), implement incremental invalidation rather than processing every update immediately. Batch document updates, identify affected cache entries once per batch, and invalidate in bulk. This reduces the operational overhead of event processing while keeping staleness bounded to the batch interval.

Cache Stampede Prevention During Invalidation

When you invalidate a popular cache entry, all subsequent requests for that query suddenly miss and hit the LLM simultaneously. For high-traffic queries, this can mean dozens or hundreds of concurrent identical LLM calls, which wastes money and can hit rate limits.

Prevent stampedes with these techniques:

Stale-while-revalidate: Continue serving the stale cached response while generating a fresh one in the background. Users get immediate (slightly stale) responses, and the cache updates without a traffic spike. This works well for TTL-based expiration but not for event-driven invalidation where the stale response is known to be incorrect.

Request coalescing: When the first cache miss arrives after invalidation, lock that cache key and queue subsequent identical requests. The first request generates the response, stores it, and then the queued requests receive the new cached response. This ensures only one LLM call per invalidated entry regardless of concurrent demand.

Probabilistic early expiration: Instead of all entries expiring at their exact TTL timestamp, add random jitter. An entry with a 24-hour TTL actually expires between 22 and 26 hours. This spreads expirations over time, preventing mass simultaneous invalidation of entries that were created around the same time.

Monitoring Invalidation Health

Track these metrics to verify your invalidation strategy works correctly:

Staleness rate: Sample cached responses periodically and compare to fresh LLM output. If more than 5-10% of cached responses differ materially from what the LLM would currently produce, your TTLs are too long or your event-driven invalidation is not catching all relevant changes.

Invalidation volume: Track how many entries are invalidated per day and why (TTL expiry, event trigger, feedback signal, manual). Sudden spikes in invalidation may indicate a misconfigured event trigger or an upstream data change that invalidated too broadly.

Re-warm latency: After a major invalidation event, measure how long it takes for hit rate to recover to pre-invalidation levels. If recovery takes too long, consider proactive cache warming after planned invalidations.

False invalidation rate: Some invalidation triggers may be too aggressive, clearing entries that are still valid. If your invalidation volume is high but your staleness rate was already low, you are likely over-invalidating. Narrow the invalidation criteria to reduce unnecessary cache churn.