Caching Strategies for AI Applications Explained

Prompt Caching

Prompt caching is the simplest and highest-ROI caching strategy because it requires no application changes beyond enabling it in your API configuration. Anthropic's prompt caching works by caching the processed computation of token sequences that appear at the beginning of your request. When a new request starts with the same token prefix as a cached request, the provider serves the cached computation at a 90 percent discount ($0.30 per million tokens instead of $3.00 for Claude Sonnet input).

The cache operates on prefixes, so the system prompt must be placed at the beginning of the message sequence. The cache has a 5-minute TTL, meaning that if no request uses the cached prefix within 5 minutes, it expires and the next request pays full price to re-cache. For applications with steady traffic (at least one request every 5 minutes), the cache stays warm and nearly every request benefits from cached pricing. For applications with bursty traffic (periods of high activity separated by quiet periods), the cache warms up during active periods and expires during quiet ones. The first request after a quiet period pays the full caching creation cost, but subsequent requests benefit immediately.

The savings math is straightforward. If your system prompt is 2,000 tokens and you make 100,000 requests per day with sustained traffic, prompt caching saves: 2,000 tokens per request, times 100,000 requests, equals 200 million tokens per day. At $3.00 per million, uncached cost is $600 per day. At $0.30 per million (cached), cost is $60 per day. Savings: $540 per day, or $16,200 per month. The only cost is the cache creation fee on the first request and any requests after a cache expiration, which is negligible at sustained traffic volumes.

Response Caching

Response caching stores the complete output of an API call and returns it for identical future requests without making an API call at all. Unlike prompt caching, which happens on the provider's infrastructure, response caching is implemented in your application using a cache backend like Redis, Memcached, or an in-memory store.

The cache key is typically a hash of the full request payload: the model, system prompt, messages, tools, temperature, and any other parameters. When a new request matches a cached key, the cached response is returned in milliseconds instead of the 1 to 5 seconds a model API call would take. This eliminates both the cost and the latency of the API call.

Response caching effectiveness depends on query repetition rate. Applications with high repetition (FAQ bots, documentation assistants, classification systems) achieve 30 to 50 percent cache hit rates. Applications with low repetition (creative writing, personalized analysis, open-ended conversation) achieve near-zero hit rates and should not invest in response caching. The key metric to measure before implementing is the percentage of requests that are exact duplicates of previous requests over a one-week period. If it exceeds 15 percent, response caching is worth implementing.

Semantic Caching

Semantic caching extends response caching by matching on meaning rather than exact text. Two requests with different wording but the same intent, such as "What is your refund policy?" and "How do I get a refund?", can share a cached response. This increases hit rates by 2x to 3x compared to exact matching, dramatically expanding the effectiveness of caching for applications with diverse user phrasing.

Implementation requires embedding each query and searching for cached responses to semantically similar queries. When a new query's embedding has cosine similarity above a threshold (typically 0.93 to 0.97) to a cached query embedding, the cached response is returned. The embedding computation adds a small cost ($0.0001 per query at current embedding prices) and latency (5 to 20 milliseconds), both of which are negligible compared to the full API call they replace on cache hits.

The risk of semantic caching is false matches: queries that look similar but require different answers. "How do I cancel my subscription?" and "How do I cancel my order?" are semantically similar but have different correct answers. The similarity threshold controls this trade-off. A high threshold (0.97) misses some valid matches but almost never returns incorrect responses. A low threshold (0.90) catches more matches but introduces errors. The optimal threshold depends on how much variation exists in correct responses for semantically similar queries. Start high and lower gradually while monitoring quality.

Embedding Caching

Embedding caching stores the vector representations of text to avoid redundant embedding API calls. If the same document chunk or query has been embedded before, the cached embedding is returned instead of making another API call. This is most valuable for RAG applications that embed user queries on every request: if a user asks the same or similar question multiple times, the embedding call can be served from cache.

For document embeddings, caching is typically handled at the ingestion level (embeddings are computed once and stored in the vector database). Query embedding caching is an additional layer that stores embeddings of recent queries in a fast cache (Redis or in-memory) keyed by the query text hash. The savings are smaller per-call than response caching ($0.0001 per embedding vs $0.01 to $0.10 per LLM call), but at high query volumes, they accumulate. An application making 500,000 queries per day saves roughly $50 per day from embedding caching, or $1,500 per month.

Layering Caches for Maximum Impact

The four caching strategies are complementary, not competing. Each operates at a different level and catches different types of redundancy. A layered caching architecture processes each request through the caches in order of speed and cost savings.

First, check the response cache. If an exact or semantic match is found, return the cached response immediately. No API call, no embedding call, no prompt processing. Cost: zero. Latency: under 10 milliseconds.

Second, if no response cache hit, check the embedding cache for the query. If the query embedding is cached, use it for retrieval without making an embedding API call. If not cached, compute the embedding and cache it.

Third, construct the request with the system prompt at the beginning to maximize prompt caching. The prompt cache reduces the cost of processing the static prefix by 90 percent.

Fourth, after receiving the response, store it in the response cache keyed by the request hash and the query embedding, so future identical or similar requests can be served from cache.

Each layer catches a different type of redundancy: response caching catches identical requests, semantic caching catches similar requests, embedding caching catches redundant vector computations, and prompt caching catches the repeated system prompt. Together, they produce compound savings that exceed any single strategy alone.