Prompt Caching vs Response Caching Explained

What Prompt Caching Does

Prompt caching operates at the LLM provider's infrastructure level. When you send a request with a long system prompt, the provider processes that prompt into internal representations (key-value pairs in the attention mechanism). With prompt caching enabled, the provider saves these processed representations so that subsequent requests with the same prefix skip the processing step.

This reduces cost, not latency in a dramatic way. Anthropic charges 10% of the normal input token price for cached prompt tokens. If your system prompt is 10,000 tokens at $3.00/million tokens, the first request costs $0.03 for those tokens. Every subsequent request with the same system prompt costs $0.003 for those cached tokens. Over 10,000 requests, you save roughly $270 on system prompt tokens alone.

Critically, prompt caching does NOT eliminate the API call. You still make a request, you still pay for output tokens at full price, and you still wait for the model to generate a response. The model processes the cached system prompt faster (reducing time-to-first-token by 60-85% for long prompts), but it still generates a fresh response every time.

How Prompt Caching Works on Each Provider

Anthropic: Add cache_control breakpoints in your message content to indicate which portions should be cached. Cached prefixes persist for 5 minutes of inactivity (extended with each new request). Minimum cacheable prefix is 1,024 tokens for Claude Sonnet and 2,048 for Claude Opus. Cache write costs 25% more than base input price. Cache reads cost 10% of base price.

OpenAI: Automatic prompt caching enabled by default on most models since late 2024. The API automatically detects when the beginning of a prompt matches a recently seen prompt and uses cached processing. Cached input tokens cost 50% of normal price. No explicit cache control needed from the developer.

Google (Gemini): Context caching lets you upload large documents or system instructions once, receive a cache ID, and reference that cache in subsequent requests. Cached content persists for a configurable duration (up to 1 hour by default). Cached input tokens cost 75% less than normal.

What Response Caching Does

Response caching operates at your application level. You store complete LLM responses and return them directly when matching queries arrive. The LLM API is never called for cache hits. Zero tokens consumed, zero API cost, sub-50ms latency.

Response caching requires you to build and maintain the caching layer: determine what constitutes a "matching" query (exact text or semantic similarity), store responses in a cache backend (Redis, vector database, or managed service), manage TTL and invalidation, and monitor accuracy.

The tradeoff compared to prompt caching is clear: response caching saves more per hit (100% of API cost vs 75-90% of input token cost) but requires engineering investment and introduces the risk of serving stale or incorrect cached responses. Prompt caching saves less per request but requires essentially zero engineering effort and introduces no accuracy risk.

Direct Cost Comparison

To make the comparison concrete, consider an application that sends 10,000 requests per day with the following structure: 8,000 token system prompt, 200 token user query, 500 token output. Using Claude Sonnet pricing ($3.00/M input, $15.00/M output).

No caching:
Input: 8,200 tokens x 10,000 requests = 82M tokens = $246/day
Output: 500 tokens x 10,000 requests = 5M tokens = $75/day
Total: $321/day = $9,630/month

Prompt caching only (Anthropic):
System prompt (cached): 8,000 tokens at 10% = $0.003/M x 80M = $2.40/day
User query (uncached): 200 tokens at full price = $3.00/M x 2M = $6/day
Cache write cost (first request per 5-min window): negligible at scale
Output: unchanged at $75/day
Total: $83.40/day = $2,502/month (74% savings)

Response caching only (50% hit rate):
Hits: 5,000 requests x $0 = $0
Misses: 5,000 requests x full cost ($0.0321) = $160.50/day
Cache infrastructure: approximately $5/day
Total: $165.50/day = $4,965/month (48% savings)

Both caches combined (50% response cache hit, prompt caching on misses):
Response cache hits: 5,000 x $0 = $0
Response cache misses with prompt caching: 5,000 x $8.34/day equivalent = $83.40/day
Cache infrastructure: approximately $5/day
Total: $88.40/day = $2,652/month (72% savings, but with faster responses for 50% of requests)

The combined approach saves similarly to prompt caching alone on cost, but delivers dramatically better latency for the 50% of requests that hit the response cache (sub-50ms vs 1-3 seconds).

Latency Comparison

Prompt caching effect on latency: Reduces time-to-first-token by 60-85% for long system prompts. A request that normally takes 800ms to first token might take 200ms with a cached prompt. Total response time decreases proportionally: a 3-second response might become 2.4 seconds. Meaningful improvement but still seconds, not milliseconds.

Response caching effect on latency: Eliminates the API call entirely. Cache hit latency is 10-50ms regardless of how long the response would take to generate. A response that normally takes 3 seconds returns in 30ms. This is the difference between "the AI is thinking" and "instant answer."

For user-facing applications where perceived speed matters, response caching provides the transformative latency improvement. Prompt caching makes uncached calls faster but does not achieve the "feels instant" threshold that response caching provides.

When to Use Each

Always use prompt caching when your provider supports it and your system prompt exceeds the minimum cacheable size (1,024+ tokens for Anthropic). The cost savings are automatic, the risk is zero, and the implementation effort is minimal (add cache_control markers for Anthropic, nothing for OpenAI). There is no reason not to use it.

Add response caching when: Your workload has meaningful query repetition (30%+ of queries are duplicates or paraphrases), your application can tolerate occasionally serving a cached response that is slightly stale, and your monthly LLM spend justifies the engineering investment (typically $500+/month).

Skip response caching when: Every query is unique (creative tasks with high temperature), responses must incorporate real-time data that changes between requests, the application handles fewer than 100 requests per day, or your domain requires 100% freshness and zero risk of stale answers.

Implementation Order

The recommended implementation sequence maximizes savings per unit of engineering effort.

Phase 1 (1 hour): Enable provider-level prompt caching. For Anthropic, add cache_control breakpoints to your system prompt. For OpenAI, it is automatic. Immediate 60-80% reduction in input token costs with zero risk.

Phase 2 (1 day): Add exact match response caching using Redis or a simple key-value store. Hash the complete prompt, store the response, set a reasonable TTL. This catches the 5-15% of requests that are textually identical. Zero risk because exact matches are deterministic.

Phase 3 (1 week): Add semantic response caching using embeddings and a vector store. This captures the additional 25-55% of requests that are paraphrases. Requires threshold tuning and accuracy monitoring but delivers the largest hit rate improvement.

Each phase builds on the previous one. Phase 1 reduces the cost of response cache misses. Phase 2 catches identical queries cheaply and safely. Phase 3 maximizes hit rate for the remaining traffic. Together, the three phases can reduce total LLM costs by 80-95% for highly repetitive workloads.

How They Interact at Runtime

In a system with both caching layers, a typical request flows through four decision points:

1. Check exact match response cache. If hit, return immediately (0 cost, sub-2ms).
2. Check semantic response cache. If hit, return immediately (0 cost, 10-30ms).
3. Response cache miss. Send request to LLM API with prompt caching enabled. Prompt caching reduces input cost (10-50% of normal input price).
4. Store the LLM response in both exact match and semantic response caches for future requests.

The response cache acts as the outer layer, eliminating calls entirely when possible. Prompt caching acts as the inner optimization, reducing costs for the calls that do reach the API. The layers do not conflict or duplicate effort. They operate on different parts of the cost and latency equation.