How to Cache Embeddings for Faster Retrieval
Every time your application converts text to a vector embedding, it typically makes an API call. In a RAG pipeline that processes the same documents repeatedly, in a semantic cache that embeds every incoming query, or in a search system that re-embeds common queries, many of these API calls produce identical results for identical inputs. Embedding functions are deterministic: the same input text always produces the same output vector. This determinism makes embeddings ideal candidates for caching.
Step 1: Identify Where Embeddings Are Redundant
Before building a cache, find where your application generates the same embedding more than once. Common sources of redundancy include:
Query embeddings in semantic caching: Your semantic cache embeds every incoming query to search for cached responses. But the cache lookup itself happens before you know whether the query is new or repeated. If the same user asks "What is your return policy?" twice in a week, you generate the same query embedding both times. With an embedding cache, the second embedding is a sub-millisecond lookup instead of a 15ms API call.
Document re-embedding in RAG updates: When you update your RAG document index, you re-process all documents to refresh their embeddings. If only 10% of documents changed, the other 90% produce identical embeddings. Caching prevents re-embedding the unchanged 90%.
Repeated search queries: In semantic search applications, popular queries arrive repeatedly. "How do I reset my password?" might be searched 50 times per day. Without embedding caching, that is 50 API calls producing the same vector. With caching, it is 1 API call and 49 cache hits.
Multi-step pipelines: Some architectures embed the same text at multiple stages. A query might be embedded once for cache lookup, then again for RAG retrieval, then again for reranking. If the same embedding function and model are used at each stage, a cache eliminates the second and third embedding calls.
Step 2: Choose a Cache Backend
Embedding caching is simpler than LLM response caching because embeddings are always exact match (the same text always produces the same vector). You do not need semantic matching or similarity thresholds. A standard key-value store works perfectly.
In-process LRU cache: For single-process applications, a dictionary or LRU cache in memory is the fastest option. No network calls, sub-microsecond lookups. Python's functools.lru_cache works directly on the embedding function. Limitation: the cache does not persist across restarts and is not shared between processes or servers.
Redis: For multi-process or multi-server applications, Redis provides a shared, persistent cache. Store the text hash as the key and the serialized embedding vector as the value. Lookup latency is 0.5-2ms over a local network connection. Redis handles TTL expiration natively.
Local file cache: For batch processing (re-embedding document collections), store embeddings in a local SQLite database keyed by text hash. This persists across runs without requiring an external service. Lookup latency is 1-5ms, fast enough for batch processing though not ideal for real-time serving.
The cache key should be a hash of the text content plus the model identifier. Include the model name because different embedding models produce different vectors for the same text. If you switch from text-embedding-3-small to voyage-3, all cached embeddings become invalid. Including the model name in the key ensures they do not collide.
Step 3: Implement the Caching Layer
Wrap your embedding function with a cache-aside pattern. The wrapper checks the cache before calling the embedding API and stores results on cache miss.
The core logic follows this flow: receive the input text, compute the cache key (hash of text + model ID), check the cache backend for the key, return the cached embedding if found, call the embedding API if not found, store the resulting embedding in cache, and return the embedding.
For batch embedding (multiple texts at once), optimize by checking all texts against cache first, identifying which ones miss, sending only the misses to the embedding API in a single batch call, storing all new embeddings, and returning the complete set (cached + newly generated) in the original order.
This batch optimization matters significantly for document processing. If you are embedding 1,000 document chunks and 800 are already cached, you send only 200 to the API instead of 1,000. The API call is faster (fewer tokens), cheaper (80% fewer tokens billed), and the overall operation completes sooner.
Handle serialization carefully. Embedding vectors are typically numpy arrays or lists of floats. For Redis storage, serialize as bytes (numpy tobytes() method) rather than JSON to minimize storage size and serialization overhead. A 1536-dimension float32 vector takes 6,144 bytes as raw bytes versus roughly 15,000 bytes as JSON text.
Step 4: Handle Cache Invalidation
Embedding caches need invalidation in fewer scenarios than LLM response caches, but the scenarios that require it are critical.
Embedding model change: When you switch embedding models (from text-embedding-3-small to text-embedding-3-large) or when the provider updates a model, all cached embeddings become invalid. Vectors from different models occupy different vector spaces and cannot be compared meaningfully. Clear the entire embedding cache when changing models. If you include the model identifier in the cache key (as recommended), old entries simply stop matching and expire naturally via TTL.
Text normalization changes: If you modify how text is preprocessed before embedding (different tokenization, adding prefixes, changing casing), cached embeddings for the original text format become stale. The same underlying text now produces a different input to the embedding model. Track your preprocessing version and include it in the cache key.
TTL for evolving corpora: For document embedding caches, set TTLs that match your re-indexing schedule. If you re-index documents weekly, a 7-day TTL ensures stale embeddings expire naturally. For query embedding caches, longer TTLs (30+ days) are safe because the same query text always embeds identically regardless of how old the cache entry is.
Step 5: Monitor and Optimize
Track four metrics to verify your embedding cache is working effectively.
Hit rate: The percentage of embedding requests served from cache. A healthy embedding cache achieves 40-80% hit rate for query embedding and 60-90% hit rate for document re-embedding. If your hit rate is below 20%, your query patterns may be too diverse to benefit from caching, or your cache is too small.
Latency savings: Compare the average embedding time with cache (should be 1-3ms for Redis-backed caches) versus without (15-25ms for API calls). The difference, multiplied by your request volume, tells you the total latency saved. For a semantic caching pipeline where embedding happens on every request, saving 15ms per request across 100,000 daily requests saves 1,500 seconds of cumulative latency per day.
Storage consumption: Each cached embedding takes 4-12KB depending on dimensions. At 1536 dimensions and float32 precision: 6,144 bytes per vector plus key overhead. One million cached embeddings require approximately 6-7GB of storage. Monitor growth and set size limits to prevent unbounded expansion.
Cost savings: Calculate: (embedding API calls avoided per month) x (cost per embedding call). At $0.02/million tokens and 30 tokens average per query, each cached hit saves $0.0000006. At 1 million cached hits per month, that saves $0.60 in API costs. The cost savings from embedding caching are small in absolute terms. The primary benefit is latency reduction, not cost. The exception is large-scale batch embedding where caching prevents re-embedding millions of unchanged documents, saving tens or hundreds of dollars per re-indexing run.
Embedding Caching vs LLM Response Caching
These are complementary caches that operate at different levels of your AI pipeline.
Embedding caching stores the intermediate vector representation of text. It speeds up any operation that requires embedding: semantic cache lookup, RAG retrieval, similarity search, classification. The savings per cached call are small (fractions of a cent) but the latency reduction is meaningful (15ms to sub-1ms per embedding).
LLM response caching stores the final output of the complete LLM call. It eliminates entire API calls worth $0.01-0.10 each. The savings per cached call are 100-1000x larger than embedding caching savings.
In a typical semantic caching pipeline, both caches operate: the embedding cache accelerates the cache lookup step (embedding the query to search for similar cached responses), and the response cache provides the actual cached LLM output when a match is found. Implement both for optimal performance.
Cache embeddings using exact match on text hash plus model identifier. The primary benefit is latency reduction (15ms to sub-1ms per embedding), not cost savings (which are minimal per call). Invalidate when switching embedding models or changing text preprocessing. For semantic caching pipelines, embedding caching accelerates the cache lookup step that runs on every incoming request.