Multi-Tenant LLM Caching for SaaS Applications

The Multi-Tenant Caching Problem

In a single-tenant application, every cached response is safe to return to any user. In a multi-tenant SaaS product, this assumption breaks. Tenant A's cached response about their custom pricing might contain confidential numbers that Tenant B should never see. Tenant C might have different product configurations that make Tenant A's cached answers incorrect for them.

The challenge intensifies when LLM responses are personalized. If your AI assistant accesses tenant-specific data (custom knowledge bases, company policies, product catalogs), the same question from different tenants requires different answers. "What is our return policy?" has a different correct response for every tenant, even though the query text is identical.

At the same time, many queries are genuinely tenant-agnostic. "What is machine learning?" or "How do I write a for loop in Python?" produces the same correct answer regardless of which tenant asks. Failing to cache these shared-knowledge queries across tenants wastes money on redundant LLM calls.

Architecture Pattern 1: Fully Isolated Caches

Each tenant gets their own cache namespace. Cache lookups and stores include the tenant ID as part of the key. Tenant A's cache entries are completely invisible to Tenant B's lookups.

Implementation: Prefix all cache keys with the tenant ID. For exact match caching: the key becomes tenant_id:hash(prompt). For semantic caching: include tenant_id as a metadata filter on vector search, so similarity matching only considers the requesting tenant's entries.

Advantages: Zero risk of data leakage between tenants. Simple to reason about, each tenant's cache operates independently. Easy to implement tenant-specific TTLs, quotas, and invalidation. Simple to delete all cached data for a tenant (required for GDPR data deletion requests).

Disadvantages: Hit rates drop dramatically. If you have 100 tenants, each with 1,000 requests/day, the cache sees 1,000 queries per tenant instead of 100,000 total. Common queries that every tenant asks ("What does this feature do?") generate 100 separate LLM calls instead of 1. Small tenants with low traffic volumes rarely build up enough cache entries to achieve meaningful hit rates.

Best for: Applications where every response includes tenant-specific data. Highly regulated industries (healthcare, finance) where data isolation is non-negotiable. Tenants with very different use cases where shared caching would produce wrong answers.

Architecture Pattern 2: Fully Shared Cache

All tenants share a single cache pool. Any tenant's query can match against any other tenant's cached response.

Implementation: Cache keys and vector searches do not include tenant ID. All entries are visible to all tenants. The cache acts as if it serves a single application.

Advantages: Maximum hit rate. Every tenant benefits from every other tenant's queries. The cache warms fastest because all traffic contributes. Simplest to implement and operate.

Disadvantages: Data leakage risk is real. If any cached response contains tenant-specific information, it can be served to other tenants. Only safe when LLM responses never include tenant-specific data, which is rare in SaaS applications. Not suitable for applications that access tenant data stores as part of response generation.

Best for: Applications where the LLM produces generic answers that do not depend on tenant context. Educational platforms, general knowledge assistants, coding helpers where all users get the same correct answer. Consumer products where "tenants" are individual users asking the same kinds of questions.

Architecture Pattern 3: Hybrid (Recommended)

The hybrid approach classifies queries into "shared" (safe to cache across tenants) and "tenant-specific" (must be isolated). Shared queries use a global cache pool. Tenant-specific queries use per-tenant cache namespaces.

Implementation: Before cache lookup, classify the incoming query. If the query involves tenant-specific data (references custom products, policies, internal documents), route to the tenant-specific cache namespace. If the query is about general knowledge (technical concepts, public information, standard features), route to the shared cache.

Classification can be rule-based (queries containing keywords like "our," "my company," "our policy" route to tenant-specific) or model-based (a lightweight classifier trained to distinguish generic from tenant-specific queries). Rule-based classification is faster and sufficient for most applications. Model-based classification handles edge cases better but adds latency and complexity.

Advantages: Combines high hit rates for general queries (shared pool) with privacy protection for tenant-specific queries (isolated namespaces). Achieves 60-80% of the hit rate of a fully shared cache while maintaining the data isolation of per-tenant caching for sensitive queries.

Disadvantages: The classification step adds complexity. Misclassification in either direction causes problems: classifying a tenant-specific query as shared risks data leakage, classifying a shared query as tenant-specific reduces hit rate. Requires ongoing monitoring and tuning of the classification logic.

Best for: Most SaaS applications. Products that combine generic AI features (explanations, how-to guidance) with tenant-specific features (custom data retrieval, personalized recommendations).

Tenant-Aware Cache Key Design

Proper cache key design is the foundation of multi-tenant caching. Keys must encode enough context to prevent cross-tenant contamination while remaining general enough to achieve cache hits.

For exact match caching: The key structure is namespace:hash(relevant_context). For shared queries: shared:hash(user_query + model_id). For tenant-specific queries: tenant_{id}:hash(user_query + system_prompt_hash + model_id). Including the system prompt hash ensures that tenants with different system prompts (different brand voices, different instructions) never share cached responses.

For semantic caching: Use metadata filtering on the vector search. Store tenant_id and query_type (shared/specific) as metadata on each cached embedding. Shared lookups filter to query_type=shared. Tenant-specific lookups filter to tenant_id=requesting_tenant AND query_type=specific. This keeps the vector index unified (one search, not N searches) while enforcing isolation at the result level.

For RAG-based responses: Include the document source scope in the cache key. If Tenant A asks "How does billing work?" and the answer comes from Tenant A's custom billing docs, the cache entry must be tagged with both the tenant ID and the document IDs used. If Tenant B asks the same question, their different billing docs should produce a different cached response.

Capacity Planning and Quotas

Multi-tenant caches need capacity management that prevents large tenants from consuming all shared resources.

Per-tenant cache quotas: Limit the number of cached entries per tenant. A tenant with 100,000 entries should not crowd out smaller tenants. Set quotas based on the tenant's plan tier: free tier gets 1,000 cache entries, standard gets 10,000, enterprise gets 100,000. When a tenant exceeds their quota, evict their least recently used entries.

Shared cache allocation: Reserve a portion of cache capacity for the shared pool and allocate the remainder across tenant-specific namespaces. A 70/30 split (70% shared, 30% tenant-specific) works well for most applications because the shared pool handles the majority of queries by volume.

Cache size estimation: Per entry: 8-12KB (embedding vector + response text + metadata). For 1,000 tenants with 5,000 entries each plus a 500,000 entry shared pool: approximately 55 million entries x 10KB = 550GB. This requires dedicated vector database infrastructure, not an in-memory cache. Plan storage capacity based on your expected tenant count and entries per tenant.

Monitoring and Observability per Tenant

Multi-tenant cache monitoring needs per-tenant visibility. Global hit rate metrics mask individual tenant experience. If your shared cache hit rate is 60% but Tenant A only hits 15% because their queries are all highly specific, Tenant A's users have a poor experience that global dashboards hide.

Per-tenant dashboards: Track hit rate, average latency, cache entry count, and eviction rate per tenant. Surface tenants whose hit rates fall below your target threshold (e.g., below 20%) so you can investigate. Low hit rates often indicate that a tenant's queries are too unique for semantic matching, or that their cache quota is too small and entries evict before reuse.

Cross-tenant leak detection: Implement automated checks that scan for potential data leakage. When a cache hit occurs, verify that the matched entry belongs to either the shared pool or the requesting tenant's namespace. Log any anomalies where the matched tenant_id differs from the requester. Run daily reports flagging mismatches, which indicate a classification bug or a metadata filtering failure in your vector search.

Capacity alerts: Monitor each tenant's approach toward their quota limit. Alert at 80% capacity so you can proactively increase limits or tune eviction before the tenant experiences degraded hit rates from excessive eviction churn. Monitor the shared pool fill rate to plan infrastructure scaling before performance degrades.

Migrating from Single-Tenant to Multi-Tenant Caching

If you built your cache for a single-tenant application and now need multi-tenancy for your SaaS offering, the migration path matters.

Step 1: Add tenant metadata to existing entries. Tag every existing cache entry with the default tenant ID. This is a backfill operation on your vector store metadata. All existing entries effectively become Tenant 0's isolated cache.

Step 2: Implement the query classifier. Build and deploy the shared vs tenant-specific classification logic. Run it in audit mode initially (classify but do not change routing) to measure accuracy before switching live.

Step 3: Create the shared pool. Identify existing cached entries that are genuinely tenant-agnostic (general knowledge, product documentation that all tenants share). Copy these to the shared namespace. Do not move them, copy them, so the original tenant's cache is not disrupted.

Step 4: Enable multi-tenant routing. Switch the cache lookup path to use tenant-aware routing. New queries go through the classifier. Existing tenants see no degradation because their existing entries remain in their namespace. New entries populate the appropriate namespace based on classification.

Privacy and Compliance

Multi-tenant caching has specific privacy implications that affect regulatory compliance.

GDPR right to deletion: When a tenant requests data deletion, you must purge all their cached entries. With per-tenant namespaces, this is straightforward: delete all entries with that tenant's ID. With a shared cache, ensure no shared entries contain data derived from the deleted tenant's documents or queries.

Data residency requirements: Some tenants may require their data to stay within specific geographic regions. Cache infrastructure must respect these requirements. If Tenant A requires EU data residency, their cache entries must reside on EU-region infrastructure, even if the shared cache runs globally.

Audit logging: For regulated industries, log every cache hit including which tenant's data was accessed and what response was served. This creates an audit trail showing that no cross-tenant data access occurred.