How to Set Similarity Thresholds for Cache Hits
Every semantic cache decision comes down to a single number: the cosine similarity between the incoming query embedding and the nearest cached query embedding. If this number exceeds your threshold, the cache returns the stored response. If it falls below, the query goes to the LLM. Getting this threshold right determines whether your cache is an asset or a liability.
Step 1: Start with a Conservative Default
Set your threshold to 0.95 cosine similarity for the initial deployment. This value works as a safe starting point across most domains and embedding models because it catches only queries that are clearly saying the same thing while rejecting queries that are merely related.
At 0.95 with text-embedding-3-small, here is what typically matches and what does not:
Matches (similarity > 0.95):
"What is your return policy?" vs "What's the return policy?" (0.98)
"How do I cancel my subscription?" vs "I want to cancel my subscription" (0.97)
"Reset my password" vs "I need to reset my password" (0.96)
Does not match (similarity < 0.95):
"What is your return policy?" vs "What is your shipping policy?" (0.89)
"How do I cancel my subscription?" vs "How do I upgrade my subscription?" (0.91)
"Reset my password" vs "Change my email address" (0.82)
At 0.95, you will miss some valid paraphrases that use very different vocabulary. "How do I send something back?" might score 0.93 against "What is your return policy?" and be classified as a miss. That is acceptable at launch. You will capture these in the tuning phase.
Step 2: Collect Production Data
Run your cache in production for 5-7 days with comprehensive logging enabled. For every cache lookup, log the following fields:
Required fields: timestamp, incoming query text, nearest cached query text (even on misses), cosine similarity score, cache decision (hit or miss), the threshold used, and response latency.
Highly recommended fields: user ID (to detect if false positives affect specific users disproportionately), query category (if you classify queries), and any user feedback signal (thumbs up/down, rephrasing, escalation).
During this data collection phase, consider running the cache in shadow mode for the first day or two. In shadow mode, the cache logs what it would do but always passes through to the LLM. This gives you comparison data: you can see what the cache would have returned versus what the LLM actually returned, making false positive identification much easier.
You need at least 1,000 total lookups and at least 200 near-threshold lookups (similarity between 0.88 and 0.97) to have enough data for meaningful threshold tuning. If your traffic is lower, extend the collection period.
Step 3: Evaluate Near-Threshold Matches
Extract all cache lookups where the similarity score fell between 0.88 and your current threshold (0.95). These are the queries that semantic matching recognized as related but did not serve from cache. Some of these are valid misses (the queries genuinely need different answers). Some are missed opportunities (the queries could have been served from cache).
For each near-threshold pair, answer this question: would the cached response have been a correct and complete answer to the incoming query? Label each pair as "correct match" (the cached response works), "partial match" (the cached response is related but incomplete or slightly wrong), or "wrong match" (the cached response answers a different question).
Also review all actual cache hits (similarity above 0.95) and label any that produced incorrect responses. These are your current false positives, the metric you need to minimize.
If manual labeling of hundreds of pairs is impractical, use sampling. Randomly select 100-200 near-threshold pairs and 50-100 above-threshold pairs. Label those. The statistical sample provides sufficient signal for threshold adjustment.
You can also use an LLM to assist with labeling. Send each query pair to a cheaper model (Claude Haiku or GPT-4o-mini) with the prompt: "Would this response correctly answer this question? The response was generated for query A. Would it be correct for query B? Answer yes, partially, or no." This automated labeling is not perfect but scales to thousands of pairs when manual review is impractical.
Step 4: Calculate Error Rates at Each Threshold
With your labeled data, calculate the false positive rate at every threshold from 0.88 to 0.99 in 0.01 increments. At each threshold level, count how many labeled pairs above that threshold were marked "wrong match." Divide by total pairs above that threshold. This gives you the false positive rate at each threshold.
You will see a pattern like this:
Threshold 0.99: 0.0% false positives, 8% hit rate
Threshold 0.97: 0.5% false positives, 22% hit rate
Threshold 0.95: 1.2% false positives, 38% hit rate
Threshold 0.93: 2.8% false positives, 52% hit rate
Threshold 0.91: 5.1% false positives, 61% hit rate
Threshold 0.89: 9.3% false positives, 68% hit rate
Threshold 0.87: 15.2% false positives, 73% hit rate
The optimal threshold sits at the point where lowering it further produces diminishing hit rate gains with accelerating false positive increases. In the example above, moving from 0.93 to 0.91 gains 9 percentage points of hit rate but doubles the false positive rate. Moving from 0.91 to 0.89 gains only 7 percentage points but nearly doubles false positives again. The sweet spot is around 0.92-0.93 for this particular workload.
Your acceptable false positive rate depends on the application. Customer support bots where wrong answers cause mild confusion: 3-5% is tolerable. Financial advice or medical information: below 1%. Internal productivity tools: 5-8% is often acceptable because users can easily recognize and ignore incorrect suggestions.
Step 5: Implement Per-Category Thresholds
A single global threshold is the right starting point but often leaves performance on the table. Different query categories have different semantic characteristics, and a threshold that works well for FAQ questions may be too aggressive or too conservative for technical troubleshooting.
FAQ and policy queries: These have a small, well-defined answer space. "What are your business hours?" has exactly one correct answer regardless of how the question is phrased. Lower thresholds (0.90-0.93) work well because even moderately similar questions in this category tend to have the same answer.
Technical support queries: Small differences in the problem description require different solutions. "My app crashes on login" needs a different answer than "My app crashes when uploading files" despite high semantic similarity. Higher thresholds (0.95-0.97) prevent serving the wrong troubleshooting steps.
Product comparison queries: "Is product A better than product B?" and "Is product B better than product A?" are semantically almost identical but have different correct answers if the response favors one product. Very high thresholds (0.97-0.99) or explicit directionality checks are necessary.
Implement per-category thresholds by classifying the incoming query before cache lookup. A simple keyword-based classifier (does the query contain "policy," "hours," "return" -> FAQ category) works for initial deployment. A lightweight text classifier trained on your query logs provides better accuracy for production.
Advanced Threshold Techniques
Dynamic thresholds based on cache confidence. Instead of a fixed threshold, adjust based on the cache entry's track record. Entries that have been served many times without negative feedback earn a lower threshold (more aggressive matching). New or rarely-served entries require a higher threshold until they prove reliable. This naturally tightens matching for untested responses and loosens it for battle-tested ones.
Multiple candidate evaluation. Instead of comparing against only the single nearest neighbor, retrieve the top 3-5 nearest cached entries. If multiple entries exceed the threshold and their responses differ significantly, that indicates the query falls in an ambiguous region, and you should pass through to the LLM rather than risk serving the wrong cached response. If multiple entries exceed the threshold and their responses are consistent, you have high confidence in the cache hit.
Embedding model calibration. Different embedding models produce different similarity score distributions. A 0.95 on text-embedding-3-small means something different than a 0.95 on voyage-3. When switching embedding models, do not assume your previous threshold still works. Re-run the calibration process with the new model's similarity scores. Some models compress the similarity range (most pairs score 0.90-0.99) while others spread it wider (0.70-0.99), requiring different threshold values for equivalent behavior.
Monitoring Threshold Performance Over Time
Your optimal threshold can drift as your user base grows, query patterns change, or you update the cached content. Implement continuous monitoring:
Weekly hit rate tracking: If hit rate drops suddenly without a change in traffic volume, your query patterns may have shifted and the cache warming set needs updating.
False positive rate estimation: Sample 50-100 cache hits per week and evaluate correctness. If false positive rate exceeds your tolerance, raise the threshold by 0.01 and re-evaluate the following week.
Score distribution monitoring: Track the distribution of similarity scores for both hits and misses. A healthy cache shows a bimodal distribution: high scores (0.95+) for genuine matches and low scores (below 0.85) for unrelated queries. If you see a growing cluster of scores near your threshold (0.90-0.95), it means your query space is becoming more ambiguous and you may need to add per-category thresholds or raise the global threshold.
Start at 0.95, collect a week of production data, calculate false positive rates at each 0.01 increment, and lower the threshold to the point where your error rate reaches your tolerance limit. Per-category thresholds unlock additional hit rate without sacrificing accuracy. Monitor continuously because optimal thresholds drift as query patterns evolve.