Semantic Caching for LLMs: The False Hit Problem Your Dashboard Can't See

Semantic caching is one of the easiest cost cuts in an LLM app. You embed each incoming question, look for a stored question that is close enough, and return its answer instead of calling the model. When your traffic is the same fifty questions asked a thousand ways, the savings are real.

This post is about the cost nobody puts on the dashboard: the false hit. A semantic cache can return the right answer to the wrong question, in a few milliseconds, with a 200 status, and your hit-rate graph will count it as a win.

We'll cover how a semantic cache decides, why no similarity threshold can separate "same question" from "close question", and a design (plus a test suite) that catches the failures before users do.

How a semantic cache actually decides

A classic cache key is a fact. user:42 either exists or it doesn't. Its worst failure is staleness: the right question with an old answer (the familiar fight covered in why your cache is serving stale data).

A semantic cache replaces equality with a nearest-neighbor search:

def naive_lookup(question, cache, embed, threshold=0.85):
    hit = cache.nearest(vector=embed(question))   # closest stored question
    if hit and hit.score >= threshold:
        return hit.answer                         # served without calling the model
    return None

That >= threshold line is doing something a cache never did before: it is classifying. Every lookup is a yes/no guess that two questions deserve the same answer. Classifiers have error rates. Most semantic caches never measure theirs.

The experiment that breaks the threshold

A 2025 study of negation in text embeddings gives the cleanest demonstration [1]. Take the headline "Polar bear DNA may help fight obesity." Compare it with two sentences:

• a paraphrase that human raters scored as nearly identical in meaning
• the negation: the same headline with its claim flipped

Across nine popular embedding models, the negation scored between 0.93 and 0.97 against the original. In every model, the negation was as close to the original as the paraphrase, or closer.

Now look at the defaults people ship. Redis LangCache defaults to a similarity threshold of 0.85 [2]. The vCache authors describe 0.8 as a commonly used static threshold [3]. Both admit the negation.

Worse: there is no threshold that fixes this example. Any value low enough to accept the paraphrase also accepts the negation. Raise it enough to block the negation and you block the paraphrase first.

It isn't one unlucky sentence

• vCache (ICLR 2026) found that correct and incorrect cache hits have highly overlapping similarity distributions across 45,000 decisions in one benchmark, and that the best threshold varies a lot from one cached entry to the next [3]. One global threshold is the wrong shape of solution.
• MeanCache (IPDPS 2025) measured GPTCache at its recommended threshold: 233 false hits on 1,000 standalone queries, precision 0.52 [4]. In that benchmark, close to half of the hits were wrong answers. One setup, one dataset, but nobody would have seen it without labeling hits by hand.

Five hinges embeddings miss

Embeddings encode topic and wording. Answers often turn on a word that barely moves the vector. These pairs are illustrations, but each takes ten minutes to test against your own cache:

Follow-ups are their own trap. MeanCache's example: after "draw a circle," a user says "change the color to red," which sits close to a cached "change line color to blue" from an unrelated conversation [4].

Three production failures that hide behind a good hit rate

Cross-tenant answers. A personalized answer lands in a shared cache. Another customer asks the identical sentence and gets the first customer's account details. That is a data exposure, not a quality bug. (Personalized answers are tricky in general; see your AI is answering a version of you that doesn't exist.)

Fixes that never ship. You improve the prompt or upgrade the model. Every cached question keeps serving answers from the old prompt, and popular questions are the most cached. The fix reaches everyone except most of your users.

Actions that never happened. An agent replies "Done, I've cancelled your subscription" after a tool call. Cache that reply and the next similar request gets the sentence without the tool call. Never cache responses that report side effects.

The fix: a Two-Key Cache

Stop asking similarity to decide things it can't see. Split the key into an exact part and a fuzzy part, then guard the fuzzy match with plain code.

import re
from dataclasses import dataclass

NEGATIONS = r"\b(not|no|never|without|except|can't|cannot|don't|doesn't|isn't|won't)\b"
NAMES = r"\b(ios|android|web|free|pro|enterprise)\b"     # your products, plans, platforms

@dataclass(frozen=True)
class Ctx:
    tenant: str
    locale: str
    model: str
    prompt_version: str
    data_version: str        # index / docs snapshot the answer came from
    cacheable: bool          # not personal, not time-bound, not an action

def hinges(text: str):
    t = text.lower()
    return (frozenset(re.findall(NEGATIONS, t)),
            frozenset(re.findall(r"\d+(?:\.\d+)*", t)),   # numbers, amounts, versions
            frozenset(re.findall(NAMES, t)))

def find_hit(question: str, ctx: Ctx, cache, embed, threshold=0.85):
    if not ctx.cacheable:                                         # 1. eligible?
        return None
    partition = (ctx.tenant, ctx.locale, ctx.model,
                 ctx.prompt_version, ctx.data_version)            # 2. exact key
    hit = cache.nearest(partition=partition, vector=embed(question))  # 3. fuzzy
    if hit is None or hit.score < threshold:
        return None
    if hinges(question) != hinges(hit.question):                  # 4. hinge check
        return None
    return hit                                                    # 5. serve + sample

What each step buys you:

1. Eligibility removes the questions where a false hit is most expensive: personal, time-bound ("today", "current"), or side-effecting.
2. The exact partition makes tenant, locale, model, prompt version and data version impossible to "approximately" match. Bumping prompt_version also retires every answer written under the old prompt. Your hit rate drops; that drop is the cache refusing to guess about things it can't judge.
3. Fuzzy match embeds the question only. Rewrite follow-ups into standalone questions first, or don't cache them.
4. The hinge check catches the polar bear: if negations, numbers or known names differ, it's a miss whatever the score.
5. Serve and sample gives you the metric the dashboard is missing.

Be honest about the gap: "MySQL to Postgres" and "Postgres to MySQL" pass the hinge check. Direction and subtle scope changes need a verifier (a cross-encoder or small model asking "does this stored answer answer this question?") or a decision to keep that category out of the cache.

Measure the false-hit rate

Hit rate tells you how often the cache answered. It says nothing about whether it was right. Sample a small slice of hits, regenerate them in the background, and compare:

import random

AUDIT_RATE = 0.02   # start small; compare by hand before automating

def answer(question, ctx):
    hit = find_hit(question, ctx, cache, embed)
    if hit is None:
        fresh = call_model(question, ctx)
        if ctx.cacheable:
            cache.store(question, fresh, ctx)
        return fresh
    if random.random() < AUDIT_RATE:
        audit_queue.put({"question": question, "matched": hit.question,
                         "score": hit.score, "cached": hit.answer, "ctx": ctx})
    return hit.answer

A worker drains audit_queue, calls the model fresh, and records whether the cached answer would have been acceptable. Plot false-hit rate next to hit rate. If you want to go further, vCache learns a per-entry threshold against an error rate you set [3].

One warning: a cache makes your app perfectly consistent, and consistency feels like correctness. It isn't. Same lesson as temperature 0 won't fix your AI: a cached wrong answer is wrong identically every time.

The Evil Twin Test (put this in CI)

For your twenty most-cached questions, write their evil twins: negated, renumbered, renamed, reversed, and asked from another tenant. Paraphrases should hit. Twins must miss.

import pytest

TWINS = [
    ("Which plans include SSO?",          "Which plans don't include SSO?"),
    ("How do I do this in Python 3.8?",   "How do I do this in Python 3.12?"),
    ("Reset my password in the iOS app",  "Reset my password in the Android app"),
    ("Migrate from MySQL to Postgres",    "Migrate from Postgres to MySQL"),
]

@pytest.mark.parametrize("original,twin", TWINS)
def test_evil_twin_never_shares_an_answer(cache, ctx, original, twin):
    cache.store(original, f"ANSWER FOR: {original}", ctx)
    assert find_hit(twin, ctx, cache, embed) is None, f"false hit: {twin!r}"

With the code above, the MySQL/Postgres twin will fail, which is the point: now you know exactly which category needs a verifier or an exclusion. Rerun the suite whenever the embedding model, threshold, or caching scope changes.

Key Takeaways

• A semantic cache is a classifier: every lookup guesses that two questions deserve the same answer.
• Embeddings capture topic and wording; answers turn on negations, numbers, names, direction and the asker.
• No single threshold separates same from close; correct and incorrect hits overlap [3].
• Put everything that changes the answer, except wording, into an exact partition, then check hinges in code.
• Track false-hit rate beside hit rate, and keep an evil-twin suite in CI.

FAQ

What is semantic caching for LLMs?

Semantic caching stores LLM responses and reuses them for new prompts whose embeddings are similar enough to a stored prompt, instead of requiring an exact text match. It reduces cost and latency for repetitive traffic such as FAQs and documentation questions.

Why does a semantic cache return wrong answers?

Because similarity measures topic and wording, not whether two questions deserve the same answer. Negated, renumbered or reversed questions can score as close as true paraphrases, so the cache serves a stored answer to a question it doesn't fit. This is a false cache hit.

What similarity threshold should I use for a semantic cache?

There is no universally safe value. Common defaults sit around 0.8 to 0.85, but research shows correct and incorrect hits have overlapping similarity distributions, so any single threshold trades missed hits for wrong ones. Pick the threshold from your own evil-twin and paraphrase tests, and add hinge checks or a verifier.

Is semantic caching safe in multi-tenant or personalized apps?

Only if tenant, user (for personal answers) and other answer-changing context are part of an exact cache key, with similarity search confined to that partition. A shared cache with no partition can serve one customer's answer to another customer who asks the same sentence.

What's the difference between semantic caching and prompt caching?

Provider prompt caching reuses computation for an exact prompt prefix, and the model still generates a fresh response, so it cannot change answers. Semantic caching skips the model entirely for similar prompts, so it can return an answer written for a different question. For the exact kind, see prompt caching for AI agents.

How do I measure semantic cache accuracy?

Log the similarity score and matched question for every hit, regenerate a small random sample of hits in the background, and judge whether the cached answer was acceptable. Report the resulting false-hit rate next to the hit rate, and run a CI suite of near-duplicate "evil twin" questions that must never share an answer.

Related reading

• Prompt Caching for AI Agents: Stop Breaking Your Own Cache: the safe, exact-match kind of LLM cache
• Why Your Cache Is Serving Stale Data (And How to Actually Fix It): the classic cache failure, for contrast
• RAG Explained: How to Make AI Actually Use Your Data: where data_version comes from
• Temperature 0 Won't Fix Your AI. It Just Hides the Doubt.: why consistency isn't correctness
• New here? Start Here: a plain-English map to understanding AI

References

1. H. Cao, "Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality," arXiv:2504.00584, 2025.
2. Redis, "Create a LangCache service," Redis Cloud documentation (default similarity threshold 0.85).
3. L. G. Schroeder et al., "vCache: Verified Semantic Prompt Caching," ICLR 2026, arXiv:2502.03771.
4. W. Gill et al., "MeanCache: User-Centric Semantic Caching for LLM Web Services," IEEE IPDPS 2025, arXiv:2403.02694.

The bottom line

Keep the semantic cache. Just stop thinking of it as a cache that can be stale, and start treating it as a model that can be wrong. Give it an error rate, a test suite, and a key that knows who is asking.

Paraphrases should hit. Twins must miss. If you can't name the twins, you can't name your threshold either.

Get one of these in your inbox each week: subscribe to Simply Explained.

Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype.