Artifact 03
Building a Semantic Cache that Does Not Trust Cosine
July 3, 2026
Usually semantic caches for LLMs work on embeddings and cosine similarity. You take the new prompt, turn it into a vector, find the closest old prompt, and if the score is high enough you reuse the old answer. Reading this feels good & easy because we can just say similar prompts should have similar answers, but how true is that actually?
- What if two prompts use the same words and talk about the same object, but ask for the opposite thing?
- What if "approve the refund" and "deny the refund" sit next to each other in embedding space?
- Who decides that a high cosine score means it is safe to reuse the cached response?
These are some questions which I had in mind when I started building SmartMemo. Cosine is good at finding neighbours. It is not the same thing as semantic equivalence. And I wanted to realize that difference by implementing it myself.
Ok, let us first understand what actually goes wrong.
You gave a prompt to your agent. The cache embeds it. Somewhere in the store there is an old prompt that looks almost the same. The cosine is 0.92, threshold was 0.90, so it is a hit. The old answer comes back. The problem is that the two prompts were never the same request. One was approve, one was deny. Same customer, same refund, opposite action. The vectors do not care about that, because the words around the action are shared and the embedding model collapses them.
So cosine can select candidates. It should not be the final judge.
What SmartMemo does is split that into two steps.
First step, same as everyone else. The prompt is converted into an embedding. A vector is basically a row of numbers that tries to hold the meaning. That vector goes into FAISS and we take the nearest cached prompts. These are only candidates. Nothing is returned yet.
Second step is the part I actually cared about. A small classifier looks at the pair: this new prompt and that old prompt. It has been trained to say whether they are asking for the same thing, or they just look similar. If the score is high enough, we reuse the cached response. If not, we call the LLM, store the new prompt with its answer, and move on.
The classifier that ships with it is a small MLP on top of MiniLM embeddings. I trained it on a lot of prompt pairs across a few domains, including the hard ones, the same-object opposite-action kind, and also negated versions of the same request. On a small gold set I kept aside, cosine at the same recall was at 0.53 precision. The classifier went to 0.83. Same recall, fewer wrong hits. That +30 is the whole reason this project exists.
It is still not magic. On some high-stakes medical or legal opposite-action pairs it is better than cosine, but it still gets a few wrong. A generic classifier cannot know your domain perfectly. So there is a feedback path. If a hit was bad, you can report it. There is also an optional implicit version: if the same prompt comes again quickly after a cache hit, treat that as a signal that the earlier answer was not useful. Those pairs can be exported and used later to train a classifier for your own domain.
The thing you actually call is one async function, get_or_call. You pass the prompt and the function that would call the LLM. You get back the response, whether it was a hit, and the classifier score. Storage is just SQLite. Nothing distributed. It is meant to sit inside one agent process and not pretend to be a cluster.
That is basically it. Embedding search to find likely matches. A classifier to decide if they are actually the same thing. And a way to learn when they were not.