Semantic LLM Cache
SmartMemo
A semantic cache for LLM agents where a trained classifier decides if a cached answer is safe to reuse, instead of raw cosine similarity.
+30 pts
precision at equal recall vs. cosine
- FAISS retrieves candidates; a learned classifier decides reuse
- +30 precision points at equal recall vs. a tuned cosine baseline
- Bundled classifier, so no training is needed for a cold start
- Implicit + explicit bad-hit feedback feeds gated retraining
Watch SmartMemo block an unsafe cache hit
Live demo API is not configured.
Set NEXT_PUBLIC_DEMOS_API_URL to the deployed FastAPI service URL.
Watch the complete SmartMemo demo
An eight-minute walkthrough of why cosine-only semantic caching breaks, how the classifier gate blocks unsafe reuse, how the live demo works, and how bad-hit feedback turns into retraining data.
Cosine similarity is not semantic equivalence
Semantic caches reuse an LLM response when a new prompt is “close enough” to an old one. But close in embedding space isn't the same as equivalent in meaning. “Approve the refund” and “Deny the refund” sit a hair apart by cosine, and a threshold-only cache will happily serve the wrong one.
In a support, medical, or finance setting, a bad hit like that means serving a wrong, confident answer. SmartMemo keeps cosine as a fast candidate selector and puts a learned classifier in charge of the decision.
Retrieve with cosine, decide with a classifier
Encode
Embed the prompt with all-MiniLM-L6-v2 (384-dim). Fast, local, no API call.
FAISS search
Find nearest cached prompts by cosine similarity. This is the candidate set, not the answer.
Pairwise classifier
A small MLP over the embedding pair scores true semantic equivalence. Below bar → cache miss.
Reuse or call
Equivalent → return the cached response. Otherwise call the LLM and store the new pair.
backbone
Embeddings: all-MiniLM-L6-v2 (384-dim). The bundled classifier-v2 is a small MLP trained on 16,576 labeled pairs across 9 domains, mixing local-paraphraser positives with templated hard negatives.
Classifier-v2 vs. a tuned cosine baseline
| Metric | Tuned cosine | Classifier-v2 |
|---|---|---|
| Precisionat equal recall | 0.53 | 0.83 |
| Recall | 0.94 | 0.94 |
| F1 | 0.67 | 0.88 |
| False positiveson the 84-pair gold set | 26 | 6 |
Gold test set: 84 held-out pairs (31 equivalent, 53 not). +30 precision points with recall held constant. The cache rejects 20 more wrong reuses without losing a single correct hit.
On a deliberately adversarial high-stakes set (16 medical/legal/finance opposite-action pairs), false-positive hits dropped from 8 → 6. A generic classifier isn't infallible out of distribution. That's exactly why the feedback-and-retraining loop exists.
One call, cache-aware
from smartmemo import SmartMemo, ClassifierConfigcache = SmartMemo(domain="customer-support",classifier=ClassifierConfig.bundled(),)result = await cache.get_or_call(prompt="Summarize this customer's latest billing ticket",llm_function=call_llm,)result.response, result.was_cache_hit, result.classifier_score
closing the loop
report_bad_hit(query_id, reason=…) and implicit re-issue detection record misses; export_feedback_pairs(path) turns them into JSONL, and smartmemo retrain ships a new classifier only if it clears the validation gates.
Why I built it this way
Classifier, not threshold
Cosine similarity is a way to shortlist candidates, not proof that two prompts mean the same thing. A small MLP over the embedding pair makes the final reuse call, so near-duplicate-but-opposite prompts don't end up sharing a cached answer.
Learns from its own mistakes
Implicit feedback flags a re-issued prompt within a window as a likely bad hit; report_bad_hit() records explicit ones. export_feedback_pairs() turns them into JSONL retraining data.
Gated retraining, not auto-reload
smartmemo retrain runs behind validation gates. A new classifier only ships if it passes. Nothing swaps itself out silently in the background and regresses precision in production.
WAL-backed SQLite
State lives in SQLite in WAL mode, so it survives a restart and is safe to touch from multiple threads. The get_or_call API is async, retries back off instead of hammering, and async-context teardown is clean.
Install from PyPI
$ pip install "smartmemo[ml]"
It ships a pretrained classifier, so a cold start needs no training. CI across Python 3.11–3.14.