meaningcache

A semantic cache for LLM calls that runs with no torch, no server, no API key, and no GPU, plus an honest benchmark of how far that gets you.

On 5,000 real Quora question pairs it serves about 70% of reworded questions from cache at its default threshold, while wrongly matching about 28% of unrelated ones. It is a cost optimizer, not a correctness guarantee. One file, about a 60 MB model, tens of thousands of embeddings per second on CPU.

Why

A plain cache keys on the exact string. "Can I get a refund?" and "What is the policy on refunds?" are the same question, but an exact cache treats them as different and calls your model again. meaningcache matches on meaning, so near-duplicate prompts hit the cache instead. The catch is that meaning-matching also fires on questions that are similar but not the same, which is why the false-hit number above is part of the headline, not buried.

Quickstart

git clone https://github.com/reezanahamed/meaningcache
cd meaningcache
uv run python demo.py

That runs instantly and shows the idea on a small FAQ. The full benchmark on real data is a separate step below (it downloads a dataset).

Use it

from meaningcache import SemanticCache

cache = SemanticCache(threshold=0.8)

def answer(prompt):
    hit = cache.get(prompt, default=None)
    if hit is not None:
        return hit                 # served from cache, no model call
    result = call_your_llm(prompt) # only runs on a miss
    cache.set(prompt, result)
    return result

Or wrap a function so caching is automatic:

@cache.wrap
def llm(prompt):
    return call_your_llm(prompt)

How it works

1. Each prompt is embedded into a vector with a small static model (model2vec, CPU only, about 60 MB).
2. A lookup compares the new prompt to every stored prompt by cosine similarity and takes the best match.
3. If that match is at or above the threshold, the stored answer is returned. Otherwise it is a miss and you call your model.

The whole thing is one readable file, meaningcache.py. It embeds tens of thousands of prompts per second on a laptop CPU, and a lookup takes about 0.1 ms.

The threshold trade-off (measured on real data)

The threshold is the only knob, and it is a real trade-off, not a free win. Higher catches fewer rephrasings but returns fewer wrong answers. Measured on 5,000 real pairs from Quora Question Pairs, a public set where each pair is labelled as a real duplicate or not:

threshold	catches rephrasings	wrongly matches unrelated
0.70	90%	48%
0.75	82%	38%
0.80	70%	28%
0.85	55%	18%
0.90	36%	10%

(Real-cache numbers: each query is compared against all 5,000 stored prompts and the best match wins. False hits grow a few points as the cache gets bigger, because a query has more entries to spuriously match, but recall rises with it so the curve barely moves.)

There is no threshold that catches most rephrasings and almost never fires wrongly. QQP is a hard test on purpose: its non-duplicate pairs are about similar topics ("how do I learn Python" vs "how do I learn Java"), which small static embeddings struggle to tell apart. Your real traffic is usually easier than this, so treat these false-hit numbers as a pessimistic ceiling. On the clearly-distinct FAQ in demo.py, false hits are zero.

Pick your threshold by the cost of a wrong answer. If a wrong cached answer is expensive, use 0.85 or higher and accept fewer hits. If it is cheap to catch downstream, go lower for more hits.

Reproduce the benchmark

uv run --with datasets python benchmark.py         # 5,000 pairs (downloads a few hundred MB the first time)
uv run --with datasets python benchmark.py 20000   # more pairs

It prints both the pairwise and the real-cache tables above, and a sanity check that the numbers use the exact similarity that SemanticCache.get uses.

Features

Feature	Notes
No torch, no server, no key	Static embeddings, pure CPU
Fast	Tens of thousands of embeds/sec, about 0.1 ms per lookup
Handles cached None	get(prompt, default=MISSING) tells a hit from a miss
Persistence	cache.save(path) and SemanticCache.load(path)
Size limit	max_size evicts oldest; inserts stay O(n) via a growable buffer
Exact-key dedup	Re-storing the same prompt updates it, not duplicates it
Pluggable embedder	Pass your own embed function

Limitations (read these)

• This is a cost and latency optimizer, not a correctness guarantee. It can return a wrong answer when two prompts are close in meaning but need different answers. The threshold is your only control.
• Static embeddings are fast but not as sharp as large embedding models. On topically similar questions they produce false hits (see the table). If you need higher accuracy, plug a stronger embedder into the embed argument.
• Best for prompts whose answer does not depend on time, user, or private context. Do not cache answers that must be fresh or personalized.
• Lookup is linear in the number of stored prompts. Fine for tens of thousands in memory. For very large caches, put the vectors in a real vector index.
• Tuned and tested on English prompts.

License

MIT. See LICENSE.