Exact-match vs semantic LLM caching: don't trust a rising hit rate
This is a firsthand account of trying to cut through caching. The author started with exact-match caching: the entire request (model, messages, ) is hashed, and a cached answer is returned only if a new request hashes identically. This is safe — a single character difference causes a miss — but it turned out nearly useless for real chat traffic, since users phrase the same intent differently.
Questions like "How do I reset my password?", "I forgot my password, how do I get back in?", and "Can't log in because I lost my password" mean the same thing to a person but produce completely different hashes, so exact-match caching missed almost all of them. Switching to fixed that: prompts are converted into (numeric vectors representing meaning) and a cache hit is returned when a new prompt lands near a previous one in that vector space. The hit-rate chart looked much better, but when the author actually read the matched prompt pairs, some were near-misses — prompts that seemed similar but actually needed different answers, causing the wrong cached answer to be reused.
The conclusion: both caching methods are still used together, but a rising hit rate alone is no longer trusted without manually checking the actual matched prompt pairs.
Key points
- Exact-match caching hashes the full request and is safe, but missed nearly all real user questions because of phrasing differences
- uses to catch rephrased questions and raised the hit rate, but sometimes reused the wrong answer for prompts that only seemed similar
- The author now uses both approaches together but manually reviews actual matched prompt pairs instead of trusting the hit-rate number alone
- A higher does not by itself mean caching quality has improved