Open-source cache reuses similar LLM calls automatically
Khazad is an open-source tool that catches requests at the HTTPX layer and reuses earlier answers when a new request is similar enough. In RAG workflows or , repeated or closely related calls can be checked before they leave the machine.
The cache uses a database to find matching past requests. If a match is found, the response can return almost immediately, which can cut waiting time and reduce .
It is designed to work with raw OpenAI clients, custom wrappers, or lightweight libraries as long as they use HTTPX underneath. It also supports server-sent events and token streaming, and it does not require running a separate proxy server.
Key points
- Khazad caches LLM requests at the HTTPX layer.
- It targets repeated calls in RAG pipelines and .
- It checks a database before sending a request to the .
- It can work across different clients if they rely on HTTPX.
- It supports token streaming and avoids an extra proxy server.