Cutting chatbot cost when data changes every 5 minutes
A chatbot needs to use backend data that is refreshed through APIs every 5 minutes. The dataset is large, so sending all of it to the LLM on every request would use too many tokens and raise cost. A RAG setup may also be awkward when the knowledge changes this often, because results can become stale or need frequent rebuilding.
A better would need to fetch only the small slice of data needed for each question, possibly using search, , caching, , and query planning together. The core challenge is making frequently changing data usable without overloading the model input.
Key points
- Backend data is updated through APIs every 5 minutes.
- The dataset is too large to send to the LLM for every chat request.
- A normal RAG setup may struggle when the knowledge changes very often.
- Possible building blocks include search, , caching, , and query planning.
- The main goal is to keep answers current while reducing tokens and cost.