A ‘context runtime’ that chooses how each AI request should run
A simple retrieval-based AI system can quickly turn into a large set of decisions. As features grow, the may need to choose the model, choose the retrieval method, handle memory for conversations, verify answers, route , use caching, and manage the budget. Putting all of that inside normal logic can make the system rigid and hard to change.
The proposed approach is to let the describe the goal, not the full execution path. A separate runtime would decide which model to use, which retrieval method to use, what memory to include, whether is worth the cost, and whether a cheaper strategy can still do the job. The idea is compared to a planner, which figures out an efficient way to run a request internally.
The approach has been implemented in Python and Go, with a and code available.
Key points
- AI retrieval systems can grow into routing for models, retrieval, memory, , tools, caching, and budgets.
- The proposed pattern lets the state the goal while a runtime chooses the execution plan.
- The runtime can decide whether a cheaper model or strategy is enough for a given request.
- The idea is similar to a planner, but applied to AI request execution.
- s in Python and Go are available, along with a and code.