How to give an LLM lots of context without wasting cost
A platform needs a to refer to many existing nodes. When creating a DAG, the model must know many nodes that were already defined and point to them correctly while building the graph.
The main challenge is giving the model enough context while keeping response time, cost, and reasoning quality under control. may help if most of the context stays the same across requests.
Another option is with a . The concern is that this may only fetch a small set of similar items, while the model may need broad access to many nodes, not just a few close matches.
Key points
- A needs to correctly reference many existing nodes.
- DAG generation can require broad structured context, not just a few related examples.
- More context can raise cost, slow , and affect reasoning quality.
- is worth considering when most context repeats across requests.
- with a may be less reliable if many nodes must be considered at once.