Cutting latency and token cost in legal agentic RAG
A legal system that searches a large document collection, uses , and fetches web pages can become slow and expensive because it may run several rounds of searching, checking, and answer building. Even with strong tools, s, and , applying the full research process to every question can make latency and token use unacceptable. The main design advice is to reduce the number of loops, batch searches instead of doing one search at a time, and run independent retrieval steps in parallel.
A query should be classified first, so simple lookups, citation-heavy answers, multi-document comparisons, current-law checks, and high-risk questions can take different paths. Retrieval should start narrow and widen only when the evidence is weak; after rerank, only the best evidence should be sent to the model. also falls when the system stops passing the full conversation and research trail into every step.
should be selective: simple checks such as citation presence, jurisdiction, date, source authority, and contradictions can be handled cheaply, while costly LLM can be saved for risky cases. For legal work, a 20 to 40 second answer can be acceptable if the answer has strong citations and an audit path, especially when the interface streams clear progress updates.
Key points
- Running the full research loop for every question can waste time and tokens.
- Classify the query first so simple and risky questions use different paths.
- Batch searches and parallelize retrieval work when steps do not depend on each other.
- Send only selected evidence to the model instead of the full research trail.
- Use cheap rule-based first, and reserve LLM for high-risk cases.