A long-context model found bugs that a RAG setup missed
An agent had to understand both a and a 300-page spec document, but the RAG setup kept failing. The stack used LangChain, LlamaIndex, Chroma, , a custom reranker, and many rounds of chunk-size tuning. Even then, the agent often pulled unhelpful instead of the actual function logic.
The clearest failure involved a global config bug. core/config.py defined the config object, main.py created it, and utils/scheduler.py changed it from a er. Because the repo had been split into chunks, the agent saw pieces of the problem but not the full chain.
It found the config definition, missed the later state change in the scheduler, and kept suggesting fixes that looked reasonable but left the in place. The alternative was to skip the vector DB, chunking, and embedding search, then put the key source files and the full spec into M3's so the agent could read everything at once.
Key points
- The RAG setup split code and docs into chunks, but missed relationships spread across files.
- The bug involved a config object created in one file and changed later by a er in another file.
- The agent found part of the evidence but missed the state change that caused the .
- The test alternative removed vector DB search, chunking, and , then used M3's .
- For agent cost planning, retrieval failure and debugging time matter as much as .