Analytics agents may fail more from weak context than bad SQL
The nao team spent a year building analytics agents on real company databases and found that SQL generation was not the main hard part. Modern LLMs can often write usable SQL. The bigger failures came from the information around the query: unclear business definitions, changing metric rules, missing , poor , made-up table joins, and weak evaluation.
For example, an agent needs to know what “active user” means for the current quarter, not just how to query a table. nao built its around , where the data team controls schemas, metrics, rules, documents, and custom tools instead of leaving the agent as a black box. The data team uses a CLI to connect a data warehouse, sync metadata, define rules, write , and run tests.
Business users then ask questions in plain English through a chat interface. The company says it is backed by Y Combinator.
Key points
- Modern LLMs may handle SQL reasonably well, but still fail when is unclear.
- Definitions like “active user” need to be controlled and kept consistent over time.
- The framework gives data teams control over schemas, metrics, rules, documents, and tools.
- A CLI is used to set up and test the context before business users ask questions in chat.
- No concrete token savings or cost reduction numbers are provided.