A local-first metric engine idea for better LLM use
The idea is an open-source tool that measures how well people use LLMs, with the goal of making usage more efficient. The proposed metrics include intentional and unintentional , , and the Bullshit metric from the Machine Bullshit paper.
The intended output is a report that helps users review whether their model use is healthy or wasteful. A RAG library is nearly finished and could support checks by using a classifier to decide whether the change in meaning was intentional.
Other metrics appear harder to build, so existing open-source libraries or research papers are being sought. Large s may already track similar usage quality internally, but there seems to be a gap in tools that let users inspect their own LLM usage in this way.
Key points
- The proposal is for a local-first, open-source metric engine for LLM usage quality.
- The target metrics include , , and a Bullshit metric.
- A nearly finished RAG library could help detect whether was intended or accidental.
- The harder parts still need existing libraries, papers, or new work.
- For AI agents, these metrics could help find wasteful behavior that increases token cost.