Simple 10-line running average beat a Bayesian model at catching agent failures
This addresses that quietly go wrong without crashing or throwing an error — drifting toward a confident, well-formatted, wrong answer. A classic case: the agent fabricates a plausible query parameter, the tool returns zero rows, and the agent reads "no results" as "no problem," then reports success, with no error and no exception until a human finds it later.
The natural framing treats the agent as moving through s — healthy, drifting, failed — and tries to infer the current state from noisy signals, the same Bayesian/ approach robotics has long used for execution . A Bayesian state estimator was planned to track belief over these failure states step by step, but before building it, a measurement rig was built first to test whether that sophistication was actually worth it.
The rig compares detectors on traces with known , scoring two things: how much healthy and pre-failure behavior overlap, and how much per-step signal exists. Two findings resulted: on synthetic traces, a simple 10-line beat the fancier Bayesian model, and — more surprising — on real traces the signal that actually separated healthy from failing behavior turned out to be semantic, not statistical.
Key points
- Problem: agents silently drifting into confident wrong answers with no crash or error
- Example: agent invents a query parameter, gets zero results, misreads it as success
- Original plan was a Bayesian state estimator using a -style approach
- A measurement rig found a simple 10-line beat the Bayesian model on synthetic traces
- On real traces, the useful detection signal was semantic, not statistical