Open-source tool lets anyone run self-improving AI agent loops locally

Major AI observability companies sell products that let agents review their own past work logs, spot errors and inefficiencies, and improve automatically — a technique called a self-improvement loop. These products are expensive, hard to adopt, and keep your data on their servers unless you pay enterprise prices. The underlying mechanism is an LLM acting as a judge that evaluates another agent's behavior.

A developer reasoned that anyone with a Claude Code subscription already has what it takes to run this locally, and built Kyoko, releasing it as free open-source software. Kyoko has four parts: local trace collection and storage using the Open Telemetry standard, an analysis step that spins up a local Claude Code instance to read through those traces, an evaluation harness that runs code and LLM checks to track performance over time, and a human oversight dashboard where you approve any fixes before they are applied. Everything runs on your own machine; you control your data.

It can be driven by a human through the dashboard or operated automatically via a command-line agent. In a test using Sierra's Tau2 benchmark, the same underlying model achieved a 24% performance gain after going through Kyoko's fix cycle.