A practical self-check pattern for hermes-agent systems
Prometheus runs on and repeats a research cycle by itself. Its first build took about one month on one consumer PC, and it has kept running since then. The system creates its own research questions, sends work to several worker programs, runs experiments, keeps the code behind those experiments, and tracks the claims that come out of them.
It has produced about 133,000 experiments and 77,000 claims so far. A major part of its work is trying to find where its own conclusions fail before trusting them. It uses , checks for counterexamples in other fields, compares claims with existing literature for novelty, and tests for independence and .
On about 500 internal checks, it picked claims that would transfer to new fields with 58% success, only slightly better than chance, so it lowered its transfer confidence. One audit found that about 98% of its discovery list had not been checked against real-world data and had only passed written by its own worker programs.
Key points
- Prometheus uses to create questions, run experiments, and track claims.
- It has logged about 133,000 experiments and 77,000 claims.
- It spends work on and checks.
- Its transfer score was 58% on about 500 internal checks, so it reduced transfer confidence.
- About 98% of its discovery list had only been tested in its own , not with real-world data.