Claude Code skills need tests too
After spending two months building a pack, a separate was added to check whether the skills actually followed their own rules. The harness found two real bugs. The clearest failure came from a skill named `rag-eval-harness`.
That skill was supposed to reject requests to measure a new retriever without comparing it against a baseline. Without a baseline, the number is hard to act on because there is no reference point. In the test, Claude noticed the problem but still built the anyway.
The fix was practical, not theoretical. The skill was updated with exact refusal wording, examples of acceptable user replies, and clear failure conditions.
Key points
- A for caught two real bugs.
- `rag-eval-harness` should have rejected an with no baseline, but it did not.
- Claude recognized the issue and still continued with the wrong action.
- The fix added exact refusal wording, acceptable user responses, and failure rules.
- Reusable AI tool instructions should be tested like code.