A pytest-style safety check for prompt changes
Prompts often change without the same testing discipline used for code. A small edit may look fine after checking a few answers by eye, but the chatbot can later start making up confident wrong answers. faithgate is built like pytest for prompts. It keeps a set of question, context, and expected-answer cases, then scores how faithful a prompt and are to the provided context.
It compares the new run against a baseline case by case, and exits with a failure if anything gets worse. When connected to CI, a bad prompt change can fail the PR before it reaches . The checks are intentionally strict: a run cannot pass if no cases match, if scoring did not happen, or if every score ends in an error. Each run saves a manifest with the judge model, RAGAS version, and test-suite hash.
If the judge changes between the baseline and the new run, faithgate exits with a separate code instead of treating the comparison as valid. s are saved as s, not converted into a 0.0 score. The default judge is Claude using the team’s own key, while RAGAS handles the scoring math.
Key points
- Prompt changes can be tested automatically instead of checked only by eye.
- The tool compares new prompt behavior against a saved baseline.
- CI can block a PR when a prompt change makes answers worse.
- Runs record the judge model, RAGAS version, and test-suite hash.
- s are tracked separately instead of being treated as zero scores.