Evaluator treats safe legal refusals as agent failures
A legal-adjacent SaaS agent is designed to refuse tax and legal advice, even when the question sounds casual, such as asking what to do about a 1099 versus a W-2. The refusal behavior has been reviewed by legal teams. The design choice is that giving misleading tax or legal guidance is worse than not answering.
TestMu’s agent-to-agent creates pressure cases that try to make the agent answer anyway, such as saying the user is frustrated and only needs a yes or no. When the agent still refuses, the rubric marks it as a problem under “unhelpful_refusal_,” which lowers the scores sharply. A custom rubric override in YAML helped at the top level, but sub-scorers under “helpfulness” still flagged the refusals.
Scenario tags such as expected_refusal=true are also being tried.
Key points
- The agent is meant to refuse legal and tax advice.
- Legal review has approved the refusal s.
- TestMu’s rubric can score correct refusals as unhelpful behavior.
- A YAML rubric override may not stop lower-level helpfulness scorers from flagging refusals.
- Safety- refusals need explicit rules.