Improving safety classifiers when labels are limited
Gnosys is presented as a tool that automatically improves prompts and when there are not enough labeled examples for normal . The test used ToxicChat, a public safety benchmark, and measured how many harmful messages were caught. Every method kept the at 5%, so the comparison shows which method caught more harm at the same level of mistaken flags.
The scores came from a that the system did not see while improving the method. In the 3,000-example run, Gnosys scored 0.777, compared with 0.731 for the starting classifier and 0.702 for GEPA. In the earlier 1,000-example run, Gnosys scored 0.909, compared with 0.788 for the starting classifier and 0.848 for GEPA.
GEPA improved in one run but dropped below the starting point in the other, which suggests that ordinary may be unreliable when labels are scarce.
Key points
- Gnosys improves prompts and when labeled examples are limited.
- On ToxicChat, Gnosys beat both the starting classifier and GEPA in two runs.
- The was fixed at 5% for every method.
- In the 3,000-example run, the scores were Gnosys 0.777, starting classifier 0.731, and GEPA 0.702.
- In the 1,000-example run, the scores were Gnosys 0.909, starting classifier 0.788, and GEPA 0.848.