A more realistic benchmark for LLM vulnerability detection
A nearly finished tests how well LLMs find security flaws in code under more realistic conditions. It uses Juliet code as the base, but changes how the code looks so it resembles a real instead of a familiar set of known examples. That keeps the while reducing the chance that an LLM succeeds only because it has seen similar CWE samples before.
The code also includes LLM-written comments that can be accurate, misleading, or neutral. This makes it possible to test whether plain-language comments can push an LLM toward the wrong security judgment. The covers hundreds of CWE types and includes enough code to nearly fill the .
The remaining work is presentation, testing published LLMs, and possibly removing a few CWE cases that are too easy to catch by accident.
Key points
- The disguises Juliet code so it looks more like a real .
- It keeps , so model answers can still be checked.
- It tests whether accurate, misleading, or neutral comments change LLM decisions.
- It includes hundreds of CWE types and long inputs that nearly fill the .
- Published LLM comparisons and final cleanup are still unfinished.