DeepSWE tests how well coding agents handle real code work
DeepSWE is a new benchmark for checking how well advanced perform on real software engineering tasks. Its tasks are written from scratch, instead of being adapted from existing commits or , which lowers the chance that a model already saw the answer during training.
The tasks cover 91 repositories across 5 s. The prompts are about half as long as prompts, but the solutions require 5.5 times more code and about 2 times more .
The tests are hand-written to check whether the software behaves correctly, not whether the code follows one expected . DeepSWE is open-source and available on GitHub.
Key points
- DeepSWE is a benchmark for doing software engineering tasks.
- The tasks are newly written, which helps avoid training-data leakage.
- It covers 91 repositories and 5 s.
- Its prompts are shorter than , but solutions need much more code and about twice as many .
- Its checks focus on whether the software works correctly, not on matching a specific coding style.