Browser agent benchmarks may hide weak real-world performance
AI agents that use a computer or browser must find the right buttons, icons, and page areas on screen. Three screen-understanding models, , , and GTA1, all scored above 90% on ScreenSpot-v2. Those high scores did not hold up when the tasks were changed in simple, realistic ways.
Accuracy fell by 27 to 56 points when the browser zoom was set to 70%, the page style was changed, or the instruction used a relationship such as “the icon above the search bar.” Training again on the examples it failed did not fix the problem. The newly trained model performed worse than the original model in every setup. Increasing the training set from 6,500 to 25,000 examples made the drop larger.
Both and real failure examples hurt , so the issue is likely the training recipe, not just bad data. ScreenSpot-v2 barely changed during this, meaning a team watching only that benchmark could think a worse model had improved.
Key points
- Three screen-use models scored above 90% on ScreenSpot-v2.
- Small realistic changes caused accuracy to fall by 27 to 56 points.
- on failure cases made it worse in every tested setup.
- Using 25,000 training examples caused a larger than using 6,500 examples.
- A single benchmark can hide whether a is becoming less reliable.