Desktop agent uses screenshot coordinate grids instead of DOM to click
A agent called Rosply finds where to click by overlaying a coordinate grid on a screenshot, rather than reading the page structure (DOM) the way web usually do. It works in a loop: take a screenshot, have a such as Qwen VL or GPT-o reason about what it sees, then perform a mouse or keyboard action, and repeat.
To make clicks land accurately, it uses a grounding approach, first estimating a rough area and then narrowing in on the precise spot, and it also includes loop detection to catch the agent getting stuck repeating the same action. The project is and .
Key points
- Rosply locates screen elements via screenshot plus a coordinate grid instead of DOM scraping
- It uses s like Qwen VL or GPT-o to interpret screenshots and decide where to click
- The loop is: screenshot, vision reasoning, mouse/keyboard action, repeat
- A grounding approach narrows from a rough area to a precise click point
- Loop detection is included to catch the agent repeating the same action