sim-use helps mobile AI agents act with fewer tokens
sim-use is a that lets AI agents read and control screens on the and Android emulators or devices. Instead of sending a large raw screen structure, it turns the screen into a short outline of visible items such as buttons, text fields, and tabs. That outline is about 16 times smaller than the raw , so an AI model can understand a whole screen with fewer tokens.
A typical loop is simple: run `sim-use ui` to read the screen, tap an item with a short name such as `tap @9`, then read the screen again to check what changed. It supports several ways to choose what to tap, including fast numbered aliases, stable IDs, visible labels, and coordinates as a backup. After the first command, a per-device reduces setup work, and each read-and-act round trip is described as taking about 300 milliseconds.
The same command style works across iOS and Android, with support for taps, swipes, typing, paste, , video recording, and app state checks. It can be installed with Homebrew, and it also includes an that teaches an AI client how to use the tool.
Key points
- AI agents can read and control and Android device screens.
- The screen outline is about 16 times smaller than the raw .
- Agents can read the screen with `ui`, act with commands like `tap @9`, then verify the result.
- A helps later read-and-act rounds finish in about 300 milliseconds.
- The tool supports taps, swipes, typing, paste, , video recording, and app state checks.