hand
more from customer0Let your AI agent use a computer without guessing where to click.
A small model picks the button. hand clicks it and checks it worked.
bun add github:acoyfellow/hand- click "Email notifications"95%
- click "Save changes"94%
- done94%
A real run. Each step shows how sure the model was.
It finished all 30 tasks, three times over, at 1/18 the cost of Claude Opus.
A real 38 s run. hand writes the message and makes every click.
How it works
Use it with an AI agent
Works with Claude, Pi or any MCP client, on a Mac.
bun add github:acoyfellow/hand swiftc -O node_modules/hand/native/ax.swift -o node_modules/hand/native/ax
{
"mcpServers": {
"hand": {
"command": "bun",
"args": ["node_modules/hand/src/mcp.ts"]
}
}
}
It never moves your mouse, and it reads back what it typed.
Use it from code
await hand.act('switch to the termlab tab'); await hand.type('coey.dev', 'the address bar'); await hand.look('Is coey.dev open?');
Not tested yet
- Apps with unlabelled icons or canvas UIs.
- A scored set of real Mac tasks.
- Windows and Linux.
Details: every result, writer models, the benchmark
All results
| setup | what drives it | finished | $/task |
|---|---|---|---|
| hand (Clef + writer) | Clef picks every action; a writer fills fields only when the task needs new text | 30.0/30 | $0.0016 |
| Clef + copier | Clef picks every action; Clairic copies values out of the task, never writes new ones | 25.0/30 | $0.0015 |
| Opus, screenshots | Claude Opus reads screenshots and clicks | 22.7/30 | $0.029 |
| Opus + Clef | Opus plans, Clef clicks | 22.0/30 | $0.029 |
Writer models
Scored alone on 44 messy requests. The default pairs a cheap model with a backup: glm-5.3-flash, then kimi-k2.6 for any field it leaves empty. 42–43 of 44 across runs, at $0.00009 a request. All writers.
The benchmark
Every change to the library has to pass a fixed benchmark: 30 tasks in a test app, success read from the app's own state, and a gate that fails if results get worse.
bun run check # types, lint, tests bun run gate # fails if results are stale or worse bun run bench # real models, writes bench/runs/<id>/