The first publicly published macOS-native computer-use benchmark for autonomous agents. As far as we know.
369 task slots defined ← matches OSWorld's task count exactly
50 implemented (v0.1) ← runnable today
319 stubbed (v0.1) ← real prompts + categories; no setup.sh / eval.sh yet
→ filled in progressively v0.2 → v1.0
15 categories Finder · Safari · Mail · Notes · Calendar ·
Reminders · Settings · Terminal · Pages ·
Numbers · Keynote · Music · Photos · Maps ·
Multi-app
3 difficulty tiers T1 single-step · T2 multi-step · T3 cross-app
agent-agnostic any binary that takes a prompt → drives macOS
kinclaw v1.15.0 + Kimi-K2.5(cloud) on macbench v0.1
IMPLEMENTED: 101 / 150 = 67.3%
STRICT: 101 / 369 = 27.4% (stubs count as fail)
Total time: ~95 minutes (with per-task isolation)
For context (these benchmark different OS surfaces, so they're not directly comparable, but it's the closest cross-comparison available):
| Agent + Brain | Benchmark | Score |
|---|---|---|
| kinclaw v1.15.0 + Kimi-K2.5 | macbench v0.1 (macOS) | 67.3% IMPLEMENTED / 27.4% STRICT |
| Anthropic Computer Use (Claude Sonnet 4) | OSWorld-Verified (Ubuntu) | ~38% |
| GPT-4o + Set-of-Mark | OSWorld | ~12-15% |
The categories where kinclaw was strongest: Finder (28/39 ≈ 72%), Reminders (75%), Settings (~68%), Calendar (>50% post-isolation). Notes / Mail / Pages / Numbers / Keynote are weakest — partly real agent limitations, partly v0.1 task-design choices (some require infrastructure beyond bash + AppleScript that's deferred to v0.2).
When you run the suite, the runner reports two separate pass rates:
IMPLEMENTED: P / I (X.X%)— passed P of I tasks that have setup.sh + eval.sh, ignoring stubs. The "interesting" score: what the agent did against tasks it could actually try.STRICT: P / 369 (Y.Y%)— passed P of 369 total task slots, stubs counted as fail. The "long-game" score: progress against the full benchmark including unimplemented work.
Both numbers go in run.json. Leaderboards / blog posts must report
both — STRICT alone hides agent capability behind v0.1's incomplete
implementation; IMPLEMENTED alone hides how much benchmark is missing.
The runner snapshots PIDs of bench-touched apps (Safari, Mail, Notes, Calendar, Reminders, Music, Photos, Maps, TextEdit, Pages, Numbers, Keynote, System Settings) at startup. Between every task, it kills only the PIDs the bench itself spawned, leaving any pre-existing user instance untouched. This:
- Prevents the "agent does 5 tasks worth of work in one prompt" pollution we observed in the first run (root cause was actually pilot-soul memory + reuse of the same Safari window across tasks).
- Stops Notes / Calendar / Reminders from accumulating AppleScript hangs after ~5-10 invocations.
- Doesn't nuke the user's pre-existing app windows — if you're running bench while Safari is open with your work, that Safari stays alive.
The startup line (isolation: N pre-existing PIDs across 14 apps will be preserved) confirms the snapshot was captured.
./warmup.sh (or make warmup) does four things before bench:
- Force-quit every app the bench touches (so PID snapshot starts empty — strongest isolation).
- Wipe the bench sandbox (
~/Desktop/kinbench/,~/.kinbench/). - Clean any KinBench-prefix data in app data stores (Notes / Reminders / Calendar / Mail / Photos / Music — leftover from prior runs that crashed before teardown).
- Probe each app via osascript with a 5-second timeout. Reports ✓ healthy / ⚠ HUNG / ✗ TCC denied per app.
make bench auto-runs warmup. Set SKIP_WARMUP=1 to skip (useful
during eval-script iteration when you don't want to nuke state).
OSWorld (NeurIPS 2024) is the de facto standard for desktop computer-use agents — but it benchmarks inside an Ubuntu/Windows VM. Nobody has published a comparable benchmark for macOS native apps. macbench fills that gap.
- Apple Intelligence is rolling out, but Apple hasn't published a benchmark for measuring agent capability on Mac.
- The OSWorld leaderboard tells you Claude Sonnet 4 hits ~38% on Linux desktop tasks. It tells you nothing about how the same model drives Finder, Mail, Calendar, Notes, System Settings — apps people actually use on Macs.
- macbench measures that. Same three-file pattern as OSWorld
(
task.json+setup.sh+eval.sh), same evaluator-script philosophy. Different OS surface.
Each task is a natural-language prompt — exactly what a user would
type into a chat box — that the agent must complete by driving real
macOS apps. The eval is deterministic: filesystem state, defaults
read, sqlite queries, AppleScript Automation queries. No "ask another
LLM if it looks right".
Sample tasks (all 50 in tasks/):
| ID | Category | Difficulty | What |
|---|---|---|---|
| 001 | finder | T1 | Rename a file |
| 005 | finder | T2 | Compress 3 files into a zip |
| 011 | safari | T1 | Search Google for a phrase |
| 016 | T1 | Compose a draft (don't send) | |
| 018 | calendar | T1 | Create event tomorrow at 12:30 PM |
| 021 | settings | T2 | Turn on Do Not Disturb |
| 029 | multi-app | T3 | Take screenshot → attach to draft email |
| 048 | multi-app | T3 | Find file in Finder → email it as attachment |
| 050 | multi-app | T3 | Pages doc → export as PDF |
See tasks/ for the full set, and
AUTHOR_GUIDE.md for the schema if you want to
write more.
You need:
- macOS 14+ (Sonoma or newer)
- Go 1.22+ (only to compile the runner; tasks themselves are shell + AppleScript)
- An agent binary that takes a prompt and drives macOS — e.g. kinclaw, or a wrapper script around any vision-LLM-driven framework
# 1. Clone + build
git clone https://github.com/LocalKinAI/macbench
cd macbench
make build
# 2. One-time AppleScript Automation TCC priming (Mail / Notes /
# Calendar / Reminders / Safari) — click "Allow" on each popup
make warmup
# 3. Run the benchmark with your agent
make bench AGENT=/path/to/kinclaw \
AGENT_ARGS='-soul pilot.soul.md -exec {prompt}'
# Or: just N tasks
make bench AGENT=./kinclaw AGENT_ARGS='-exec {prompt}' TASKS=001,005,016
# Or: with screen recording (mp4 per task)
make bench-record AGENT=./kinclaw AGENT_ARGS='-exec {prompt}'Output:
macbench: 50 task(s), agent=kinclaw, args="-soul pilot.soul.md -exec {prompt}", record=false
──────────────────────────────────────────────────────────────────────
✓ 001-finder-rename T1 1284ms
✓ 002-safari-open-url T1 3122ms
✗ 005-finder-zip T2 8941ms [eval] expected ≥3 files in archive
...
──────────────────────────────────────────────────────────────────────
PASSED: 19 / 50 (38.0%)
→ wrote results/20260508-141530/run.json
Per-task report saved to results/<timestamp>/run.json. If
-record was on, mp4s land in results/<timestamp>/recordings/.
-agent-args is a template. The literal string {prompt} gets
replaced with the task's natural-language prompt at run time.
Tokens are split on whitespace (no shell quoting), so each {prompt}
becomes exactly one argv slot.
# kinclaw (LocalKinAI agent)
-agent kinclaw \
-agent-args "-soul pilot.soul.md -exec {prompt}"
# Anthropic Computer Use wrapper (hypothetical)
-agent anthropic-cua \
-agent-args "--task {prompt} --max-tokens 4096"
# Any custom shell wrapper
-agent ./my-wrapper.sh \
-agent-args "{prompt}"If your agent's CLI shape isn't expressible as a single template, write a 3-line shell wrapper. macbench treats the agent as a black box.
macbench tasks drive real macOS apps, so your agent binary needs:
- Accessibility — System Settings → Privacy & Security → Accessibility → toggle on.
- Screen Recording — same screen, Screen Recording → on.
- AppleScript Automation for Mail / Notes / Calendar / Reminders /
Safari / System Events.
make warmuptriggers any missing dialogs in one batch. - (optional) Screen Recording for
kinrec— only if you usemake bench-record. The kinrec binary needs its own grant.
All grants are one-time; macOS remembers per code-signing identity.
| macbench | OSWorld | WebArena | AndroidWorld | WindowsAgentArena | |
|---|---|---|---|---|---|
| Platform | macOS native | Ubuntu / Windows VM | Browser only | Android emulator | Windows VM |
| Total task slots | 369 | 369 | ~800 | ~116 | ~150 |
| Implemented (v0.1) | 50 | 369 | 800+ | 116 | 150 |
| Eval style | filesystem + defaults + AppleScript + sqlite |
filesystem + ROS + screenshot match | DOM + side-effect | UI tree + side-effect | filesystem + registry |
| Agent contract | exec(prompt) → drive macOS | VNC channel into VM | browser DOM API | adb-style | RDP-style |
| First public release | 2026 | 2024 | 2023 | 2024 | 2024 |
We borrow OSWorld's three-file pattern and difficulty taxonomy, but the implementation is original.
- 319 stub tasks need setup.sh + eval.sh — every stub already has
a real, specific prompt and a well-thought category/difficulty
assignment; what's missing is the deterministic eval. See
ROADMAP.mdfor the per-category implementation schedule across v0.2 → v1.0. - Anthropic / OpenAI / open-source agent backends. Right now we've only run the suite with kinclaw; contributions wiring up other agents are first-class welcome.
- CI — needs a Mac runner with TCC pre-granted. GitHub Actions macos runners can't be granted programmatically, so this requires a self-hosted Mac mini. Probably v0.3 or v1.0.
- Token / cost tracking — most agents don't expose this uniformly. Punted to v0.2.
- Leaderboard site. The README will start carrying numbers as we collect them.
# 1. Pick a stub from tasks/ — its task.json has the spec
cat tasks/051-finder-show-hidden-files/task.json
# 2. Write setup.sh + eval.sh per AUTHOR_GUIDE.md
$EDITOR tasks/051-finder-show-hidden-files/setup.sh
$EDITOR tasks/051-finder-show-hidden-files/eval.sh
chmod +x tasks/051-finder-show-hidden-files/{setup,eval}.sh
# 3. Remove the "stub" status field from task.json
$EDITOR tasks/051-finder-show-hidden-files/task.json
# 4. Verify standalone (per AUTHOR_GUIDE.md self-check)
cd tasks/051-finder-show-hidden-files
bash setup.sh
# (manually do what the prompt says)
bash eval.sh && echo PASS
# 5. Open a PRMIT. See LICENSE.
The three-file evaluator pattern + difficulty taxonomy are inspired by OSWorld (Apache-2.0); all task content + runner implementation here are original.
- kinclaw — Pure-Go macOS computer-use agent. The reference implementation that drives this benchmark.
- OSWorld — the inspiration. If you're benchmarking on Linux/Windows, use that.
AUTHOR_GUIDE.md— how to write a new task.ROADMAP.md— path from 50 → 369.