Computer-use plugin: live-install receipt + first look-act loop

[Hmbown]

Remaining Engine acceptance — triage 2026-09-07

The comments establish that this bundle is built in; a separate plugin-install ceremony is not the acceptance path for a build that already includes it.

• Discover the built-in bundle in the intended release build, review/trust/enable the exact bytes and connect through the existing Engine MCP path.
• Verify the selected tools are present for the actual model before inference and remain correctly discoverable after connection changes.
• Complete a real model-triggered observe–act–verify task with the chosen app/computer, required approval and observed final result.
• Retain the originating task's sanitized tool/approval/result receipt; direct stdio or tools/list success is not a model-loop receipt.
• Qualify cancellation, disconnect/reconnect and host permission failures without stale input replay.
• Keep host self-approval prevention explicit: Computer Use must not serve as a way to approve its own pending authorization.

Related: plugin launch tracker, text-only/vision qualification, production desktop package.

The older screenshot-failure diagnosis below was corrected in the existing comments; do not repeat it as a confirmed current TCC defect.

---

Original scope and acceptance (preserved)

Draft PR #5855 holds the bundle (manifest, mcp.json stdio server, skill, /computer command). Remaining: install it in a release build via /plugin install, connect the MCP server, and run one real look-act-verify loop (screenshot, click, verify) with receipts. Server protocol (9/9), bundle validation, and live macOS screenshot already proven.

[Hmbown]

Progress receipt (2026-09-06, release build of main 7fdb613b2, then of #5918's branch; isolated CODEWHALE_HOME, no provider keys):

Install. The bundle is already a builtin in this build (crates/tui/plugins/computer-use, ID builtin/…/computer-use), so /plugin install is not the path; discovery finds it disabled and not-reviewed. Driven through the release TUI:

/plugin trust computer-use <content-hash>  -> Trust: trusted
/plugin enable computer-use                -> Plugin bundle 'computer-use': enabled.
/plugin show computer-use                  -> State: active · Active components: [skills, mcp, commands]

Connect (before #5918). /mcp showed plugin-12-computer-use-computer [error] — stdio · tools: 0; codewhale mcp connect printed only Stdio transport closed. Root cause is #5916: on macOS the reviewed launch fed Node the entry as /dev/fd/N, so ../src/registry.mjs resolved to /dev/src/….

Connect (with #5918).

$ codewhale mcp connect   -> Connected to all configured MCP servers.
$ codewhale mcp tools     -> 38 tools: computer_list … cursor_position … screenshot … left_click … recording_*

Look-act-verify loop. Driven directly over the bundle's stdio server from this host: request_access answered ok, open_application Calculator launched it, but screenshot failed (screencapture exited 1: could not create image from display), get_app_state could not find the app, and key cmd+q timed out in osascript. That is Screen Recording and Accessibility not being granted to the app hosting this engine, and request_access claiming them anyway is #5917. The loop with receipts needs a human step first: grant both permissions to the terminal app that will host the engine (System Settings → Privacy & Security), then rerun. The client script and receipts JSON are kept in the session scratchpad.

[Hmbown]

Update: the live probe from #5928 (which asks TCC directly) shows both Accessibility and Screen Recording granted to Ghostty on this Mac, so the earlier could not create image from display was a locked or sleeping display, not a permission gap. With #5918 merged the bundle's MCP server connects from the engine (codewhale mcp tools lists 38 tools). What remains for the look-act-verify receipt is running it through the guarded API from #5928 (open_application { activate: true } must report frontmost: true, then key/type with app_ref), and after the #5927 incident I am not driving keystrokes on this desktop again tonight; it should be run at the keyboard with the terminal's own session in view.

[Hmbown]

Moved to v0.9.14 for the 0.9.13 release cut (head f8f3211). Remaining step is the live look-act-verify receipt at the keyboard — MCP connect (38 tools) and TCC grants are already evidenced; the keystroke loop was deliberately deferred to an attended run. Release-blocker intent carries forward to 0.9.14.

[Hmbown]

Status (v0.9.14 run): not reached in this slice — the live-install receipt for the computer-use plugin needs a live plugin repo/marketplace install, which this environment cannot provide (recorded stop-and-ask).

[Hmbown]

Status from the 0.10.0 readiness pass (PR #6370): the Engine acceptance checklist above was not executed, and nothing is claimed.

• The pass ran in a headless Linux container: no macOS TCC, no attended keyboard, no model-triggered observe–act–verify loop.
• What was seen: the Extensions panel lists the built-in computer-use bundle as disabled with its macOS-beta note, captured through a real PTY at five terminal sizes on the rebuilt binary. That is a discovery receipt, not a model-loop receipt.

This stays open and release-gating. It needs an attended macOS run for the trust/enable, live tool discovery, one real look–act–verify task, and the cancellation / reconnect / permission-failure cases.

---

Generated by Claude Code

[Hmbown]

Removed the release-blocker label: Hunter confirms Computer Use is working on his machine, so this issue no longer gates the v0.10.0 tag. The issue stays open for the remaining acceptance receipts listed above (model-triggered observe–act–verify loop with its sanitized receipt, cancellation and reconnect qualification), which need a native macOS session and are not claimed here.

---

Generated by Claude Code