Skip to content

feat(tools): add computer-use tool (§6.13) - #331

Merged
gnanirahulnutakki merged 1 commit into
devfrom
feat/computer-use-6.13-2026-07-14
Jul 17, 2026
Merged

feat(tools): add computer-use tool (§6.13)#331
gnanirahulnutakki merged 1 commit into
devfrom
feat/computer-use-6.13-2026-07-14

Conversation

@gnanirahulnutakki

Copy link
Copy Markdown
Member

Summary

  • Adds crates/computer-use (ardur-computer-use), Phase 1 of the §6.13 computer-use blueprint: a closed 10-action vocabulary (capture, click, double_click, right_click, middle_click, drag, scroll, type, key, wait), an in-substrate BlockedActionPolicy denylist (5 hard-blocked key combos + 6 dangerous-text patterns, adapted from Hermes's computer_use/tool.py:80-107) enforced regardless of cap-token or consent state, a cap-token-shaped ComputerUseCaveat (per-action-kind permitted set), a session-scoped ConsentBroker (grant/revoke/TTL), the computer.{session.started,consent.granted,consent.revoked,action.invoked,action.denied,screenshot.taken}.v1 receipt family with a DenialSource that distinguishes blocked-action/cap-token/consent refusals, and a ComputerUseTool (computer.use) gated by a custom computer_use capability. Gate order: denylist → caveat → consent.
  • Phase 1 backend (LocalInputDriver) uses enigo (mouse/keyboard synthesis) + xcap (screen capture) over public macOS APIs. The blueprint's own MVP decision explicitly rejects the private SkyLight-SPI path Hermes uses (SLEventPostToPid, etc.), gating it behind a separate computer_use_skylight_spi cap-token caveat reserved for a future phase — so this PR isn't under-delivering relative to the blueprint's own stated MVP scope.
  • Documented Phase 2 TODOs in lib.rs: set_value/list_apps/focus_app (need AXUIElement accessibility-tree access, which neither enigo nor xcap expose), the som/ax capture modes, AXIsProcessTrustedWithOptions/CGPreflightScreenCaptureAccess OS-permission preflight, a real interactive consent dialog UI (ConsentBroker ships the state machine only), and cap-token-to-caveat projection from a verified Biscuit block.
  • Companion PR to feat(tools): add sandboxed code-execution tool (§6.7) #321 (§6.7 code-execution) — together they close the /goal directive's tool-surface domain (code-execution + browser/computer-use). Browser automation (§6.9/§6.10) already ships as crates/browser and is not touched here.

Test plan

  • cargo build -p ardur-computer-use
  • cargo test -p ardur-computer-use (33/33 passing — denylist/caveat/consent gate-order tests, action-parsing tests, and a wait dispatch test; deliberately no test exercises real click/type/screenshot execution, since that would require Accessibility/ScreenRecording permission granted to the CI runner)
  • cargo fmt --check
  • cargo clippy -p ardur-computer-use --all-targets -- -D warnings
  • cargo check --workspace --all-features (no cross-crate breakage from the new enigo/xcap/image/hex dependencies)

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 851e5c24-ea4b-482e-85ce-36575c03b60d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/computer-use-6.13-2026-07-14

Comment @coderabbitai help to get the list of available commands.

gnanirahulnutakki pushed a commit that referenced this pull request Jul 14, 2026
PR #331's CI failed two checks: cargo-deny (RUSTSEC-2026-0194/-0195,
DoS advisories in quick-xml 0.39.4) and ubuntu-latest/stable (wayland-sys
build script couldn't find wayland-client.pc via pkg-config). Root cause
for both: enigo/xcap/image were plain dependencies, so Cargo resolved
xcap's Linux Wayland backend (libwayshot-xcap -> wayland-client ->
wayland-scanner -> quick-xml) into the graph even though this crate only
targets macOS.

Move enigo/xcap/image to a `[target.'cfg(target_os = "macos")'.dependencies]`
section and cfg-gate driver.rs's real implementation behind
`#[cfg(target_os = "macos")]`, with a `#[cfg(not(target_os = "macos"))]`
fallback that reports the platform as unsupported rather than failing to
compile. Verified via cargo tree --target x86_64-unknown-linux-gnu and
cargo metadata's dep_kinds that the wayland chain is now unreachable for
any single real build target.

Also add the two RUSTSEC ids to deny.toml's advisories.ignore (matching
the file's existing 4-entry precedent) as defense-in-depth: wayland-scanner
0.31.10 (latest) hard-pins quick-xml = "^0.39" with no upgrade path
available, and the parser only runs at build time over vendored, trusted
protocol XML, not attacker-reachable input.

Checkpoint: architect/sessions/2026-07-14-computer-use-6.13/journal.md
Signed-off-by: Gnani Nutakki <gnani.nutakki@gmail.com>
@gnanirahulnutakki
gnanirahulnutakki force-pushed the feat/computer-use-6.13-2026-07-14 branch from 805d4ca to 30dae2c Compare July 14, 2026 07:08
Signed-off-by: Ardur <team@ardur.ai>
@gnanirahulnutakki
gnanirahulnutakki force-pushed the feat/computer-use-6.13-2026-07-14 branch from bb10f6e to 2480bb0 Compare July 17, 2026 19:25
@gnanirahulnutakki
gnanirahulnutakki merged commit 49a2f15 into dev Jul 17, 2026
12 checks passed
@gnanirahulnutakki
gnanirahulnutakki deleted the feat/computer-use-6.13-2026-07-14 branch July 17, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant