From 80fd8c8dda13ee0feb916daf07811d3e928b96d5 Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Tue, 11 Aug 2026 16:26:10 +0530 Subject: [PATCH 01/75] docs(computer-use): approach - replicate the mobile-use stack on desktop Research + decided direction: intents and MCP as primary action paths, vision-based agent loop as fallback, engine as @offgrid/use in shared, vision model as a downloadable catalog entry (GUI-Owl-1.5 / Qwen3-VL). Co-Authored-By: Claude Fable 5 --- docs/COMPUTER_USE.md | 146 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 docs/COMPUTER_USE.md diff --git a/docs/COMPUTER_USE.md b/docs/COMPUTER_USE.md new file mode 100644 index 00000000..c60fa560 --- /dev/null +++ b/docs/COMPUTER_USE.md @@ -0,0 +1,146 @@ +# Computer use - replicate the mobile-use stack on desktop + +**Status:** direction decided August 11, 2026. Intents + MCP are the primary action paths; the vision-based agent is the fallback. We do not innovate on agent architecture - we study the mobile-use ecosystem and replicate it. The engine is built in the shared repo (`off-grid-ai/shared`) as an `@offgrid/*` package. Model size is not a design constraint - local models keep improving and the vision model ships as a downloadable catalog entry. +**Constraint (standing):** local models only. No hosted APIs. No screenshot ever leaves the device. + +--- + +## 1. The decision + +The agent executes actions on the user's Mac - from "create a calendar event Thursday 3pm" to "pick the best photo from the vacation album in the family WhatsApp chat and send it". The decided shape: + +1. **Intents + MCP first.** Deterministic action surfaces - MCP connectors, URL schemes, AppleScript/Apple Events, EventKit and friends, Shortcuts - handle everything they can. No pixels, no coordinates. +2. **Vision-based agent as fallback.** When no deterministic surface exists, a multi-agent GUI loop takes over, perceiving through the accessibility tree plus screenshots and acting through synthetic input. +3. **Replicate, do not invent.** The mobile-use ecosystem solved this problem in 2025-26 with measured results (100% on AndroidWorld). We port its architecture to macOS. +4. **Engine in shared.** The agent loop, action schema, router, and verification logic are platform-agnostic and land as a package in `off-grid-ai/shared`, with a desktop adapter in this repo - the same engine + adapter split `@offgrid/clipboard` already uses. A mobile adapter can follow later. + +## 2. What we are replicating + +Three systems define the mobile-use pattern; their published numbers are the calibration: + +| System | What it is | Score | License | +| --- | --- | --- | --- | +| **minitap/mobile-use** | multi-agent framework (LangGraph-style), a11y-tree-primary + vision-selective | **100% AndroidWorld** (116/116; human ~80%) | Apache-2.0 | +| **mobilerun** (ex-droidrun) | Manager/Executor framework over an on-device accessibility portal | 91.4% AndroidWorld | MIT | +| **Alibaba Mobile-Agent-v3 / GUI-Owl-1.5** | 4-role framework + open-weight GUI models (2B-32B, Qwen3-VL base) | 73.3% AndroidWorld; GUI-Owl-1.5 is open SOTA on desktop (56.5 OSWorld) | MIT weights | + +The field converged on one architecture, and it matches the decided direction exactly: + +- **Route to the cheapest surface.** Deterministic path if one exists (MCP, CLI, deep link/intent) - structured UI action second - vision-grounded action last. The 2026 benchmarks (MobileWorld, PhoneHarness, OSWorld-MCP) all show hybrid routing beating GUI-only. +- **A11y tree is the primary perception, vision is selective.** mobilerun's measurement: the tree is ~2 KB against ~1 MB screenshots - smaller, faster, semantically richer. The decision model receives tree and screenshot together; structure targets, pixels disambiguate. +- **Cognitive separation across small agents.** Role-scoped contexts, most roles on small models, one strong vision model where it counts. minitap's ablation: this separation alone is worth +21 points. +- **Deterministic verification.** Fragile operations (typing) are verified procedures: act, re-read device state, diff. Worth +7 points. A reflector step classifies each transition SUCCESS/FAILURE and feeds replanning. Cycle detection with forced strategy change is worth +9. + +### 2.1 The loop (minitap's graph, the richest reference) + +| Role | Job | Model class needed | +| --- | --- | --- | +| Planner | decompose the goal into ordered subgoals | small text model | +| Orchestrator | subgoal lifecycle (pending / in-progress / completed / failed), decides what runs next | small text model | +| Contextor | fetch fresh device state before each decision: a11y tree + screenshot + focused app | small text model | +| **Cortex** | the one decision maker: gets tree + screenshot, emits one structured JSON decision; detects action cycles and forces strategy changes | **the strong vision model** | +| Executor | parse the decision into concrete tool calls, deterministically - no free-form reasoning | none (code) | +| Summarizer | compact history so context never overflows | small text model | +| Reflector (Mobile-Agent-v3) | compare intended vs actual state transition, SUCCESS/FAILURE + diagnosis | vision model | +| Notetaker (Mobile-Agent-v3) | persist critical on-screen facts (codes, names) across subgoals | small text model | + +The 100% AndroidWorld run mixed models per role and only the Cortex was frontier-grade. That maps directly onto our stack: bundled Gemma runs planner/orchestrator/contextor/summarizer; the downloadable vision model runs Cortex and Reflector. + +### 2.2 The action schema + +A small closed enum with structured arguments, not an open toolbox. mobilerun ships nine tools: `click, long_press, type, system_button, swipe, open_app, get_state, take_screenshot, complete`. Mobile-Agent-v3's desktop set: `key, type, mouse_move, click, drag, right_click, middle_click, double_click, scroll, wait, terminate`. Element targeting always has a fallback chain: stable ID - text match - coordinates. We adopt the same shape (our chain: AXIdentifier/role+title - text match - coordinates). + +## 3. Desktop translation - what maps, what does not + +The macOS automation research (unchanged, still the ground truth for the adapter): + +| Mobile concept | macOS equivalent | +| --- | --- | +| Accessibility portal APK / UIAutomator2 / ADB | none needed - our app runs on the target machine and IS the portal: `AXUIElement` (read + `AXPress` + set `AXValue`), `CGEvent` post, ScreenCaptureKit capture we already have | +| Android intents / deep links | URL schemes (`open -u`), `open -b ` launch, AppleScript/Apple Events, `shortcuts run` (App Intents), MCP connectors | +| `resource-id` targeting | AXIdentifier / AXRole+AXTitle - text - coordinates | +| One fullscreen app | multi-window, multi-display, z-order: window-scoped capture + per-window AX trees; Retina points-vs-pixels scaling (2x) handled in the adapter | +| Uniformly rich Android a11y trees | uneven: AppKit good, Electron needs `AXManualAccessibility` poked on (Electron 25+), Catalyst varies, canvas apps expose nothing. **The vision fallback carries a larger share of steps on macOS than on Android - which is why the vision model is a first-class component, not an afterthought** | +| Sideloaded accessibility service | TCC permissions: Accessibility (AX + CGEvent), Automation per target app (`com.apple.security.automation.apple-events` entitlement + usage strings in the build), Calendars/Contacts/Photos usage keys, Screen Recording (held). Sequenced behind an explicit onboarding | +| AndroidWorld benchmark | OSWorld-Verified + OSWorld-MCP + macOSWorld for sanity checks. Expect desktop scores 20-30 points below mobile headlines for the same models - desktop is the harder domain | + +Known hard target, kept as the acceptance case: WhatsApp Desktop (Catalyst, near-dead AX tree). Route: `whatsapp://send?phone=...` URL scheme to open the chat (intent path), keyboard-first navigation, vision-grounded clicks for the gaps, photo ranking by the vision model, the send click behind the approval gate. + +## 4. Models + +No size constraint. The vision model is a downloadable catalog entry (our catalog already gates on RAM and ships mmproj projectors), and it is swappable as the space improves - which it does quarterly. + +| Model | Sizes | License | Why | Scores | +| --- | --- | --- | --- | --- | +| **GUI-Owl-1.5** (Alibaba) | 2B/4B/8B/32B, Instruct + Thinking | MIT, Qwen3-VL base (llama.cpp-supported) | trained natively for desktop + mobile + browser; best open desktop numbers | family 56.5 OSWorld; 8B-Instruct: 52.3 OSWorld-Verified, 69.0 AndroidWorld | +| **Holo3.1** (H Company) | 0.8B-9B dense, 35B-A3B MoE | Apache-2.0, official Q4 GGUF | built for local: ~140 ms step time, MoE decodes fast | 79.3 AndroidWorld (35B-A3B), 71.0 (4B) | +| UI-TARS-2 lineage | 7B+ | Apache-2.0 stack | single-model alternative if we ever want end-to-end | 47.5 OSWorld | + +Working recommendation: **GUI-Owl-1.5-8B-Instruct as the default Cortex/Reflector model** (MIT, best desktop training), 32B for big Macs, Holo3.1 as the speed-focused alternative; bundled Gemma for every other role. Decide in review; the engine treats the model as per-role config (minitap's `llm-config` pattern), so this is not a one-way door. + +## 5. Where the code lives + +### 5.1 `@offgrid/use` - new package in `off-grid-ai/shared` + +Platform-agnostic engine, mirroring the `@offgrid/clipboard` engine + adapter pattern and slotting into the roadmap's syscalls layer next to `@offgrid/skills`: + +- the agent graph (planner / orchestrator / contextor / cortex / executor / summarizer / reflector / notetaker) with per-role model config against an OpenAI-compatible endpoint (our gateway) +- the closed action schema + structured decision types +- the router: deterministic surface - structured UI - vision, chosen by the orchestrator +- verification: deterministic post-condition checks, SUCCESS/FAILURE reflection, cycle detection +- risk classification + an approval-callback seam (the host app decides how approval happens) +- a `DeviceController` interface the platform adapters implement: `getState()` (serialized tree + screenshot + focused app), `act(action)`, `openIntent(url)`, `listSemanticSurfaces()` + +Tested in the package against a fake `DeviceController` before any surface wires it in (the shared repo's standing rule). + +### 5.2 Desktop adapter - this repo + +| Seam | Today | Change | +| --- | --- | --- | +| Swift helper | AX read-only (`electron/accessibility/main.swift`); CGEvent tap listens only | add `AXUIElementPerformAction`, set-`AXValue`, `AXManualAccessibility` wake, `CGEvent` post, indexed-element serialization; same `execFile` pattern as `src/main/ocr.ts` | +| Perception | `src/main/vision.ts` capture + Vision-framework OCR | window-scoped capture into `getState()`; OCR emits `{text, bbox}` | +| Semantic surfaces | MCP connectors via `ToolExtension` (`src/main/tools.ts:401`) | add EventKit/Contacts/PhotoKit Swift-helper tools, AppleScript tools, `shortcuts run`, URL-scheme opener | +| Tool dispatch | agentic loop `toolChat`, abort guard already drops unexecuted side effects on cancel | the use-engine runs as a new extension; `ToolResult` side channel (`src/main/tools.ts:50`) extended so state fetches can return images | +| Approvals | `mcp:proposeApproval` hook, name-regex risk | widen to transport-agnostic `actions:proposeApproval` with `{kind, title, risk, args, source}`; engine supplies per-action risk | +| Scheduling | capture at modality-queue tier 3 | agent actions at tier 2 alongside chat, so background capture cannot evict the model mid-task | +| Packaging | Developer ID + hardened runtime | apple-events entitlement + usage-description keys | + +Open-core: helper primitives and adapter plumbing in core; the wired agent surface and approvals integration follow the existing pro spine. `pro/` changes land in `desktop-pro` first, submodule bump after. + +## 6. Safety + +The shipped-product template is Gemini Intelligence's UX, which matches our approvals spine: + +- agent acts only on explicit user start; always-visible progress with a hard stop (user input halts execution; the existing abort guard already guarantees a cancelled turn fires no side effects) +- actions classified read / navigate / mutate / irreversible; the last two gate through the approval queue; everything lands in the audit log +- screen content is untrusted input: published studies show 86% attack success from adversarial pop-ups against GUI agents, and prompt-level defenses do not work - the gate and the app allowlist are system-level for that reason +- never see or type credentials: secure-input detection (`IsSecureEventInputEnabled()`) hands password fields to the user + +## 7. Phasing + +| Phase | Scope | Exit test | +| --- | --- | --- | +| 1. Intents + semantic rail | Swift helper (EventKit, Contacts, PhotoKit), AppleScript tools, `shortcuts run`, URL schemes, widened approval hook with risk classes | "create a calendar event Thursday 3pm" end to end, approval-gated, with the bundled model | +| 2. `@offgrid/use` engine | agent graph + action schema + router + verification in shared, tested against a fake `DeviceController` | engine passes its suite with a scripted fake device | +| 3. Desktop adapter + vision fallback | helper act-primitives, tree serialization, window-scoped capture, GUI-Owl/Holo catalog entries, Cortex on the vision model | a multi-step GUI task on a well-behaved app, verified per step | +| 4. Hard targets | Electron AX wake, WhatsApp flow, per-app recipes where trees are dead | the WhatsApp album task, supervised, send behind approval | + +## 8. Open questions + +1. Default Cortex model: GUI-Owl-1.5-8B-Instruct (MIT, best desktop) or Holo3.1 (fastest local)? Both ship as catalog entries either way. +2. Package name: `@offgrid/use` proposed (adapters make it computer use on desktop, phone use later on mobile). +3. How much of minitap's graph do we port in v1 - full eight roles, or start with Planner/Cortex/Executor/Reflector and add Orchestrator/Summarizer when task length demands them? +4. Approval UX for multi-step runs: per-risky-step approval, or plan-level approval with a live step view and hold-to-stop? +5. Do we adopt OSWorld-MCP/macOSWorld as a CI-adjacent eval harness from phase 3, so regressions in the loop are measured rather than felt? + +## 9. Sources + +- minitap/mobile-use: https://github.com/minitap-ai/mobile-use - paper (100% AndroidWorld, ablations): arXiv:2602.07787 +- mobilerun (ex-droidrun): https://github.com/droidrun/mobilerun - tree-vs-screenshot payload data: https://www.mobilerun.ai/benchmark +- Mobile-Agent-v3 / GUI-Owl: https://github.com/X-PLUG/MobileAgent - arXiv:2508.15144, v3.5: arXiv:2602.16855 - GUI-Owl-1.5-8B: https://huggingface.co/mPLUG/GUI-Owl-1.5-8B-Instruct +- Holo3.1: https://huggingface.co/blog/Hcompany/holo31 +- Hybrid-routing benchmarks: MobileWorld https://github.com/Tongyi-MAI/MobileWorld - PhoneHarness https://phoneharness.github.io/ +- Gemini Intelligence safety UX: https://www.engadget.com/2170770/gemini-intelligence-brings-app-automation-to-android/ +- UI-TARS: https://github.com/bytedance/UI-TARS - OSWorld: https://os-world.github.io/ +- Pop-up injection attacks (86% success): arXiv:2411.02391 +- macOS surface: Electron `AXManualAccessibility` https://www.electronjs.org/docs/latest/tutorial/accessibility/ + electron/electron#38102 - secure input TN2150: https://developer.apple.com/library/mac/technotes/tn2150/_index.html - Shortcuts CLI: https://blakecrosley.com/guides/shortcuts - node-mac-permissions: https://github.com/codebytere/node-mac-permissions - WhatsApp Desktop AX findings: https://gist.github.com/hakanensari/99a7ddafbf1b92ce040dc68f43aa25d4 From 3739d6eea128c7b3320e16b5d37bb04538bee250 Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Tue, 11 Aug 2026 16:26:23 +0530 Subject: [PATCH 02/75] docs(computer-use): build plan with dated timeline Five phases, Aug 12 - Dec 2 2026, one demoable checkpoint per phase. Phase 1 (semantic rail) ships standalone value by Sep 9; the vision model install is only needed from phase 3 week 3. Co-Authored-By: Claude Fable 5 --- docs/COMPUTER_USE_PLAN.md | 95 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 docs/COMPUTER_USE_PLAN.md diff --git a/docs/COMPUTER_USE_PLAN.md b/docs/COMPUTER_USE_PLAN.md new file mode 100644 index 00000000..3b5a4f5c --- /dev/null +++ b/docs/COMPUTER_USE_PLAN.md @@ -0,0 +1,95 @@ +# Computer use - build plan and timeline + +Companion to `COMPUTER_USE.md` (the approach). This doc is the execution plan: phases, dated milestones, checkpoints, dependencies, risks. It is the source of truth for schedule - adjust the dates here at the weekly checkpoint, do not fork a second plan. + +**Assumptions** + +- One engineer focused on this track. With a second engineer, phase 2 runs parallel to phase 1 (different repos, no shared files) and the end date pulls in by ~3 weeks. +- Dates start Wednesday, August 12, 2026. +- Checkpoint discipline from the shared roadmap applies: a checkpoint is a verifiable, demoable milestone; the next phase does not start until it passes. +- **Vision model:** GUI-Owl-1.5-8B-Instruct (MIT) is the working default; Qwen3-VL-8B (Apache-2.0) is the alternate. Both are Qwen3-VL-architecture models, so every line of engine and adapter work is identical for either - the pick is a catalog decision made at install time (phase 3, week 3) and is not on the critical path before that. + +## Timeline at a glance + +| Phase | Dates | Length | Checkpoint (demoable) | +| --- | --- | --- | --- | +| 0. Foundations | Aug 12 - Aug 19 | 1 wk | approval seam widened with risk classes; entitlements land; decisions locked | +| 1. Semantic rail | Aug 20 - Sep 9 | 3 wk | "create a calendar event Thursday 3pm" end to end, approval-gated, bundled model | +| 2. `@offgrid/use` engine | Sep 10 - Oct 7 | 4 wk | engine suite green against a scripted fake device | +| 3. Desktop adapter + vision | Oct 8 - Nov 4 | 4 wk | multi-step GUI task on a native app, verified per step, live step view | +| 4. Hard targets + hardening | Nov 5 - Dec 2 | 4 wk | WhatsApp album task supervised end to end; safety checklist passes | + +Phase 1 is independently shippable - calendar, messages, mail, photos, reminders actions with approvals deliver user value on Sep 9 regardless of what follows. + +## Phase 0 - foundations (Aug 12 - Aug 19) + +| Work item | Detail | Verification | +| --- | --- | --- | +| Lock decisions | v1 graph roles (proposed: Planner / Cortex / Executor / Reflector, add Orchestrator + Summarizer when task length demands), package name (`@offgrid/use`), approval UX (proposed: plan-level approval + live step view + per-step gates on irreversible actions) | recorded in `COMPUTER_USE.md` section 8 | +| Widen the approval seam | `mcp:proposeApproval` becomes transport-agnostic `actions:proposeApproval` carrying `{kind, title, detail, risk, args, source}`; per-executor `riskOf(name, args)` replaces the name regex; MCP extension migrates onto it | existing approval tests stay green (`mcp-connector-tool-extension.dbtest.ts`); new tests per risk class; pro executor updated in `desktop-pro`, submodule bumped | +| Packaging groundwork | `com.apple.security.automation.apple-events` entitlement + `NSAppleEventsUsageDescription` + Calendars/Contacts/Photos usage keys in the electron-builder config | local packaged build per `local-build.local.md`; TCC prompts name the app | +| Permission onboarding design | sequence of grants (Accessibility held check exists, Automation per target, per-framework) behind an explicit "enable computer actions" flow | design reviewed; no dead-end states | + +## Phase 1 - semantic rail (Aug 20 - Sep 9) + +- **Week of Aug 20:** `actions` Swift helper skeleton (JSON over stdio, same `execFile` pattern as `src/main/ocr.ts`); EventKit create/read events + reminders; Contacts read. Unit tests on the Node wrapper; helper built alongside the existing Swift helpers. +- **Week of Aug 27:** AppleScript surfaces - Messages send, Mail compose, Notes create/search, Finder basics; URL-scheme opener; `open -b` app launcher; `shortcuts run` wrapper with timeout guards (a prompting shortcut hangs the CLI - wrap every call). +- **Week of Sep 3:** expose everything as tools through the `ToolExtension` seam with risk classes; approval-gated writes; audit entries; permission onboarding wired. Tests: unit per risk class and arg mapping, integration on a temp profile, e2e smoke asserting the approval queue renders the new action kinds. + +**Checkpoint (Sep 9):** calendar event created end to end from chat with the bundled model, gated, audited. Messages send queues for approval and executes on approve. + +## Phase 2 - `@offgrid/use` engine in shared (Sep 10 - Oct 7) + +All work in `off-grid-ai/shared`; testable with zero macOS dependencies. + +- **Week of Sep 10:** package scaffold; `DeviceController` interface (`getState()`, `act(action)`, `openIntent(url)`, `listSemanticSurfaces()`); closed action schema + structured decision types; fake `DeviceController` with scripted scenarios. +- **Week of Sep 17:** graph v1 - Planner (subgoal decomposition), Cortex (one structured decision per step from tree + screenshot), Executor (deterministic parse, no free-form reasoning). +- **Week of Sep 24:** Reflector (SUCCESS/FAILURE per transition with diagnosis), deterministic post-validation for typing (act, re-read, diff - minitap's +7 pt mechanism), cycle detection with forced strategy change (+9 pt mechanism); the router (deterministic surface - structured UI - vision). +- **Week of Oct 1:** per-role model config against an OpenAI-compatible endpoint (our gateway); risk + approval callback seam; suite hardening; package docs. + +**Checkpoint (Oct 7):** suite green on the fake device, including a type-verify scenario (field readback mismatch triggers retry) and a loop-escape scenario (repeated state triggers strategy change). + +## Phase 3 - desktop adapter + vision fallback (Oct 8 - Nov 4) + +- **Week of Oct 8:** Swift helper act primitives - `AXUIElementPerformAction`, set-`AXValue`, `AXManualAccessibility` wake, `CGEvent` post; coordinate mapping (AX points vs screenshot pixels, `backingScaleFactor` per display, multi-display origins). +- **Week of Oct 15:** AX tree serialization (indexed interactive elements + bounding boxes, drop unlabeled wrappers, diffs after actions); window-scoped capture via `src/main/vision.ts`; OCR helper emits `{text, bbox}`. +- **Week of Oct 22:** model catalog entries (GUI-Owl-1.5-8B-Instruct GGUF + mmproj; Qwen3-VL-8B alternate) - **model install happens here**; Cortex/Reflector wired through the gateway; agent runs at modality-queue tier 2. +- **Week of Oct 29:** supervised run UI - live step view, hold-to-stop, per-step approvals for irreversible actions; e2e with screenshots; small eval harness (macOSWorld subset) as a sanity gate, not CI-blocking. + +**Checkpoint (Nov 4):** a multi-step GUI task on a well-behaved native app (for example: create and reschedule a reminder through the Reminders UI) completes with per-step verification visible in the run view. + +## Phase 4 - hard targets + hardening (Nov 5 - Dec 2) + +- **Week of Nov 5:** Electron AX wake flows; secure-input detection surfaced in System Health; app allowlist setting. +- **Weeks of Nov 12 + 19:** the WhatsApp acceptance case - `whatsapp://` intent open, keyboard-first navigation, vision-grounded clicks for the gaps, photo ranking by the vision model, send behind the gate. Per-app recipe container (skills format) so the flow is data, not code. +- **Week of Nov 26:** injection-resistance review against the screen-content threat model; kill-switch e2e; release-readiness pass (docs, System Health states, permission recovery). + +**Checkpoint (Dec 2):** WhatsApp album task runs supervised end to end on a demo profile; the safety checklist (gates, allowlist, secure input, kill switch, audit) passes. + +## Dependencies + +| What | Needed by | Owner | +| --- | --- | --- | +| Vision model install (GUI-Owl-1.5-8B or Qwen3-VL-8B GGUF + mmproj) | phase 3, week of Oct 22 | Siddharth - nothing earlier depends on it | +| `off-grid-ai/shared` CI for the new package | phase 2 start | team | +| Dev-machine TCC grants (Accessibility, Automation) with a stable dev signing identity - grants key to code signature | phase 1 and 3 | each dev, per `local-build.local.md` | +| `desktop-pro` changes for the approval executor | phase 0 | same engineer, separate repo commits + submodule bump | + +## Risks + +| Risk | Mitigation | +| --- | --- | +| macOS AX trees are uneven, vision carries more steps than on mobile | GUI-Owl is desktop-trained; window-scoped capture keeps resolution sane; recipes for the worst apps | +| Desktop agent scores run 20-30 pts below mobile headlines | supervised UX with per-step verify is the product shape, not autonomy claims | +| App updates break GUI flows (WhatsApp foremost) | flows live in recipe data; the acceptance case is a demo target, not a launch gate | +| TCC dev trap: grants orphan when the signing identity changes | stable dev identity, `tccutil reset` runbook note | +| Model landscape shifts before phase 3 | per-role model config; swapping the Cortex model touches zero engine code | + +## Out of scope for v1 + +Windows adapter, mobile adapter, teach-and-repeat recording, background/headless runs, Mac App Store distribution (sandbox blocks the Accessibility path - we ship Developer ID). + +## Tracking + +- Branch: `feat/computer-use`. Small commits per verified unit, merge not squash. PR evidence rules apply (screenshots per changed surface; video for the run-view and WhatsApp demos). +- Weekly checkpoint review against this doc; date changes are edits to this doc. From 562f2ee424ce84f30c7a3fa83eeba51ed4459708 Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Tue, 11 Aug 2026 16:33:31 +0530 Subject: [PATCH 03/75] docs(computer-use): compress timeline for solo AI-assisted development 16 weeks to 9 (Aug 12 - Oct 14). Code-heavy phases shrink the most (engine 4wk to 2wk); integration-heavy phases keep slack since review, TCC flows, and real-app iteration do not compress. Co-Authored-By: Claude Fable 5 --- docs/COMPUTER_USE_PLAN.md | 81 +++++++++++++++++++++------------------ 1 file changed, 43 insertions(+), 38 deletions(-) diff --git a/docs/COMPUTER_USE_PLAN.md b/docs/COMPUTER_USE_PLAN.md index 3b5a4f5c..06250606 100644 --- a/docs/COMPUTER_USE_PLAN.md +++ b/docs/COMPUTER_USE_PLAN.md @@ -4,24 +4,24 @@ Companion to `COMPUTER_USE.md` (the approach). This doc is the execution plan: p **Assumptions** -- One engineer focused on this track. With a second engineer, phase 2 runs parallel to phase 1 (different repos, no shared files) and the end date pulls in by ~3 weeks. -- Dates start Wednesday, August 12, 2026. +- Solo developer (Siddharth), with AI doing the code authoring end to end. Durations are therefore paced by what does NOT compress: review, on-device verification, TCC permission flows, packaged-build checks, and iteration against real apps. Code-heavy phases are scheduled aggressively; integration-heavy phases keep slack. +- Dates start Wednesday, August 12, 2026 and assume this is the main focus. If a week goes elsewhere, shift the dates here - the phase order and checkpoints do not change. - Checkpoint discipline from the shared roadmap applies: a checkpoint is a verifiable, demoable milestone; the next phase does not start until it passes. -- **Vision model:** GUI-Owl-1.5-8B-Instruct (MIT) is the working default; Qwen3-VL-8B (Apache-2.0) is the alternate. Both are Qwen3-VL-architecture models, so every line of engine and adapter work is identical for either - the pick is a catalog decision made at install time (phase 3, week 3) and is not on the critical path before that. +- **Vision model:** GUI-Owl-1.5-8B-Instruct (MIT) is the working default; Qwen3-VL-8B (Apache-2.0) is the alternate. Both are Qwen3-VL-architecture models, so every line of engine and adapter work is identical for either - the pick is a catalog decision made at install time (phase 3) and is not on the critical path before that. ## Timeline at a glance | Phase | Dates | Length | Checkpoint (demoable) | | --- | --- | --- | --- | -| 0. Foundations | Aug 12 - Aug 19 | 1 wk | approval seam widened with risk classes; entitlements land; decisions locked | -| 1. Semantic rail | Aug 20 - Sep 9 | 3 wk | "create a calendar event Thursday 3pm" end to end, approval-gated, bundled model | -| 2. `@offgrid/use` engine | Sep 10 - Oct 7 | 4 wk | engine suite green against a scripted fake device | -| 3. Desktop adapter + vision | Oct 8 - Nov 4 | 4 wk | multi-step GUI task on a native app, verified per step, live step view | -| 4. Hard targets + hardening | Nov 5 - Dec 2 | 4 wk | WhatsApp album task supervised end to end; safety checklist passes | +| 0. Foundations | Aug 12 - Aug 14 | 3 days | approval seam widened with risk classes; entitlements land; decisions locked | +| 1. Semantic rail | Aug 17 - Aug 26 | 1.5 wk | "create a calendar event Thursday 3pm" end to end, approval-gated, bundled model | +| 2. `@offgrid/use` engine | Aug 27 - Sep 9 | 2 wk | engine suite green against a scripted fake device | +| 3. Desktop adapter + vision | Sep 10 - Sep 29 | 3 wk | multi-step GUI task on a native app, verified per step, live step view | +| 4. Hard targets + hardening | Sep 30 - Oct 14 | 2 wk | WhatsApp album task supervised end to end; safety checklist passes | -Phase 1 is independently shippable - calendar, messages, mail, photos, reminders actions with approvals deliver user value on Sep 9 regardless of what follows. +Nine weeks total. Phase 1 is independently shippable - calendar, messages, mail, photos, reminders actions with approvals deliver user value on Aug 26 regardless of what follows. Where the compression came from: the engine phase is pure TypeScript against a fake device (ideal AI-authoring territory, 4 wk to 2 wk); the semantic rail is many small similar wrappers (3 wk to 1.5 wk). Phase 3 keeps the most slack because it mixes Swift interop, a model install, and real-device iteration; phase 4 is paced by WhatsApp itself, not by code. -## Phase 0 - foundations (Aug 12 - Aug 19) +## Phase 0 - foundations (Aug 12 - Aug 14) | Work item | Detail | Verification | | --- | --- | --- | @@ -30,55 +30,60 @@ Phase 1 is independently shippable - calendar, messages, mail, photos, reminders | Packaging groundwork | `com.apple.security.automation.apple-events` entitlement + `NSAppleEventsUsageDescription` + Calendars/Contacts/Photos usage keys in the electron-builder config | local packaged build per `local-build.local.md`; TCC prompts name the app | | Permission onboarding design | sequence of grants (Accessibility held check exists, Automation per target, per-framework) behind an explicit "enable computer actions" flow | design reviewed; no dead-end states | -## Phase 1 - semantic rail (Aug 20 - Sep 9) +## Phase 1 - semantic rail (Aug 17 - Aug 26) -- **Week of Aug 20:** `actions` Swift helper skeleton (JSON over stdio, same `execFile` pattern as `src/main/ocr.ts`); EventKit create/read events + reminders; Contacts read. Unit tests on the Node wrapper; helper built alongside the existing Swift helpers. -- **Week of Aug 27:** AppleScript surfaces - Messages send, Mail compose, Notes create/search, Finder basics; URL-scheme opener; `open -b` app launcher; `shortcuts run` wrapper with timeout guards (a prompting shortcut hangs the CLI - wrap every call). -- **Week of Sep 3:** expose everything as tools through the `ToolExtension` seam with risk classes; approval-gated writes; audit entries; permission onboarding wired. Tests: unit per risk class and arg mapping, integration on a temp profile, e2e smoke asserting the approval queue renders the new action kinds. +- **Aug 17 - 19:** `actions` Swift helper skeleton (JSON over stdio, same `execFile` pattern as `src/main/ocr.ts`); EventKit create/read events + reminders; Contacts read. Unit tests on the Node wrapper; helper built alongside the existing Swift helpers. +- **Aug 20 - 21:** AppleScript surfaces - Messages send, Mail compose, Notes create/search, Finder basics; URL-scheme opener; `open -b` app launcher; `shortcuts run` wrapper with timeout guards (a prompting shortcut hangs the CLI - wrap every call). +- **Aug 24 - 26:** expose everything as tools through the `ToolExtension` seam with risk classes; approval-gated writes; audit entries; permission onboarding wired. Tests: unit per risk class and arg mapping, integration on a temp profile, e2e smoke asserting the approval queue renders the new action kinds. -**Checkpoint (Sep 9):** calendar event created end to end from chat with the bundled model, gated, audited. Messages send queues for approval and executes on approve. +**Checkpoint (Aug 26):** calendar event created end to end from chat with the bundled model, gated, audited. Messages send queues for approval and executes on approve. -## Phase 2 - `@offgrid/use` engine in shared (Sep 10 - Oct 7) +## Phase 2 - `@offgrid/use` engine in shared (Aug 27 - Sep 9) -All work in `off-grid-ai/shared`; testable with zero macOS dependencies. +All work in `off-grid-ai/shared`; testable with zero macOS dependencies - the phase where AI authoring compresses the most. -- **Week of Sep 10:** package scaffold; `DeviceController` interface (`getState()`, `act(action)`, `openIntent(url)`, `listSemanticSurfaces()`); closed action schema + structured decision types; fake `DeviceController` with scripted scenarios. -- **Week of Sep 17:** graph v1 - Planner (subgoal decomposition), Cortex (one structured decision per step from tree + screenshot), Executor (deterministic parse, no free-form reasoning). -- **Week of Sep 24:** Reflector (SUCCESS/FAILURE per transition with diagnosis), deterministic post-validation for typing (act, re-read, diff - minitap's +7 pt mechanism), cycle detection with forced strategy change (+9 pt mechanism); the router (deterministic surface - structured UI - vision). -- **Week of Oct 1:** per-role model config against an OpenAI-compatible endpoint (our gateway); risk + approval callback seam; suite hardening; package docs. +- **Aug 27 - 31:** package scaffold; `DeviceController` interface (`getState()`, `act(action)`, `openIntent(url)`, `listSemanticSurfaces()`); closed action schema + structured decision types; fake `DeviceController` with scripted scenarios. +- **Sep 1 - 4:** graph v1 - Planner (subgoal decomposition), Cortex (one structured decision per step from tree + screenshot), Executor (deterministic parse, no free-form reasoning); the router (deterministic surface - structured UI - vision). +- **Sep 7 - 9:** Reflector (SUCCESS/FAILURE per transition with diagnosis), deterministic post-validation for typing (act, re-read, diff - minitap's +7 pt mechanism), cycle detection with forced strategy change (+9 pt mechanism); per-role model config against an OpenAI-compatible endpoint (our gateway); risk + approval callback seam; suite hardening; package docs. -**Checkpoint (Oct 7):** suite green on the fake device, including a type-verify scenario (field readback mismatch triggers retry) and a loop-escape scenario (repeated state triggers strategy change). +**Checkpoint (Sep 9):** suite green on the fake device, including a type-verify scenario (field readback mismatch triggers retry) and a loop-escape scenario (repeated state triggers strategy change). -## Phase 3 - desktop adapter + vision fallback (Oct 8 - Nov 4) +## Phase 3 - desktop adapter + vision fallback (Sep 10 - Sep 29) -- **Week of Oct 8:** Swift helper act primitives - `AXUIElementPerformAction`, set-`AXValue`, `AXManualAccessibility` wake, `CGEvent` post; coordinate mapping (AX points vs screenshot pixels, `backingScaleFactor` per display, multi-display origins). -- **Week of Oct 15:** AX tree serialization (indexed interactive elements + bounding boxes, drop unlabeled wrappers, diffs after actions); window-scoped capture via `src/main/vision.ts`; OCR helper emits `{text, bbox}`. -- **Week of Oct 22:** model catalog entries (GUI-Owl-1.5-8B-Instruct GGUF + mmproj; Qwen3-VL-8B alternate) - **model install happens here**; Cortex/Reflector wired through the gateway; agent runs at modality-queue tier 2. -- **Week of Oct 29:** supervised run UI - live step view, hold-to-stop, per-step approvals for irreversible actions; e2e with screenshots; small eval harness (macOSWorld subset) as a sanity gate, not CI-blocking. +Keeps the most slack: Swift interop, a model install, and real-device iteration all live here. -**Checkpoint (Nov 4):** a multi-step GUI task on a well-behaved native app (for example: create and reschedule a reminder through the Reminders UI) completes with per-step verification visible in the run view. +- **Sep 10 - 15:** Swift helper act primitives - `AXUIElementPerformAction`, set-`AXValue`, `AXManualAccessibility` wake, `CGEvent` post; coordinate mapping (AX points vs screenshot pixels, `backingScaleFactor` per display, multi-display origins). +- **Sep 16 - 21:** AX tree serialization (indexed interactive elements + bounding boxes, drop unlabeled wrappers, diffs after actions); window-scoped capture via `src/main/vision.ts`; OCR helper emits `{text, bbox}`. +- **Sep 22 - 24:** model catalog entries (GUI-Owl-1.5-8B-Instruct GGUF + mmproj; Qwen3-VL-8B alternate) - **model install happens here**; Cortex/Reflector wired through the gateway; agent runs at modality-queue tier 2. +- **Sep 25 - 29:** supervised run UI - live step view, hold-to-stop, per-step approvals for irreversible actions; e2e with screenshots; small eval harness (macOSWorld subset) as a sanity gate, not CI-blocking. -## Phase 4 - hard targets + hardening (Nov 5 - Dec 2) +**Checkpoint (Sep 29):** a multi-step GUI task on a well-behaved native app (for example: create and reschedule a reminder through the Reminders UI) completes with per-step verification visible in the run view. -- **Week of Nov 5:** Electron AX wake flows; secure-input detection surfaced in System Health; app allowlist setting. -- **Weeks of Nov 12 + 19:** the WhatsApp acceptance case - `whatsapp://` intent open, keyboard-first navigation, vision-grounded clicks for the gaps, photo ranking by the vision model, send behind the gate. Per-app recipe container (skills format) so the flow is data, not code. -- **Week of Nov 26:** injection-resistance review against the screen-content threat model; kill-switch e2e; release-readiness pass (docs, System Health states, permission recovery). +## Phase 4 - hard targets + hardening (Sep 30 - Oct 14) -**Checkpoint (Dec 2):** WhatsApp album task runs supervised end to end on a demo profile; the safety checklist (gates, allowlist, secure input, kill switch, audit) passes. +Paced by iteration against real apps, not by code volume. + +- **Sep 30 - Oct 2:** Electron AX wake flows; secure-input detection surfaced in System Health; app allowlist setting. +- **Oct 5 - 9:** the WhatsApp acceptance case - `whatsapp://` intent open, keyboard-first navigation, vision-grounded clicks for the gaps, photo ranking by the vision model, send behind the gate. Per-app recipe container (skills format) so the flow is data, not code. +- **Oct 12 - 14:** injection-resistance review against the screen-content threat model; kill-switch e2e; release-readiness pass (docs, System Health states, permission recovery). + +**Checkpoint (Oct 14):** WhatsApp album task runs supervised end to end on a demo profile; the safety checklist (gates, allowlist, secure input, kill switch, audit) passes. ## Dependencies -| What | Needed by | Owner | +| What | Needed by | Note | | --- | --- | --- | -| Vision model install (GUI-Owl-1.5-8B or Qwen3-VL-8B GGUF + mmproj) | phase 3, week of Oct 22 | Siddharth - nothing earlier depends on it | -| `off-grid-ai/shared` CI for the new package | phase 2 start | team | -| Dev-machine TCC grants (Accessibility, Automation) with a stable dev signing identity - grants key to code signature | phase 1 and 3 | each dev, per `local-build.local.md` | -| `desktop-pro` changes for the approval executor | phase 0 | same engineer, separate repo commits + submodule bump | +| Vision model install (GUI-Owl-1.5-8B or Qwen3-VL-8B GGUF + mmproj) | Sep 22 | nothing earlier depends on it | +| `off-grid-ai/shared` CI for the new package | Aug 27 | one-time setup, ~half a day | +| Dev-machine TCC grants (Accessibility, Automation) with a stable dev signing identity - grants key to code signature | Aug 17 and Sep 10 | per `local-build.local.md`; keep one identity or grants orphan | +| `desktop-pro` changes for the approval executor | Aug 12 - 14 | separate repo commits + submodule bump here | ## Risks | Risk | Mitigation | | --- | --- | +| Solo schedule: one blocked day is a lost day, no parallel track | phases are independently valuable; a slip moves dates in this doc, never skips a checkpoint | +| AI-authored code lands faster than it is verified | the pace is set by the checkpoints, not by lines written: nothing is "done" until its tests and on-device check pass (repo rule: commit only green units) | | macOS AX trees are uneven, vision carries more steps than on mobile | GUI-Owl is desktop-trained; window-scoped capture keeps resolution sane; recipes for the worst apps | | Desktop agent scores run 20-30 pts below mobile headlines | supervised UX with per-step verify is the product shape, not autonomy claims | | App updates break GUI flows (WhatsApp foremost) | flows live in recipe data; the acceptance case is a demo target, not a launch gate | From a9ac675ab35e4eb168ab6b74193bdb051ec9e1fc Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Tue, 11 Aug 2026 17:17:15 +0530 Subject: [PATCH 04/75] docs(computer-use): phase 0 shared-repo setup + mobile sequencing note Sibling shared checkout + file: consumption decision added to phase 0; mobile follows desktop as an adapter-only project on the same engine. Co-Authored-By: Claude Fable 5 --- docs/COMPUTER_USE_PLAN.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/COMPUTER_USE_PLAN.md b/docs/COMPUTER_USE_PLAN.md index 06250606..1b0d6daf 100644 --- a/docs/COMPUTER_USE_PLAN.md +++ b/docs/COMPUTER_USE_PLAN.md @@ -29,6 +29,7 @@ Nine weeks total. Phase 1 is independently shippable - calendar, messages, mail, | Widen the approval seam | `mcp:proposeApproval` becomes transport-agnostic `actions:proposeApproval` carrying `{kind, title, detail, risk, args, source}`; per-executor `riskOf(name, args)` replaces the name regex; MCP extension migrates onto it | existing approval tests stay green (`mcp-connector-tool-extension.dbtest.ts`); new tests per risk class; pro executor updated in `desktop-pro`, submodule bumped | | Packaging groundwork | `com.apple.security.automation.apple-events` entitlement + `NSAppleEventsUsageDescription` + Calendars/Contacts/Photos usage keys in the electron-builder config | local packaged build per `local-build.local.md`; TCC prompts name the app | | Permission onboarding design | sequence of grants (Accessibility held check exists, Automation per target, per-framework) behind an explicit "enable computer actions" flow | design reviewed; no dead-end states | +| Shared repo setup | sibling clone of `off-grid-ai/shared` next to this repo (CLAUDE.md already assumes `../shared`); decide the consumption pattern for `@offgrid/use` - proposed: the README's `file:../shared/packages/use`, NOT a fifth vendored copy under `packages/` (confirm with the lead; makes a sibling checkout a build requirement on this branch) | checkout builds; decision recorded here | ## Phase 1 - semantic rail (Aug 17 - Aug 26) @@ -94,6 +95,8 @@ Paced by iteration against real apps, not by code volume. Windows adapter, mobile adapter, teach-and-repeat recording, background/headless runs, Mac App Store distribution (sandbox blocks the Accessibility path - we ship Developer ID). +**Mobile sequencing:** desktop ships first; mobile follows as an adapter-only project on the same `@offgrid/use` engine (Android: accessibility-service portal + intents; iOS: intents-only - the platform does not allow an app to read or drive other apps). The engine phase already does all the mobile prep that matters: a platform-free `DeviceController` seam, an action schema aligned with the mobile-use frameworks, and a suite that runs on a fake device. OGAM also does not consume the shared monorepo yet, so a mobile adapter has that migration as a prerequisite regardless of when we start it. + ## Tracking - Branch: `feat/computer-use`. Small commits per verified unit, merge not squash. PR evidence rules apply (screenshots per changed surface; video for the run-view and WhatsApp demos). From 34d0290b06adaa0a170cbde57ecb4e1d4b50ed49 Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Tue, 11 Aug 2026 17:20:53 +0530 Subject: [PATCH 05/75] docs(computer-use): record decision status; break out the after-v1 mobile adapter Approach doc's open questions become a decision log (graph roles, eval harness, sequencing decided; model pick narrowed to install time; package name pending lead confirm). Plan gets an After v1 section with the mobile adapter's prerequisites. Co-Authored-By: Claude Fable 5 --- docs/COMPUTER_USE.md | 15 +++++++++------ docs/COMPUTER_USE_PLAN.md | 10 ++++++++-- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/docs/COMPUTER_USE.md b/docs/COMPUTER_USE.md index c60fa560..c1ae5996 100644 --- a/docs/COMPUTER_USE.md +++ b/docs/COMPUTER_USE.md @@ -125,13 +125,16 @@ The shipped-product template is Gemini Intelligence's UX, which matches our appr | 3. Desktop adapter + vision fallback | helper act-primitives, tree serialization, window-scoped capture, GUI-Owl/Holo catalog entries, Cortex on the vision model | a multi-step GUI task on a well-behaved app, verified per step | | 4. Hard targets | Electron AX wake, WhatsApp flow, per-app recipes where trees are dead | the WhatsApp album task, supervised, send behind approval | -## 8. Open questions +## 8. Decisions and open questions -1. Default Cortex model: GUI-Owl-1.5-8B-Instruct (MIT, best desktop) or Holo3.1 (fastest local)? Both ship as catalog entries either way. -2. Package name: `@offgrid/use` proposed (adapters make it computer use on desktop, phone use later on mobile). -3. How much of minitap's graph do we port in v1 - full eight roles, or start with Planner/Cortex/Executor/Reflector and add Orchestrator/Summarizer when task length demands them? -4. Approval UX for multi-step runs: per-risky-step approval, or plan-level approval with a live step view and hold-to-stop? -5. Do we adopt OSWorld-MCP/macOSWorld as a CI-adjacent eval harness from phase 3, so regressions in the loop are measured rather than felt? +Status as of August 11, 2026 (details live in `COMPUTER_USE_PLAN.md`): + +1. **Cortex model - narrowed, decided at install time.** GUI-Owl-1.5-8B-Instruct (working default) or Qwen3-VL-8B. Same Qwen3-VL architecture, so all engine and adapter work is identical for either; the pick is a phase 3 catalog decision and blocks nothing before that. +2. **Package name - proposed, pending lead confirm.** `@offgrid/use`, consumed as `file:../shared/packages/use` from a sibling checkout per the shared README (not a vendored copy under this repo's `packages/`). Confirm both with the lead in phase 0. +3. **v1 graph - decided.** Planner / Cortex / Executor / Reflector; Orchestrator and Summarizer are added when task length demands them. +4. **Approval UX - proposed.** Plan-level approval + live step view + hold-to-stop, with per-step gates on irreversible actions. Validate against real runs in phase 3. +5. **Eval harness - decided.** macOSWorld subset as a non-blocking sanity gate from phase 3. +6. **Sequencing - decided.** Desktop first; mobile follows as an adapter-only project on the same engine (see the plan's "After v1" section; iOS is intents-only by platform rules). ## 9. Sources diff --git a/docs/COMPUTER_USE_PLAN.md b/docs/COMPUTER_USE_PLAN.md index 1b0d6daf..0384654f 100644 --- a/docs/COMPUTER_USE_PLAN.md +++ b/docs/COMPUTER_USE_PLAN.md @@ -93,9 +93,15 @@ Paced by iteration against real apps, not by code volume. ## Out of scope for v1 -Windows adapter, mobile adapter, teach-and-repeat recording, background/headless runs, Mac App Store distribution (sandbox blocks the Accessibility path - we ship Developer ID). +Windows adapter, teach-and-repeat recording, background/headless runs, Mac App Store distribution (sandbox blocks the Accessibility path - we ship Developer ID). -**Mobile sequencing:** desktop ships first; mobile follows as an adapter-only project on the same `@offgrid/use` engine (Android: accessibility-service portal + intents; iOS: intents-only - the platform does not allow an app to read or drive other apps). The engine phase already does all the mobile prep that matters: a platform-free `DeviceController` seam, an action schema aligned with the mobile-use frameworks, and a suite that runs on a fake device. OGAM also does not consume the shared monorepo yet, so a mobile adapter has that migration as a prerequisite regardless of when we start it. +## After v1 - the mobile adapter (unscheduled) + +Desktop ships first; mobile follows as an adapter-only project on the same `@offgrid/use` engine. The engine phase already does the mobile prep that matters - a platform-free `DeviceController` seam, an action schema aligned with the mobile-use frameworks, a suite that runs on a fake device - and the vision model family (GUI-Owl-1.5) is trained for mobile as well as desktop. Prerequisites before scheduling it: + +1. OGAM adopts the shared monorepo - it does not consume `@offgrid/*` packages yet; that migration is a standing roadmap item and a prerequisite regardless of computer use. +2. Android portal: an accessibility-service app (the droidrun-portal pattern) for tree reading + input, MediaProjection for screens, Android intents/deep links as the semantic rail. +3. iOS stays intents-only (App Intents / Shortcuts) - the platform does not allow an app to read or drive other apps, so no vision fallback exists there. Set expectations accordingly. ## Tracking From 259095cd79786e69872b2dd277c9be414979afe3 Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Tue, 11 Aug 2026 17:39:16 +0530 Subject: [PATCH 06/75] docs(computer-use): per-phase repo column + write-access dependency Timeline table now shows which repo each phase builds in; dependencies record the OGAD push-access ask (pull-only as of Aug 11, shared has push). Co-Authored-By: Claude Fable 5 --- docs/COMPUTER_USE_PLAN.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/COMPUTER_USE_PLAN.md b/docs/COMPUTER_USE_PLAN.md index 0384654f..13d0291e 100644 --- a/docs/COMPUTER_USE_PLAN.md +++ b/docs/COMPUTER_USE_PLAN.md @@ -11,13 +11,15 @@ Companion to `COMPUTER_USE.md` (the approach). This doc is the execution plan: p ## Timeline at a glance -| Phase | Dates | Length | Checkpoint (demoable) | -| --- | --- | --- | --- | -| 0. Foundations | Aug 12 - Aug 14 | 3 days | approval seam widened with risk classes; entitlements land; decisions locked | -| 1. Semantic rail | Aug 17 - Aug 26 | 1.5 wk | "create a calendar event Thursday 3pm" end to end, approval-gated, bundled model | -| 2. `@offgrid/use` engine | Aug 27 - Sep 9 | 2 wk | engine suite green against a scripted fake device | -| 3. Desktop adapter + vision | Sep 10 - Sep 29 | 3 wk | multi-step GUI task on a native app, verified per step, live step view | -| 4. Hard targets + hardening | Sep 30 - Oct 14 | 2 wk | WhatsApp album task supervised end to end; safety checklist passes | +| Phase | Dates | Length | Repo | Checkpoint (demoable) | +| --- | --- | --- | --- | --- | +| 0. Foundations | Aug 12 - Aug 14 | 3 days | this repo + `desktop-pro` | approval seam widened with risk classes; entitlements land; decisions locked | +| 1. Semantic rail | Aug 17 - Aug 26 | 1.5 wk | this repo | "create a calendar event Thursday 3pm" end to end, approval-gated, bundled model | +| 2. `@offgrid/use` engine | Aug 27 - Sep 9 | 2 wk | `shared` | engine suite green against a scripted fake device | +| 3. Desktop adapter + vision | Sep 10 - Sep 29 | 3 wk | this repo + `desktop-pro` | multi-step GUI task on a native app, verified per step, live step view | +| 4. Hard targets + hardening | Sep 30 - Oct 14 | 2 wk | this repo + `desktop-pro` | WhatsApp album task supervised end to end; safety checklist passes | + +The split to remember: `shared` holds the durable, cross-platform brain (2 of the 9 weeks, reused later by mobile); this repo holds the hands and the product integration (everything else). Nine weeks total. Phase 1 is independently shippable - calendar, messages, mail, photos, reminders actions with approvals deliver user value on Aug 26 regardless of what follows. Where the compression came from: the engine phase is pure TypeScript against a fake device (ideal AI-authoring territory, 4 wk to 2 wk); the semantic rail is many small similar wrappers (3 wk to 1.5 wk). Phase 3 keeps the most slack because it mixes Swift interop, a model install, and real-device iteration; phase 4 is paced by WhatsApp itself, not by code. @@ -78,6 +80,7 @@ Paced by iteration against real apps, not by code volume. | `off-grid-ai/shared` CI for the new package | Aug 27 | one-time setup, ~half a day | | Dev-machine TCC grants (Accessibility, Automation) with a stable dev signing identity - grants key to code signature | Aug 17 and Sep 10 | per `local-build.local.md`; keep one identity or grants orphan | | `desktop-pro` changes for the approval executor | Aug 12 - 14 | separate repo commits + submodule bump here | +| Push access to `off-grid-ai/OGAD` (and `desktop-pro` if not held) | before the first push - local commits proceed without it | as of Aug 11 the account is pull-only on OGAD; `shared` already has push. Ask the lead | ## Risks From eb96d1c651f45ade69bb795e006885f386dac16f Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Wed, 12 Aug 2026 12:32:29 +0530 Subject: [PATCH 07/75] docs(computer-use): agent-browser rail, reuse list, brand guidelines, re-phased timeline From the Clawbot/product-UX research: embedded agent browser (WebContentsView + webContents.debugger CDP, indexed snapshot, per-site cards, takeover with capture-kill) becomes phase 3, ahead of native GUI - zero OS permissions, no new models, shippable cut at Sep 23. Native adapter + vision moves to phase 4, hard targets to phase 5 (ends Oct 27). Direct-reuse list added (UI-TARS sdk/ScreenMarker, nanobrowser, nut-js fork, macos-automator-mcp, bytebot takeover pattern, Peekaboo). Brand guidelines (off-grid-ai/brand + @offgrid/design tokens + docs/DESIGN.md) bound as standing build rules. OpenClaw teardown informs the zero-setup bar and the safety avoid-list. Co-Authored-By: Claude Fable 5 --- docs/COMPUTER_USE.md | 60 +++++++++++++++++++++---- docs/COMPUTER_USE_PLAN.md | 93 ++++++++++++++++++++++++--------------- 2 files changed, 110 insertions(+), 43 deletions(-) diff --git a/docs/COMPUTER_USE.md b/docs/COMPUTER_USE.md index c1ae5996..daf48556 100644 --- a/docs/COMPUTER_USE.md +++ b/docs/COMPUTER_USE.md @@ -1,6 +1,6 @@ # Computer use - replicate the mobile-use stack on desktop -**Status:** direction decided August 11, 2026. Intents + MCP are the primary action paths; the vision-based agent is the fallback. We do not innovate on agent architecture - we study the mobile-use ecosystem and replicate it. The engine is built in the shared repo (`off-grid-ai/shared`) as an `@offgrid/*` package. Model size is not a design constraint - local models keep improving and the vision model ships as a downloadable catalog entry. +**Status:** direction decided August 11, 2026; agent-browser rail and build guidelines added August 12 after the Clawbot / product-UX research. Intents + MCP are the primary action paths; an embedded agent browser handles web tasks; the vision-based agent is the fallback for native apps. We do not innovate on agent architecture - we study the mobile-use ecosystem and replicate it, with the browser UX replicated from Codex desktop and Claude Desktop. The engine is built in the shared repo (`off-grid-ai/shared`) as an `@offgrid/*` package. Model size is not a design constraint - local models keep improving and the vision model ships as a downloadable catalog entry. **Constraint (standing):** local models only. No hosted APIs. No screenshot ever leaves the device. --- @@ -10,9 +10,11 @@ The agent executes actions on the user's Mac - from "create a calendar event Thursday 3pm" to "pick the best photo from the vacation album in the family WhatsApp chat and send it". The decided shape: 1. **Intents + MCP first.** Deterministic action surfaces - MCP connectors, URL schemes, AppleScript/Apple Events, EventKit and friends, Shortcuts - handle everything they can. No pixels, no coordinates. -2. **Vision-based agent as fallback.** When no deterministic surface exists, a multi-agent GUI loop takes over, perceiving through the accessibility tree plus screenshots and acting through synthetic input. -3. **Replicate, do not invent.** The mobile-use ecosystem solved this problem in 2025-26 with measured results (100% on AndroidWorld). We port its architecture to macOS. -4. **Engine in shared.** The agent loop, action schema, router, and verification logic are platform-agnostic and land as a package in `off-grid-ai/shared`, with a desktop adapter in this repo - the same engine + adapter split `@offgrid/clipboard` already uses. A mobile adapter can follow later. +2. **Agent browser for the web.** An embedded browser pane inside the app (clean profile, separate cookie jar) that the agent drives while the user watches - the surface Codex desktop and Claude Desktop both converged on in 2026. It handles research, forms, portals, and orders, and it needs zero OS permissions, so it ships before any native-GUI control. See 2.3. +3. **Vision-based agent as the native fallback.** When no deterministic surface or web path exists, a multi-agent GUI loop takes over, perceiving through the accessibility tree plus screenshots and acting through synthetic input. +4. **Replicate, do not invent.** The mobile-use ecosystem solved the agent loop (100% on AndroidWorld); Codex and Claude Desktop solved the browser UX; OpenClaw proved the capability composition - and its incident record is our avoid-list (see 2.4). We port, we do not design from scratch. +5. **Engine in shared.** The agent loop, action schema, router, and verification logic are platform-agnostic and land as a package in `off-grid-ai/shared`, with a desktop adapter in this repo - the same engine + adapter split `@offgrid/clipboard` already uses. A mobile adapter can follow later. +6. **The setup bar: none.** The target outcome is OpenClaw-class capability with zero setup. Our structure already delivers it: the model is bundled (no API keys), everything runs in-process (no gateway, no ports, no daemon), and permissions are just-in-time OS prompts, never config files. ## 2. What we are replicating @@ -46,6 +48,29 @@ The field converged on one architecture, and it matches the decided direction ex The 100% AndroidWorld run mixed models per role and only the Cortex was frontier-grade. That maps directly onto our stack: bundled Gemma runs planner/orchestrator/contextor/summarizer; the downloadable vision model runs Cortex and Reflector. +### 2.3 The agent browser (the web rail) - added August 12 + +The standalone AI browsers died in 2026 (OpenAI shut down Atlas, Google shut down Project Mariner); what survived at both OpenAI and Anthropic is a **tabbed browser pane embedded in the app**: clean profile, separate cookie jar, the user logs into sites deliberately in-pane, the agent drives the page in a live shared view, per-site permission cards gate actions. Codex desktop ships this in the thread; Claude Desktop ships it as the Browser pane. We adopt it as the preferred rail for every web task: + +- **Electron `WebContentsView`, driven in-process via `webContents.debugger`** (raw CDP - no debug port, no new dependencies; browser-use itself moved from Playwright to raw CDP for speed and control). +- **Perception is an indexed DOM/accessibility snapshot** (port of nanobrowser's TypeScript serialization) - multiple-choice element targeting that suits the bundled model - with screenshots as the fallback, per Agent TARS's dom / visual-grounding / hybrid strategy. +- **Zero OS permissions.** The pane is our own surface: no Accessibility grant, no Screen Recording, no TCC prompts at all. This makes it the first GUI capability we can ship. +- **Per-site cards** (allow once / always allow / deny; localhost exempt), **takeover mode** at logins and payments - agent input pauses and frame capture stops while the user types - and opt-in session persistence per site. + +### 2.4 What we reuse directly + +| Source | License | What we take | +| --- | --- | --- | +| `@ui-tars/sdk` + the UI-TARS desktop app | Apache-2.0 | two-method Operator seam + action parser; the ScreenMarker overlay trio (animated screen border, content-protected pause widget, pre-action click markers); desktopCapturer screenshot scaling; the macOS permission gate | +| nanobrowser | Apache-2.0 | TypeScript DOM-to-indexed-elements serialization for the browser rail | +| `@computer-use/nut-js` (or `@nut-tree-fork/nut-js`) | Apache-2.0 | input synthesis on the native rail, alongside our Swift helper | +| macos-automator-mcp | MIT | wrapped as an intents layer: AppleScript/JXA execution plus its recipe knowledge base | +| bytebot (archived) | Apache-2.0 | takeover-as-recorded-actions (the human demonstration lands in the same action log the agent uses) and the explicit needs_help task state | +| Peekaboo (OpenClaw org) | MIT | reference implementation for the a11y-tree + vision hybrid on the native rail | +| OpenClaw | AGPL (patterns only) | the capability composition and the proactive cron/skills pattern. Equally its incident record as the avoid-list: 30,000+ exposed gateways, sandbox-off default, weak local auth, unvetted skills. We ship none of those surfaces - no gateway, no ports, no BYO keys, skills approval-gated | + +Everything adopted as code is Apache-2.0 or MIT - clean for the AGPL core + proprietary pro split (ported files keep their license headers). + ### 2.2 The action schema A small closed enum with structured arguments, not an open toolbox. mobilerun ships nine tools: `click, long_press, type, system_button, swipe, open_app, get_state, take_screenshot, complete`. Mobile-Agent-v3's desktop set: `key, type, mouse_move, click, drag, right_click, middle_click, double_click, scroll, wait, terminate`. Element targeting always has a fallback chain: stable ID - text match - coordinates. We adopt the same shape (our chain: AXIdentifier/role+title - text match - coordinates). @@ -107,6 +132,15 @@ Tested in the package against a fake `DeviceController` before any surface wires Open-core: helper primitives and adapter plumbing in core; the wired agent surface and approvals integration follow the existing pro spine. `pro/` changes land in `desktop-pro` first, submodule bump after. +### 5.3 Build guidelines (brand and design) - binding on all phases + +Every computer-use surface (browser pane chrome, run view, approval cards, overlays, onboarding) follows the canonical guidelines: + +- **`off-grid-ai/brand`** - `DESIGN_PHILOSOPHY.md` is the cross-platform design source of truth: brutalist/terminal aesthetic, Menlo everywhere, emerald as the only accent (`#34D399` dark / `#059669` light), black/white base, hierarchy through size and weight rather than color, and every color/spacing/type value from `@offgrid/design` tokens - no hardcoded hex. Copy follows `brand_tone_voice.md` plus the outcomes-first rule in the brand README: lead with what the user gets, mechanism as proof. +- **`off-grid-ai/shared`** - `@offgrid/design` is where the tokens live; components inherit light/dark through the token mapping. +- **This repo's `docs/DESIGN.md`** - the desktop-first density rules (multi-column grids, tight 4/8/12 spacing, progressive disclosure, sticky context, micro-interactions). +- The agent's overlay widgets and permission cards are product UI like any other: no emojis, no gradients, no second accent, quiet by default. + ## 6. Safety The shipped-product template is Gemini Intelligence's UX, which matches our approvals spine: @@ -115,15 +149,20 @@ The shipped-product template is Gemini Intelligence's UX, which matches our appr - actions classified read / navigate / mutate / irreversible; the last two gate through the approval queue; everything lands in the audit log - screen content is untrusted input: published studies show 86% attack success from adversarial pop-ups against GUI agents, and prompt-level defenses do not work - the gate and the app allowlist are system-level for that reason - never see or type credentials: secure-input detection (`IsSecureEventInputEnabled()`) hands password fields to the user +- **hard-blocked action classes** no permission can unlock (the Claude in Chrome / Gemini precedent): purchases and financial transactions, account creation, permanent deletions, CAPTCHA bypass. Protected actions that always confirm even on always-allowed sites: sends, downloads, sensitive-data entry, authorization grants +- **takeover with a stated guarantee**: at logins and payments the agent pauses and frame capture stops while the user controls the surface; the human's actions during takeover are recorded into the same action log (the bytebot pattern) so the agent resumes with context +- **the agent cannot see or dismiss its own controls**: overlay widgets call `setContentProtection(true)` so they never appear in screenshots; Esc aborts globally and the keypress is consumed so on-screen content cannot dismiss an approval dialog +- **per-surface grants**: allow once / always allow / deny per site on the browser rail (localhost exempt); per-app grants on the native rail ## 7. Phasing | Phase | Scope | Exit test | | --- | --- | --- | -| 1. Intents + semantic rail | Swift helper (EventKit, Contacts, PhotoKit), AppleScript tools, `shortcuts run`, URL schemes, widened approval hook with risk classes | "create a calendar event Thursday 3pm" end to end, approval-gated, with the bundled model | -| 2. `@offgrid/use` engine | agent graph + action schema + router + verification in shared, tested against a fake `DeviceController` | engine passes its suite with a scripted fake device | -| 3. Desktop adapter + vision fallback | helper act-primitives, tree serialization, window-scoped capture, GUI-Owl/Holo catalog entries, Cortex on the vision model | a multi-step GUI task on a well-behaved app, verified per step | -| 4. Hard targets | Electron AX wake, WhatsApp flow, per-app recipes where trees are dead | the WhatsApp album task, supervised, send behind approval | +| 1. Intents + semantic rail | Swift helper (EventKit, Contacts, PhotoKit), AppleScript tools, `shortcuts run`, URL schemes, widened approval hook with risk classes, morning-briefing skill template | "create a calendar event Thursday 3pm" end to end, approval-gated, with the bundled model | +| 2. `@offgrid/use` engine | agent graph + action schema (including the browser action set) + router + verification in shared, tested against a fake `DeviceController` | engine passes its suite with a scripted fake device | +| 3. Agent browser | `WebContentsView` pane + CDP driver + indexed snapshot + per-site cards + takeover + live overlays | a multi-step web task in-pane, watched live, with a login takeover - shippable on its own | +| 4. Desktop adapter + vision fallback | helper act-primitives, tree serialization, window-scoped capture, GUI-Owl/Qwen3-VL catalog entries, Cortex on the vision model | a multi-step GUI task on a well-behaved app, verified per step | +| 5. Hard targets | Electron AX wake, WhatsApp flow, per-app recipes where trees are dead | the WhatsApp album task, supervised, send behind approval | ## 8. Decisions and open questions @@ -135,6 +174,8 @@ Status as of August 11, 2026 (details live in `COMPUTER_USE_PLAN.md`): 4. **Approval UX - proposed.** Plan-level approval + live step view + hold-to-stop, with per-step gates on irreversible actions. Validate against real runs in phase 3. 5. **Eval harness - decided.** macOSWorld subset as a non-blocking sanity gate from phase 3. 6. **Sequencing - decided.** Desktop first; mobile follows as an adapter-only project on the same engine (see the plan's "After v1" section; iOS is intents-only by platform rules). +7. **Agent browser rail - decided August 12.** Embedded pane over `WebContentsView` + `webContents.debugger`, indexed-snapshot perception, per-site cards, takeover mode. Ships before the native rail (zero OS permissions, no new models). +8. **Build guidelines - binding.** All UI and copy follow `off-grid-ai/brand` (design philosophy + tone), `@offgrid/design` tokens from `off-grid-ai/shared`, and this repo's `docs/DESIGN.md` (see 5.3). ## 9. Sources @@ -147,3 +188,6 @@ Status as of August 11, 2026 (details live in `COMPUTER_USE_PLAN.md`): - UI-TARS: https://github.com/bytedance/UI-TARS - OSWorld: https://os-world.github.io/ - Pop-up injection attacks (86% success): arXiv:2411.02391 - macOS surface: Electron `AXManualAccessibility` https://www.electronjs.org/docs/latest/tutorial/accessibility/ + electron/electron#38102 - secure input TN2150: https://developer.apple.com/library/mac/technotes/tn2150/_index.html - Shortcuts CLI: https://blakecrosley.com/guides/shortcuts - node-mac-permissions: https://github.com/codebytere/node-mac-permissions - WhatsApp Desktop AX findings: https://gist.github.com/hakanensari/99a7ddafbf1b92ce040dc68f43aa25d4 +- Agent browser + product UX: Electron debugger API https://www.electronjs.org/docs/latest/api/debugger - Claude Desktop browser pane https://code.claude.com/docs/en/desktop - Claude in Chrome permissions https://support.claude.com/en/articles/12902446-claude-in-chrome-permissions-guide - Codex embedded browser https://chierhu.medium.com/openai-codexs-browser-use-feature-b7dffa761d45 - browser-use's move to raw CDP https://browser-use.com/posts/playwright-to-cdp - nanobrowser https://github.com/nanobrowser/nanobrowser - UI-TARS desktop https://github.com/bytedance/UI-TARS-desktop - bytebot takeover https://github.com/bytebot-ai/bytebot - chrome-devtools-mcp https://github.com/ChromeDevTools/chrome-devtools-mcp - macos-automator-mcp https://github.com/steipete/macos-automator-mcp +- OpenClaw teardown: https://github.com/openclaw/openclaw - Peekaboo https://github.com/openclaw/Peekaboo - exposed-gateway findings https://www.bitsight.com/blog/openclaw-ai-security-risks-exposed-instances - ClawJacked https://www.infosecurity-magazine.com/news/clawjacked-bug-covert-ai-agent/ +- Brand and design: https://github.com/off-grid-ai/brand (DESIGN_PHILOSOPHY.md, brand_tone_voice.md, copywriting-rulebook.md) - `@offgrid/design` in https://github.com/off-grid-ai/shared diff --git a/docs/COMPUTER_USE_PLAN.md b/docs/COMPUTER_USE_PLAN.md index 13d0291e..fa4c1338 100644 --- a/docs/COMPUTER_USE_PLAN.md +++ b/docs/COMPUTER_USE_PLAN.md @@ -4,10 +4,18 @@ Companion to `COMPUTER_USE.md` (the approach). This doc is the execution plan: p **Assumptions** -- Solo developer (Siddharth), with AI doing the code authoring end to end. Durations are therefore paced by what does NOT compress: review, on-device verification, TCC permission flows, packaged-build checks, and iteration against real apps. Code-heavy phases are scheduled aggressively; integration-heavy phases keep slack. +- Solo developer (Siddharth), with AI doing the code authoring end to end. Durations are therefore paced by what does NOT compress: review, on-device verification, TCC permission flows, packaged-build checks, and iteration against real apps and websites. Code-heavy phases are scheduled aggressively; integration-heavy phases keep slack. - Dates start Wednesday, August 12, 2026 and assume this is the main focus. If a week goes elsewhere, shift the dates here - the phase order and checkpoints do not change. - Checkpoint discipline from the shared roadmap applies: a checkpoint is a verifiable, demoable milestone; the next phase does not start until it passes. -- **Vision model:** GUI-Owl-1.5-8B-Instruct (MIT) is the working default; Qwen3-VL-8B (Apache-2.0) is the alternate. Both are Qwen3-VL-architecture models, so every line of engine and adapter work is identical for either - the pick is a catalog decision made at install time (phase 3) and is not on the critical path before that. +- **Vision model:** GUI-Owl-1.5-8B-Instruct (MIT) is the working default; Qwen3-VL-8B (Apache-2.0) is the alternate. Both are Qwen3-VL-architecture models, so every line of engine and adapter work is identical for either - the pick is a catalog decision made at install time (phase 4) and is not on the critical path before that. The agent-browser phase needs no new model at all. + +## Build guidelines (standing, all phases) + +Binding on every UI surface and string this plan produces (pane chrome, run view, approval cards, overlays, onboarding, notifications): + +- **Design:** `off-grid-ai/brand` `DESIGN_PHILOSOPHY.md` is the canonical source - brutalist/terminal, Menlo everywhere, emerald as the only accent, black/white base, hierarchy by size and weight not color, no gradients, no emojis. All color/spacing/type values come from `@offgrid/design` tokens (`off-grid-ai/shared`) - no hardcoded hex. Desktop layout density per this repo's `docs/DESIGN.md`. +- **Copy:** `off-grid-ai/brand` `brand_tone_voice.md` plus the outcomes-first rule in the brand README - lead with what the user gets, mechanism as proof; no em dashes, no curly quotes, no exclamation marks, banned-word list applies. +- Keep a local clone of `off-grid-ai/brand` next to the repos and re-read before each UI-heavy phase (3, 4, 5). ## Timeline at a glance @@ -16,12 +24,13 @@ Companion to `COMPUTER_USE.md` (the approach). This doc is the execution plan: p | 0. Foundations | Aug 12 - Aug 14 | 3 days | this repo + `desktop-pro` | approval seam widened with risk classes; entitlements land; decisions locked | | 1. Semantic rail | Aug 17 - Aug 26 | 1.5 wk | this repo | "create a calendar event Thursday 3pm" end to end, approval-gated, bundled model | | 2. `@offgrid/use` engine | Aug 27 - Sep 9 | 2 wk | `shared` | engine suite green against a scripted fake device | -| 3. Desktop adapter + vision | Sep 10 - Sep 29 | 3 wk | this repo + `desktop-pro` | multi-step GUI task on a native app, verified per step, live step view | -| 4. Hard targets + hardening | Sep 30 - Oct 14 | 2 wk | this repo + `desktop-pro` | WhatsApp album task supervised end to end; safety checklist passes | +| 3. Agent browser | Sep 10 - Sep 23 | 2 wk | this repo | a multi-step web task in-pane, watched live, per-site cards, login takeover - **shippable v1-with-browser** | +| 4. Desktop adapter + vision | Sep 24 - Oct 13 | 3 wk | this repo + `desktop-pro` | multi-step GUI task on a native app, verified per step, live run view | +| 5. Hard targets + hardening | Oct 14 - Oct 27 | 2 wk | this repo + `desktop-pro` | WhatsApp album task supervised end to end; safety checklist passes | -The split to remember: `shared` holds the durable, cross-platform brain (2 of the 9 weeks, reused later by mobile); this repo holds the hands and the product integration (everything else). +Eleven weeks total, two shippable cuts inside it: phase 1 alone (semantic actions, Aug 26) and phases 0-3 (everything plus the agent browser, Sep 23). The split to remember: `shared` holds the durable, cross-platform brain (reused later by mobile); this repo holds the hands, the browser pane, and the product integration. -Nine weeks total. Phase 1 is independently shippable - calendar, messages, mail, photos, reminders actions with approvals deliver user value on Aug 26 regardless of what follows. Where the compression came from: the engine phase is pure TypeScript against a fake device (ideal AI-authoring territory, 4 wk to 2 wk); the semantic rail is many small similar wrappers (3 wk to 1.5 wk). Phase 3 keeps the most slack because it mixes Swift interop, a model install, and real-device iteration; phase 4 is paced by WhatsApp itself, not by code. +Where the browser phase came from (added August 12): the embedded agent-browser pane is the surface Codex desktop and Claude Desktop both converged on in 2026. It needs zero OS permissions and no new models, and it covers most real-world GUI tasks (anything web), so it jumps ahead of native-GUI control. ## Phase 0 - foundations (Aug 12 - Aug 14) @@ -30,73 +39,87 @@ Nine weeks total. Phase 1 is independently shippable - calendar, messages, mail, | Lock decisions | v1 graph roles (proposed: Planner / Cortex / Executor / Reflector, add Orchestrator + Summarizer when task length demands), package name (`@offgrid/use`), approval UX (proposed: plan-level approval + live step view + per-step gates on irreversible actions) | recorded in `COMPUTER_USE.md` section 8 | | Widen the approval seam | `mcp:proposeApproval` becomes transport-agnostic `actions:proposeApproval` carrying `{kind, title, detail, risk, args, source}`; per-executor `riskOf(name, args)` replaces the name regex; MCP extension migrates onto it | existing approval tests stay green (`mcp-connector-tool-extension.dbtest.ts`); new tests per risk class; pro executor updated in `desktop-pro`, submodule bumped | | Packaging groundwork | `com.apple.security.automation.apple-events` entitlement + `NSAppleEventsUsageDescription` + Calendars/Contacts/Photos usage keys in the electron-builder config | local packaged build per `local-build.local.md`; TCC prompts name the app | -| Permission onboarding design | sequence of grants (Accessibility held check exists, Automation per target, per-framework) behind an explicit "enable computer actions" flow | design reviewed; no dead-end states | +| Permission onboarding design | just-in-time grants only (the browser rail needs none; Accessibility/Automation prompt on first native-rail use, deep-linked to the right Settings pane) | design reviewed; no dead-end states | | Shared repo setup | sibling clone of `off-grid-ai/shared` next to this repo (CLAUDE.md already assumes `../shared`); decide the consumption pattern for `@offgrid/use` - proposed: the README's `file:../shared/packages/use`, NOT a fifth vendored copy under `packages/` (confirm with the lead; makes a sibling checkout a build requirement on this branch) | checkout builds; decision recorded here | +| Brand setup | sibling clone of `off-grid-ai/brand`; skim `DESIGN_PHILOSOPHY.md` + `brand_tone_voice.md` before any UI work | clone present | ## Phase 1 - semantic rail (Aug 17 - Aug 26) - **Aug 17 - 19:** `actions` Swift helper skeleton (JSON over stdio, same `execFile` pattern as `src/main/ocr.ts`); EventKit create/read events + reminders; Contacts read. Unit tests on the Node wrapper; helper built alongside the existing Swift helpers. -- **Aug 20 - 21:** AppleScript surfaces - Messages send, Mail compose, Notes create/search, Finder basics; URL-scheme opener; `open -b` app launcher; `shortcuts run` wrapper with timeout guards (a prompting shortcut hangs the CLI - wrap every call). -- **Aug 24 - 26:** expose everything as tools through the `ToolExtension` seam with risk classes; approval-gated writes; audit entries; permission onboarding wired. Tests: unit per risk class and arg mapping, integration on a temp profile, e2e smoke asserting the approval queue renders the new action kinds. +- **Aug 20 - 21:** AppleScript surfaces - Messages send, Mail compose, Notes create/search, Finder basics; URL-scheme opener; `open -b` app launcher; `shortcuts run` wrapper with timeout guards (a prompting shortcut hangs the CLI - wrap every call). Reference: macos-automator-mcp (MIT) for script bodies and its recipe-KB pattern. +- **Aug 24 - 26:** expose everything as tools through the `ToolExtension` seam with risk classes; approval-gated writes; audit entries; permission onboarding wired. **Morning-briefing skill template** (cron trigger + read-only rails + notification output) as the first proactive workflow. Tests: unit per risk class and arg mapping, integration on a temp profile, e2e smoke asserting the approval queue renders the new action kinds. -**Checkpoint (Aug 26):** calendar event created end to end from chat with the bundled model, gated, audited. Messages send queues for approval and executes on approve. +**Checkpoint (Aug 26):** calendar event created end to end from chat with the bundled model, gated, audited. Messages send queues for approval and executes on approve. The briefing skill runs on a demo profile. ## Phase 2 - `@offgrid/use` engine in shared (Aug 27 - Sep 9) All work in `off-grid-ai/shared`; testable with zero macOS dependencies - the phase where AI authoring compresses the most. -- **Aug 27 - 31:** package scaffold; `DeviceController` interface (`getState()`, `act(action)`, `openIntent(url)`, `listSemanticSurfaces()`); closed action schema + structured decision types; fake `DeviceController` with scripted scenarios. -- **Sep 1 - 4:** graph v1 - Planner (subgoal decomposition), Cortex (one structured decision per step from tree + screenshot), Executor (deterministic parse, no free-form reasoning); the router (deterministic surface - structured UI - vision). +- **Aug 27 - 31:** package scaffold; `DeviceController` interface (`getState()`, `act(action)`, `openIntent(url)`, `listSemanticSurfaces()`); closed action schema + structured decision types - **including the browser action set** (navigate, click-ref, type-ref, scroll, back, tab ops) alongside native actions; fake `DeviceController` with scripted scenarios. +- **Sep 1 - 4:** graph v1 - Planner (subgoal decomposition), Cortex (one structured decision per step from indexed snapshot + optional screenshot), Executor (deterministic parse, no free-form reasoning); the router (deterministic surface - agent browser - structured UI - vision). - **Sep 7 - 9:** Reflector (SUCCESS/FAILURE per transition with diagnosis), deterministic post-validation for typing (act, re-read, diff - minitap's +7 pt mechanism), cycle detection with forced strategy change (+9 pt mechanism); per-role model config against an OpenAI-compatible endpoint (our gateway); risk + approval callback seam; suite hardening; package docs. -**Checkpoint (Sep 9):** suite green on the fake device, including a type-verify scenario (field readback mismatch triggers retry) and a loop-escape scenario (repeated state triggers strategy change). +**Checkpoint (Sep 9):** suite green on the fake device, including a type-verify scenario (field readback mismatch triggers retry), a loop-escape scenario (repeated state triggers strategy change), and a browser-action scenario against a fake browser controller. + +## Phase 3 - agent browser (Sep 10 - Sep 23) + +The embedded web rail. Zero OS permissions, no new models - the bundled model works from indexed snapshots. Reference implementations: nanobrowser (Apache-2.0, TS serialization to port), UI-TARS desktop `ui-helper` (Apache-2.0, in-page overlay UX), Claude Desktop browser pane (permission card UX to replicate). + +- **Sep 10 - 14:** the pane - `WebContentsView` with tabs and brand-compliant chrome inside the app; `BrowserOperator` over `webContents.debugger` (raw CDP: navigate, `Input.dispatch*` click/type/scroll, `Page.captureScreenshot`); navigation-lifecycle waits; clean persistent profile separate from any user browser. +- **Sep 15 - 17:** perception - indexed DOM/accessibility snapshot (port nanobrowser's serialization over `DOM.getDocument` + `Accessibility.getFullAXTree`); hybrid snapshot/vision strategy behind the operator seam; per-step GBNF grammar constraining actions to element refs that exist on the current page. +- **Sep 18 - 21:** control - per-site permission cards (allow once / always allow / deny; localhost exempt) with a review screen; hard-blocked classes (purchase, account creation, permanent delete, CAPTCHA) and always-confirm protected actions; **takeover mode** (agent input pauses, frame capture stops, human actions recorded into the action log, resume with context); live overlays injected via `executeJavaScript` (status panel, click ripples); step feed in chat. +- **Sep 22 - 23:** e2e with screenshots; session-persistence opt-in per site; checkpoint demo run. + +**Checkpoint (Sep 23):** a real multi-step web task (research a product across two sites, fill a form) completes in-pane, watched live, with a per-site card raised on first action and a login handled by takeover. **This cut is shippable: semantic actions + proactive skills + web automation.** -## Phase 3 - desktop adapter + vision fallback (Sep 10 - Sep 29) +## Phase 4 - desktop adapter + vision fallback (Sep 24 - Oct 13) -Keeps the most slack: Swift interop, a model install, and real-device iteration all live here. +Native-app control. Keeps the most slack: Swift interop, a model install, and real-device iteration all live here. -- **Sep 10 - 15:** Swift helper act primitives - `AXUIElementPerformAction`, set-`AXValue`, `AXManualAccessibility` wake, `CGEvent` post; coordinate mapping (AX points vs screenshot pixels, `backingScaleFactor` per display, multi-display origins). -- **Sep 16 - 21:** AX tree serialization (indexed interactive elements + bounding boxes, drop unlabeled wrappers, diffs after actions); window-scoped capture via `src/main/vision.ts`; OCR helper emits `{text, bbox}`. -- **Sep 22 - 24:** model catalog entries (GUI-Owl-1.5-8B-Instruct GGUF + mmproj; Qwen3-VL-8B alternate) - **model install happens here**; Cortex/Reflector wired through the gateway; agent runs at modality-queue tier 2. -- **Sep 25 - 29:** supervised run UI - live step view, hold-to-stop, per-step approvals for irreversible actions; e2e with screenshots; small eval harness (macOSWorld subset) as a sanity gate, not CI-blocking. +- **Sep 24 - 29:** Swift helper act primitives - `AXUIElementPerformAction`, set-`AXValue`, `AXManualAccessibility` wake, `CGEvent` post; coordinate mapping (AX points vs screenshot pixels, `backingScaleFactor` per display, multi-display origins). +- **Sep 30 - Oct 5:** AX tree serialization (indexed interactive elements + bounding boxes, diffs after actions); window-scoped capture via `src/main/vision.ts`; OCR helper emits `{text, bbox}`; **ScreenMarker port** (UI-TARS trio: animated screen border, content-protected pause/stop widget, pre-action markers - Apache-2.0 files, keep headers). +- **Oct 6 - 8:** model catalog entries (GUI-Owl-1.5-8B-Instruct GGUF + mmproj; Qwen3-VL-8B alternate) - **model install happens here**; Cortex/Reflector wired through the gateway; agent runs at modality-queue tier 2. +- **Oct 9 - 13:** supervised run flow end to end - live step view, hold-to-stop, per-app grants, per-step approvals for irreversible actions; e2e with screenshots; small eval harness (macOSWorld subset) as a sanity gate, not CI-blocking. -**Checkpoint (Sep 29):** a multi-step GUI task on a well-behaved native app (for example: create and reschedule a reminder through the Reminders UI) completes with per-step verification visible in the run view. +**Checkpoint (Oct 13):** a multi-step GUI task on a well-behaved native app (for example: create and reschedule a reminder through the Reminders UI) completes with per-step verification visible in the run view. -## Phase 4 - hard targets + hardening (Sep 30 - Oct 14) +## Phase 5 - hard targets + hardening (Oct 14 - Oct 27) Paced by iteration against real apps, not by code volume. -- **Sep 30 - Oct 2:** Electron AX wake flows; secure-input detection surfaced in System Health; app allowlist setting. -- **Oct 5 - 9:** the WhatsApp acceptance case - `whatsapp://` intent open, keyboard-first navigation, vision-grounded clicks for the gaps, photo ranking by the vision model, send behind the gate. Per-app recipe container (skills format) so the flow is data, not code. -- **Oct 12 - 14:** injection-resistance review against the screen-content threat model; kill-switch e2e; release-readiness pass (docs, System Health states, permission recovery). +- **Oct 14 - 16:** Electron AX wake flows; secure-input detection surfaced in System Health; app allowlist setting; global Esc abort with consumed keypress. +- **Oct 19 - 23:** the WhatsApp acceptance case - `whatsapp://` intent open, keyboard-first navigation, vision-grounded clicks for the gaps, photo ranking by the vision model, send behind the gate. Per-app recipe container (skills format) so the flow is data, not code. +- **Oct 26 - 27:** injection-resistance review against the screen-content threat model (browser rail included); kill-switch e2e; release-readiness pass (docs, System Health states, permission recovery). -**Checkpoint (Oct 14):** WhatsApp album task runs supervised end to end on a demo profile; the safety checklist (gates, allowlist, secure input, kill switch, audit) passes. +**Checkpoint (Oct 27):** WhatsApp album task runs supervised end to end on a demo profile; the safety checklist (gates, hard-blocked classes, allowlist, secure input, takeover capture-kill, kill switch, audit) passes. ## Dependencies | What | Needed by | Note | | --- | --- | --- | -| Vision model install (GUI-Owl-1.5-8B or Qwen3-VL-8B GGUF + mmproj) | Sep 22 | nothing earlier depends on it | -| `off-grid-ai/shared` CI for the new package | Aug 27 | one-time setup, ~half a day | -| Dev-machine TCC grants (Accessibility, Automation) with a stable dev signing identity - grants key to code signature | Aug 17 and Sep 10 | per `local-build.local.md`; keep one identity or grants orphan | -| `desktop-pro` changes for the approval executor | Aug 12 - 14 | separate repo commits + submodule bump here | -| Push access to `off-grid-ai/OGAD` (and `desktop-pro` if not held) | before the first push - local commits proceed without it | as of Aug 11 the account is pull-only on OGAD; `shared` already has push. Ask the lead | +| Vision model install (GUI-Owl-1.5-8B or Qwen3-VL-8B GGUF + mmproj) | Oct 6 | phases 0-3 do not need it | +| `off-grid-ai/shared` CI for the new package | Aug 27 | one-time setup, ~half a day; push access already held | +| Sibling clones: `../shared`, `../brand` | Aug 12 | build guideline + consumption pattern | +| Dev-machine TCC grants (Accessibility, Automation) with a stable dev signing identity - grants key to code signature | Aug 17 and Sep 24 | per `local-build.local.md`; keep one identity or grants orphan | +| `desktop-pro` changes for the approval executor | Aug 12 - 14 | repo access still pending as of Aug 12 - stub the pro executor if access lags | +| Push access to `off-grid-ai/OGAD` | done (Aug 11) | write access granted and verified | ## Risks | Risk | Mitigation | | --- | --- | -| Solo schedule: one blocked day is a lost day, no parallel track | phases are independently valuable; a slip moves dates in this doc, never skips a checkpoint | +| Solo schedule: one blocked day is a lost day, no parallel track | phases are independently valuable (two shippable cuts); a slip moves dates in this doc, never skips a checkpoint | | AI-authored code lands faster than it is verified | the pace is set by the checkpoints, not by lines written: nothing is "done" until its tests and on-device check pass (repo rule: commit only green units) | +| Web pages vary wildly; snapshot quality drives browser-rail success | port nanobrowser's proven serialization rather than inventing; screenshots as fallback; per-step grammar keeps the model on valid refs | | macOS AX trees are uneven, vision carries more steps than on mobile | GUI-Owl is desktop-trained; window-scoped capture keeps resolution sane; recipes for the worst apps | -| Desktop agent scores run 20-30 pts below mobile headlines | supervised UX with per-step verify is the product shape, not autonomy claims | +| Desktop agent scores run 20-30 pts below mobile headlines | supervised UX with per-step verify is the product shape, not autonomy claims; the browser rail carries most tasks deterministically | | App updates break GUI flows (WhatsApp foremost) | flows live in recipe data; the acceptance case is a demo target, not a launch gate | | TCC dev trap: grants orphan when the signing identity changes | stable dev identity, `tccutil reset` runbook note | -| Model landscape shifts before phase 3 | per-role model config; swapping the Cortex model touches zero engine code | +| Model landscape shifts before phase 4 | per-role model config; swapping the Cortex model touches zero engine code | ## Out of scope for v1 -Windows adapter, teach-and-repeat recording, background/headless runs, Mac App Store distribution (sandbox blocks the Accessibility path - we ship Developer ID). +Windows adapter, teach-and-repeat recording, background/headless runs, chat-channel clients (WhatsApp/Telegram as control surfaces - the OpenClaw pattern; revisit after v1), Mac App Store distribution (sandbox blocks the Accessibility path - we ship Developer ID). ## After v1 - the mobile adapter (unscheduled) @@ -108,5 +131,5 @@ Desktop ships first; mobile follows as an adapter-only project on the same `@off ## Tracking -- Branch: `feat/computer-use`. Small commits per verified unit, merge not squash. PR evidence rules apply (screenshots per changed surface; video for the run-view and WhatsApp demos). +- Branch: `feat/computer-use`. Small commits per verified unit, merge not squash. PR evidence rules apply (screenshots per changed surface; video for the browser pane, run-view, and WhatsApp demos). - Weekly checkpoint review against this doc; date changes are edits to this doc. From 0a4e563a3f62aa6587aad0e953d9baa3272bc5a8 Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Wed, 12 Aug 2026 12:48:34 +0530 Subject: [PATCH 08/75] feat(actions): transport-agnostic approval seam with risk classes Phase 0 of computer use. Widens the MCP-only `mcp:proposeApproval` hook into a shared `actions:proposeApproval` seam every executor (MCP today, computer + browser next) can gate through. Adds an ActionRisk taxonomy (read/navigate/mutate/irreversible) with a single shouldGate() source of truth, and per-executor riskOf() classification - a GUI click's risk is not derivable from a tool name the way the old isActionTool regex assumed. Backward compatible: proposeActionApproval prefers the new hook but falls back to the legacy name via hasHook(), so a desktop-pro build that has not yet migrated keeps gating MCP writes instead of silently running them. MCP behaviour is unchanged (read-verb tools -> read, everything else -> mutate), pinned by the existing queue-vs-execute dbtest plus new risk tests. Verified: tsc (node + web), 41 unit tests, 7 dbtest, eslint clean. Co-Authored-By: Claude Fable 5 --- .../mcp-connector-tool-extension.dbtest.ts | 7 +- src/main/actions/__tests__/approval.test.ts | 78 +++++++++++++++++++ src/main/actions/approval.ts | 65 ++++++++++++++++ .../bootstrap/__tests__/hookRegistry.test.ts | 31 +++++++- src/main/bootstrap/hookRegistry.ts | 23 +++++- .../mcpConnectorToolExtension.test.ts | 26 ++++++- .../tools/mcpConnectorToolExtension-logic.ts | 10 +++ src/main/tools/mcpConnectorToolExtension.ts | 22 +++--- 8 files changed, 248 insertions(+), 14 deletions(-) create mode 100644 src/main/actions/__tests__/approval.test.ts create mode 100644 src/main/actions/approval.ts diff --git a/src/main/__tests__/mcp-connector-tool-extension.dbtest.ts b/src/main/__tests__/mcp-connector-tool-extension.dbtest.ts index 82b02338..82c2b800 100644 --- a/src/main/__tests__/mcp-connector-tool-extension.dbtest.ts +++ b/src/main/__tests__/mcp-connector-tool-extension.dbtest.ts @@ -25,6 +25,7 @@ import { type McpConnectorToolBoundary } from '../tools/mcpConnectorToolExtension' import type { ConnectorToolDefinition } from '../tools/mcpConnectorToolExtension-logic' +import type { ActionApprovalRequest } from '../actions/approval' interface ToolExecution { connectorId: number @@ -38,7 +39,7 @@ class FakeMcpBoundary implements McpConnectorToolBoundary { readonly tools = new Map() readonly results = new Map() readonly executions: ToolExecution[] = [] - readonly approvals: Record[] = [] + readonly approvals: ActionApprovalRequest[] = [] approveWrites = false async fetchTools(connectorId: number): Promise { @@ -62,7 +63,7 @@ class FakeMcpBoundary implements McpConnectorToolBoundary { return result } - proposeApproval(request: Record): boolean { + proposeApproval(request: ActionApprovalRequest): boolean { this.approvals.push(request) return this.approveWrites } @@ -149,6 +150,8 @@ describe('McpConnectorToolExtension with real connector state', () => { expect(output).toContain('Queued for the user') expect(boundary.approvals).toEqual([ expect.objectContaining({ + kind: 'mcp', + risk: 'mutate', connectorId, tool: 'send_message', connector: 'Slack', diff --git a/src/main/actions/__tests__/approval.test.ts b/src/main/actions/__tests__/approval.test.ts new file mode 100644 index 00000000..fee1f236 --- /dev/null +++ b/src/main/actions/__tests__/approval.test.ts @@ -0,0 +1,78 @@ +/** + * Unit tests for the transport-agnostic action-approval seam. High blast radius: + * every executor that can act on the user's behalf (MCP connectors today, computer + * and browser actions next) gates through shouldGate + proposeActionApproval, and + * the free/pro split hinges on whether a hook is registered. + */ +import { afterEach, describe, expect, it } from 'vitest' +import { + shouldGate, + proposeActionApproval, + type ActionApprovalRequest, + type ActionRisk +} from '../approval' +import { registerHook, unregisterHook, HOOKS } from '../../bootstrap/hookRegistry' + +const NEW = HOOKS.actionsProposeApproval +const LEGACY = HOOKS.legacyMcpProposeApproval + +function request(risk: ActionRisk): ActionApprovalRequest { + return { kind: 'mcp', title: 't', detail: 'd', risk, args: {}, source: 'chat' } +} + +afterEach(() => { + unregisterHook(NEW) + unregisterHook(LEGACY) +}) + +describe('shouldGate', () => { + it('gates mutate and irreversible, runs read and navigate freely', () => { + expect(shouldGate('mutate')).toBe(true) + expect(shouldGate('irreversible')).toBe(true) + expect(shouldGate('read')).toBe(false) + expect(shouldGate('navigate')).toBe(false) + }) +}) + +describe('proposeActionApproval', () => { + it('returns undefined when nothing is listening (free build runs the action)', () => { + expect(proposeActionApproval(request('mutate'))).toBeUndefined() + }) + + it('routes to the new hook and forwards its verdict', () => { + registerHook(NEW, () => true) + expect(proposeActionApproval(request('mutate'))).toBe(true) + }) + + it('honours a registered new hook that declined to queue (returns false)', () => { + registerHook(NEW, () => false) + expect(proposeActionApproval(request('mutate'))).toBe(false) + }) + + it('trusts a registered new hook even when it returns undefined — no legacy fallback', () => { + let legacyCalled = false + registerHook(NEW, () => undefined) + registerHook(LEGACY, () => { + legacyCalled = true + return true + }) + expect(proposeActionApproval(request('mutate'))).toBeUndefined() + expect(legacyCalled).toBe(false) + }) + + it('falls back to the legacy hook when the new name is unregistered', () => { + registerHook(LEGACY, () => true) + expect(proposeActionApproval(request('mutate'))).toBe(true) + }) + + it('passes the full request through to the handler', () => { + let seen: ActionApprovalRequest | undefined + registerHook(NEW, (req: ActionApprovalRequest) => { + seen = req + return true + }) + const req = request('irreversible') + proposeActionApproval(req) + expect(seen).toEqual(req) + }) +}) diff --git a/src/main/actions/approval.ts b/src/main/actions/approval.ts new file mode 100644 index 00000000..dd196a49 --- /dev/null +++ b/src/main/actions/approval.ts @@ -0,0 +1,65 @@ +// Transport-agnostic action-approval seam (core). Every executor that acts on the +// user's behalf — MCP connectors today, computer/GUI actions and the agent browser +// next — classifies each action's risk and, when it is consequential, offers it to +// the approval hook BEFORE doing it. Pro registers the hook to route the action +// through its approval queue + audit log; the free build registers nothing, so the +// action just runs (unchanged free behaviour). +// +// This replaces the MCP-specific `mcp:proposeApproval` hook: the old one carried a +// connector-shaped payload and derived risk from a tool-name regex, neither of +// which generalises to a GUI click (a click is always a write; a screenshot never +// is). Risk is classified per executor via its own riskOf(); the shape below is the +// one thing every executor shares. + +import { callHook, hasHook, HOOKS } from '../bootstrap/hookRegistry' + +/** How consequential an action is, independent of which executor produced it. + * - read: observes only, never changes the world (a screenshot, a list call) + * - navigate: moves focus/location without committing (open a URL, scroll) + * - mutate: changes state, usually recoverable (send a message, create an event) + * - irreversible: cannot be undone (delete, pay, submit, create an account) + * read/navigate run freely; mutate/irreversible are offered for approval. */ +export type ActionRisk = 'read' | 'navigate' | 'mutate' | 'irreversible' + +/** Which executor raised the action — lets the approval UI and audit log group and + * label without branching on executor-specific fields. */ +export type ActionKind = 'mcp' | 'computer' | 'browser' + +export interface ActionApprovalRequest { + kind: ActionKind + /** One-line, user-facing summary of what will happen. */ + title: string + /** Longer context for the approval card (arguments, source surface). */ + detail: string + risk: ActionRisk + /** Structured arguments, passed through to the executor on approval. */ + args: Record + /** Where the action originated (e.g. 'chat', a skill id). */ + source: string + /** Executor-specific fields (connectorId/tool for mcp, selector for browser). + * Left open so the seam never needs to know each executor's payload shape. */ + [extra: string]: unknown +} + +/** mutate and irreversible actions gate; read and navigate run freely. The single + * source of truth for the gating rule — executors and tests both call this rather + * than re-encoding the set. */ +export function shouldGate(risk: ActionRisk): boolean { + return risk === 'mutate' || risk === 'irreversible' +} + +/** Offer an action to the approval hook. Returns true when it was queued (the + * caller must NOT execute), false when a handler ran but did not queue it, and + * undefined when nothing is listening (free build — execute now). + * + * Falls back to the legacy `mcp:proposeApproval` hook so a pro build that has not + * yet migrated keeps gating MCP writes instead of silently running them. hasHook + * distinguishes "new handler present" from "new handler returned undefined", so a + * registered new handler is always authoritative and the legacy path is only used + * when the new name is genuinely unregistered. */ +export function proposeActionApproval(request: ActionApprovalRequest): boolean | undefined { + if (hasHook(HOOKS.actionsProposeApproval)) { + return callHook(HOOKS.actionsProposeApproval, request) + } + return callHook(HOOKS.legacyMcpProposeApproval, request) +} diff --git a/src/main/bootstrap/__tests__/hookRegistry.test.ts b/src/main/bootstrap/__tests__/hookRegistry.test.ts index 1730ceeb..60b7d57a 100644 --- a/src/main/bootstrap/__tests__/hookRegistry.test.ts +++ b/src/main/bootstrap/__tests__/hookRegistry.test.ts @@ -5,7 +5,14 @@ * and universal-search sources both route through it. */ import { describe, it, expect } from 'vitest' -import { registerHook, callHook, callHookAsync, HOOKS } from '../hookRegistry' +import { + registerHook, + unregisterHook, + hasHook, + callHook, + callHookAsync, + HOOKS +} from '../hookRegistry' describe('hookRegistry', () => { it('registers a hook and callHook returns its result', () => { @@ -48,8 +55,30 @@ describe('hookRegistry', () => { await expect(callHookAsync('t.sync-via-async')).resolves.toBe('plain') }) + it('hasHook reports registration and unregisterHook removes it', () => { + expect(hasHook('t.presence')).toBe(false) + registerHook('t.presence', () => 1) + expect(hasHook('t.presence')).toBe(true) + unregisterHook('t.presence') + expect(hasHook('t.presence')).toBe(false) + expect(callHook('t.presence')).toBeUndefined() + }) + + it('hasHook is true even for a hook that returns undefined', () => { + registerHook('t.returns-undefined', () => undefined) + expect(hasHook('t.returns-undefined')).toBe(true) + expect(callHook('t.returns-undefined')).toBeUndefined() + unregisterHook('t.returns-undefined') + }) + + it('unregisterHook is a no-op for an unknown key', () => { + expect(() => unregisterHook('t.never')).not.toThrow() + }) + it('exposes the known hook-name constants core and pro share', () => { expect(HOOKS.chatAugmentContext).toBe('chat.augmentContext') expect(HOOKS.searchExtraSources).toBe('search.extraSources') + expect(HOOKS.actionsProposeApproval).toBe('actions:proposeApproval') + expect(HOOKS.legacyMcpProposeApproval).toBe('mcp:proposeApproval') }) }) diff --git a/src/main/bootstrap/hookRegistry.ts b/src/main/bootstrap/hookRegistry.ts index 7565bf70..07411c92 100644 --- a/src/main/bootstrap/hookRegistry.ts +++ b/src/main/bootstrap/hookRegistry.ts @@ -16,6 +16,19 @@ export function registerHook(name: string, fn: HookFn): void { hooks[name] = fn } +/** Remove a registered hook. No-op when absent. Mainly for test isolation and + * for retiring a legacy hook name once its replacement is registered. */ +export function unregisterHook(name: string): void { + delete hooks[name] +} + +/** Whether a hook is currently registered. Lets a caller distinguish "no handler" + * from "handler ran and returned undefined" — needed when falling back from a new + * hook name to a legacy one. */ +export function hasHook(name: string): boolean { + return name in hooks +} + /** Call a hook if registered; returns its result, or undefined when absent. */ export function callHook(name: string, ...args: unknown[]): R | undefined { const fn = hooks[name] @@ -38,5 +51,13 @@ export const HOOKS = { * system/context with captured memory + entity/observation context (pro). */ chatAugmentContext: 'chat.augmentContext', /** () => Promise — extra universal-search sources (pro). */ - searchExtraSources: 'search.extraSources' + searchExtraSources: 'search.extraSources', + /** (request: ActionApprovalRequest) => boolean — offer a consequential action + * for approval; returns true when queued (caller must not execute). Pro + * registers it to route the action through its approval queue + audit log. */ + actionsProposeApproval: 'actions:proposeApproval', + /** Legacy MCP-only predecessor of actionsProposeApproval. Kept so a pro build + * that has not yet migrated still gates connector writes; remove once + * desktop-pro registers actionsProposeApproval. */ + legacyMcpProposeApproval: 'mcp:proposeApproval' } as const diff --git a/src/main/tools/__tests__/mcpConnectorToolExtension.test.ts b/src/main/tools/__tests__/mcpConnectorToolExtension.test.ts index 5e3a7591..63e43685 100644 --- a/src/main/tools/__tests__/mcpConnectorToolExtension.test.ts +++ b/src/main/tools/__tests__/mcpConnectorToolExtension.test.ts @@ -2,8 +2,10 @@ import { describe, expect, it } from 'vitest' import { buildConnectorToolSchema, formatConnectorToolResult, - isActionTool + isActionTool, + riskOf } from '../mcpConnectorToolExtension-logic' +import { shouldGate } from '../../actions/approval' describe('isActionTool', () => { it.each([ @@ -37,6 +39,28 @@ describe('isActionTool', () => { }) }) +describe('riskOf', () => { + it('maps read-verb tools to a non-gating read risk', () => { + for (const tool of ['list_channels', 'get_user', 'search_docs', 'read_file']) { + expect(riskOf(tool)).toBe('read') + expect(shouldGate(riskOf(tool))).toBe(false) + } + }) + + it('maps every other tool to a gating mutate risk', () => { + for (const tool of ['send_message', 'create_issue', 'delete_record']) { + expect(riskOf(tool)).toBe('mutate') + expect(shouldGate(riskOf(tool))).toBe(true) + } + }) + + it('agrees with isActionTool on which tools gate', () => { + for (const tool of ['list_channels', 'send_message', 'get_user', 'delete_record']) { + expect(shouldGate(riskOf(tool))).toBe(isActionTool(tool)) + } + }) +}) + describe('buildConnectorToolSchema', () => { it('namespaces the tool and retains its description and input schema', () => { expect( diff --git a/src/main/tools/mcpConnectorToolExtension-logic.ts b/src/main/tools/mcpConnectorToolExtension-logic.ts index 4ada1f39..dcbf5b53 100644 --- a/src/main/tools/mcpConnectorToolExtension-logic.ts +++ b/src/main/tools/mcpConnectorToolExtension-logic.ts @@ -1,3 +1,5 @@ +import type { ActionRisk } from '../actions/approval' + export const MCP_TOOL_PREFIX = 'mcp__' export interface ConnectorToolDefinition { @@ -19,6 +21,14 @@ export function isActionTool(tool: string): boolean { return !/^(list|get|search|read|fetch|whoami|describe)[_-]/i.test(tool) } +/** Classify a connector tool for the shared approval seam. MCP gives us only the + * tool name, so read-verb tools are reads and everything else is a mutate — we + * cannot tell an irreversible connector call from a recoverable one by name, so + * we gate conservatively as mutate rather than guessing 'irreversible'. */ +export function riskOf(tool: string): ActionRisk { + return isActionTool(tool) ? 'mutate' : 'read' +} + export function buildConnectorToolSchema( connector: { id: number; name: string }, tool: ConnectorToolDefinition diff --git a/src/main/tools/mcpConnectorToolExtension.ts b/src/main/tools/mcpConnectorToolExtension.ts index 12192dc9..0a77241b 100644 --- a/src/main/tools/mcpConnectorToolExtension.ts +++ b/src/main/tools/mcpConnectorToolExtension.ts @@ -2,18 +2,19 @@ // loop via registerToolExtension. Connector tools are exposed to the model // namespaced as `mcp____` and executed directly. // -// Open-core seam: write tools first offer themselves to the `mcp:proposeApproval` -// hook — Pro registers it to route writes through its approval queue. In the free -// build no hook is registered, so connector tools just run. +// Open-core seam: mutating tools first offer themselves to the shared +// `actions:proposeApproval` hook via proposeActionApproval — Pro registers it to +// route writes through its approval queue. In the free build no hook is registered, +// so connector tools just run. import type { ToolExtension } from '../tools' import { listConnectors, fetchTools, callConnectorTool, setConnectorStatus } from '../mcp' -import { callHook } from '../bootstrap/hookRegistry' +import { proposeActionApproval, shouldGate, type ActionApprovalRequest } from '../actions/approval' import { MCP_TOOL_PREFIX, buildConnectorToolSchema, formatConnectorToolResult, - isActionTool, + riskOf, type ConnectorToolDefinition } from './mcpConnectorToolExtension-logic' @@ -30,13 +31,13 @@ export interface McpConnectorToolBoundary { tool: string, args: Record ) => Promise - proposeApproval: (request: Record) => boolean | undefined + proposeApproval: (request: ActionApprovalRequest) => boolean | undefined } const productionBoundary: McpConnectorToolBoundary = { fetchTools, callTool: callConnectorTool, - proposeApproval: (request) => callHook('mcp:proposeApproval', request) + proposeApproval: proposeActionApproval } export class McpConnectorToolExtension implements ToolExtension { @@ -90,11 +91,14 @@ export class McpConnectorToolExtension implements ToolExtension { async execute(name: string, args: Record): Promise { const meta = this.byName.get(name) if (!meta) return `Error: unknown connector tool ${name}` - // Pro can intercept writes for approval; returns true if it queued the action. - if (isActionTool(meta.tool)) { + // Pro can intercept mutating tools for approval; returns true if it queued them. + const risk = riskOf(meta.tool) + if (shouldGate(risk)) { const queued = this.boundary.proposeApproval({ + kind: 'mcp', title: `${meta.tool} via ${meta.connector}`, detail: `Requested from chat. Arguments: ${JSON.stringify(args)}`, + risk, connectorId: meta.id, connector: meta.connector, tool: meta.tool, From 2aa94119666808cbf677a9e46737143ee815b771 Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Wed, 12 Aug 2026 12:59:26 +0530 Subject: [PATCH 09/75] feat(packaging): TCC usage strings for the computer-use semantic rail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 0 packaging groundwork. Adds the Info.plist usage-description keys the semantic action rail needs — Apple Events (Messages/Mail/Notes), Calendars (+ pre-14 legacy key), Reminders, Contacts, Photos — so a hardened-runtime build is granted each capability instead of being refused before the OS prompt. The apple-events entitlement was already present. Copy follows off-grid-ai/brand: outcome first, privacy as proof, no em dashes. A source-reading test pins every key + the entitlement and enforces the no-em-dash rule on the strings added here. Verified: 8 new tests, YAML parses, config-reading tests green, eslint clean. Co-Authored-By: Claude Fable 5 --- electron-builder.yml | 10 ++++ .../computer-use-entitlements.test.ts | 46 +++++++++++++++++++ 2 files changed, 56 insertions(+) create mode 100644 src/main/__tests__/computer-use-entitlements.test.ts diff --git a/electron-builder.yml b/electron-builder.yml index 57d547d3..eb75d7db 100644 --- a/electron-builder.yml +++ b/electron-builder.yml @@ -60,6 +60,16 @@ mac: - NSCameraUsageDescription: Off Grid AI may use the camera for on-device vision features. Frames are processed locally and never leave your device. - NSDocumentsFolderUsageDescription: Off Grid AI reads documents you add to a project so it can answer questions about them — locally, on your device. - NSDownloadsFolderUsageDescription: Off Grid AI reads files you add to a project so it can answer questions about them — locally, on your device. + # Computer use — the semantic action rail. Each key is the OS prompt shown the + # first time the agent uses that capability; without it a hardened-runtime build + # is refused access before any prompt. Copy follows off-grid-ai/brand: lead with + # what the user gets, privacy as the proof, no em dashes. + - NSAppleEventsUsageDescription: Off Grid AI controls apps like Messages, Mail, and Notes to carry out actions you approve. Everything runs on your Mac and nothing leaves the device. + - NSCalendarsFullAccessUsageDescription: Off Grid AI reads and creates calendar events when you ask, on your device. Your calendar never leaves your Mac. + - NSCalendarsUsageDescription: Off Grid AI reads and creates calendar events when you ask, on your device. Your calendar never leaves your Mac. + - NSRemindersFullAccessUsageDescription: Off Grid AI reads and creates reminders when you ask, on your device. Your reminders never leave your Mac. + - NSContactsUsageDescription: Off Grid AI reads your contacts to complete actions you ask for, like sending a message. Your contacts never leave your Mac. + - NSPhotoLibraryUsageDescription: Off Grid AI works with photos you ask it to, like picking one to send. Your photos never leave your Mac. notarize: true icon: resources/icon.png # NOTE: no afterSign re-sign hook. electron-builder already signs every nested diff --git a/src/main/__tests__/computer-use-entitlements.test.ts b/src/main/__tests__/computer-use-entitlements.test.ts new file mode 100644 index 00000000..418a9169 --- /dev/null +++ b/src/main/__tests__/computer-use-entitlements.test.ts @@ -0,0 +1,46 @@ +/** + * Packaging contract for computer use's semantic action rail. Each TCC usage + * string must survive in electron-builder.yml and the apple-events entitlement in + * the plist: a hardened-runtime build is refused the capability BEFORE any prompt + * when its Info.plist key is missing, so a dropped key is a silent, ship-breaking + * regression (the exact "half-built in the safe direction" failure the computer-use + * plan warns about). Guarded by reading the source, per CLAUDE.md contract guards. + */ +import fs from 'node:fs' +import path from 'node:path' +import { describe, expect, it } from 'vitest' + +const root = path.resolve(import.meta.dirname, '../../..') +const builder = fs.readFileSync(path.join(root, 'electron-builder.yml'), 'utf8') +const entitlements = fs.readFileSync(path.join(root, 'build/entitlements.mac.plist'), 'utf8') + +// The usage-description keys the semantic rail needs. Calendars and Reminders +// carry BOTH the macOS 14+ FullAccess key and the pre-14 legacy key, because the +// build advertises minimumSystemVersion 13.0. +const REQUIRED_USAGE_KEYS = [ + 'NSAppleEventsUsageDescription', + 'NSCalendarsFullAccessUsageDescription', + 'NSCalendarsUsageDescription', + 'NSRemindersFullAccessUsageDescription', + 'NSContactsUsageDescription', + 'NSPhotoLibraryUsageDescription' +] + +describe('computer-use packaging entitlements', () => { + it('declares the apple-events entitlement AppleScript needs under hardened runtime', () => { + expect(entitlements).toContain('com.apple.security.automation.apple-events') + }) + + it.each(REQUIRED_USAGE_KEYS)('carries a non-empty %s usage string', (key) => { + const match = builder.match(new RegExp(`${key}:\\s*(\\S.*)$`, 'm')) + expect(match, `${key} missing from electron-builder.yml extendInfo`).not.toBeNull() + expect(match?.[1]?.trim().length ?? 0).toBeGreaterThan(0) + }) + + it('keeps the computer-use usage strings free of em dashes (brand rule)', () => { + for (const key of REQUIRED_USAGE_KEYS) { + const line = builder.match(new RegExp(`${key}:.*$`, 'm'))?.[0] ?? '' + expect(line, `${key} uses an em dash; the brand voice bans it (use " - ")`).not.toContain('—') + } + }) +}) From 8f4cfc4d7ed16bc2639003ed8bc890803ca1f383 Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Wed, 12 Aug 2026 13:02:55 +0530 Subject: [PATCH 10/75] docs(computer-use): lock @offgrid/use package name (lead confirmed) Co-Authored-By: Claude Fable 5 --- docs/COMPUTER_USE.md | 2 +- docs/COMPUTER_USE_PLAN.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/COMPUTER_USE.md b/docs/COMPUTER_USE.md index daf48556..12d84533 100644 --- a/docs/COMPUTER_USE.md +++ b/docs/COMPUTER_USE.md @@ -169,7 +169,7 @@ The shipped-product template is Gemini Intelligence's UX, which matches our appr Status as of August 11, 2026 (details live in `COMPUTER_USE_PLAN.md`): 1. **Cortex model - narrowed, decided at install time.** GUI-Owl-1.5-8B-Instruct (working default) or Qwen3-VL-8B. Same Qwen3-VL architecture, so all engine and adapter work is identical for either; the pick is a phase 3 catalog decision and blocks nothing before that. -2. **Package name - proposed, pending lead confirm.** `@offgrid/use`, consumed as `file:../shared/packages/use` from a sibling checkout per the shared README (not a vendored copy under this repo's `packages/`). Confirm both with the lead in phase 0. +2. **Package name - decided (August 12).** `@offgrid/use`, consumed as `file:../shared/packages/use` from a sibling checkout per the shared README (not a vendored copy under this repo's `packages/`). 3. **v1 graph - decided.** Planner / Cortex / Executor / Reflector; Orchestrator and Summarizer are added when task length demands them. 4. **Approval UX - proposed.** Plan-level approval + live step view + hold-to-stop, with per-step gates on irreversible actions. Validate against real runs in phase 3. 5. **Eval harness - decided.** macOSWorld subset as a non-blocking sanity gate from phase 3. diff --git a/docs/COMPUTER_USE_PLAN.md b/docs/COMPUTER_USE_PLAN.md index fa4c1338..f4a13282 100644 --- a/docs/COMPUTER_USE_PLAN.md +++ b/docs/COMPUTER_USE_PLAN.md @@ -40,7 +40,7 @@ Where the browser phase came from (added August 12): the embedded agent-browser | Widen the approval seam | `mcp:proposeApproval` becomes transport-agnostic `actions:proposeApproval` carrying `{kind, title, detail, risk, args, source}`; per-executor `riskOf(name, args)` replaces the name regex; MCP extension migrates onto it | existing approval tests stay green (`mcp-connector-tool-extension.dbtest.ts`); new tests per risk class; pro executor updated in `desktop-pro`, submodule bumped | | Packaging groundwork | `com.apple.security.automation.apple-events` entitlement + `NSAppleEventsUsageDescription` + Calendars/Contacts/Photos usage keys in the electron-builder config | local packaged build per `local-build.local.md`; TCC prompts name the app | | Permission onboarding design | just-in-time grants only (the browser rail needs none; Accessibility/Automation prompt on first native-rail use, deep-linked to the right Settings pane) | design reviewed; no dead-end states | -| Shared repo setup | sibling clone of `off-grid-ai/shared` next to this repo (CLAUDE.md already assumes `../shared`); decide the consumption pattern for `@offgrid/use` - proposed: the README's `file:../shared/packages/use`, NOT a fifth vendored copy under `packages/` (confirm with the lead; makes a sibling checkout a build requirement on this branch) | checkout builds; decision recorded here | +| Shared repo setup | sibling clone of `off-grid-ai/shared` next to this repo (CLAUDE.md already assumes `../shared`); package name `@offgrid/use` and consumption via `file:../shared/packages/use` are decided (Aug 12) - NOT a vendored copy under `packages/`; a sibling checkout is now a build requirement on this branch | checkout builds | | Brand setup | sibling clone of `off-grid-ai/brand`; skim `DESIGN_PHILOSOPHY.md` + `brand_tone_voice.md` before any UI work | clone present | ## Phase 1 - semantic rail (Aug 17 - Aug 26) From 614b79f87b0e322f4d6308fecfc0fdd23bffd9cc Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Wed, 12 Aug 2026 13:11:59 +0530 Subject: [PATCH 11/75] feat(actions): native helper invoker + EventKit calendar backend Phase 1 semantic rail, first slice. Adds the single seam every native capability goes through: a Swift one-shot helper (scripts/actions-helper) that takes one JSON command, performs a scoped EventKit action, and prints one JSON line; and a Node invoker (runNativeAction) that resolves the binary packaged-vs-dev like ocr.ts and parses the reply. Handled failures (denied permission, bad args) are in-band { ok: false } results, never throws, so the tool loop has one shape to report. Backend covers calendar create + list to start; the switch and command namespacing leave reminders/contacts/photos as additive cases. Pure logic (contract, path resolution, response parsing) is split into native-helper-logic.ts and unit tested; the Swift compiles clean under swiftc 6.2 targeting macos13. Not yet wired to a tool or shipped in CI - that lands with the tool that calls it, so no dead binary ships early. Verified: tsc node, 11 logic tests, swiftc build, eslint clean. Co-Authored-By: Claude Fable 5 --- .gitignore | 1 + scripts/actions-helper/main.swift | 127 ++++++++++++++++++ scripts/build-actions-helper.sh | 12 ++ .../__tests__/native-helper-logic.test.ts | 105 +++++++++++++++ src/main/actions/native-helper-logic.ts | 85 ++++++++++++ src/main/actions/native-helper.ts | 64 +++++++++ 6 files changed, 394 insertions(+) create mode 100644 scripts/actions-helper/main.swift create mode 100755 scripts/build-actions-helper.sh create mode 100644 src/main/actions/__tests__/native-helper-logic.test.ts create mode 100644 src/main/actions/native-helper-logic.ts create mode 100644 src/main/actions/native-helper.ts diff --git a/.gitignore b/.gitignore index 0482c83d..634aa5fa 100644 --- a/.gitignore +++ b/.gitignore @@ -50,6 +50,7 @@ test-results/ # Marketing material (dev.to drafts, emails, assets) — publishing, not app code /marketing/ scripts/dictation-hotkey/dictation-hotkey +scripts/actions-helper/actions-helper # Local demo profile — synthetic-data run target for `npm run demo` (never real userData) .demo-profile/ diff --git a/scripts/actions-helper/main.swift b/scripts/actions-helper/main.swift new file mode 100644 index 00000000..989de8e6 --- /dev/null +++ b/scripts/actions-helper/main.swift @@ -0,0 +1,127 @@ +import Foundation +import EventKit + +// Off Grid AI Desktop - native actions helper (macOS), the backend of the computer-use +// semantic rail. One-shot CLI: reads a single JSON command argument, performs one +// scoped native action (EventKit today; Reminders / Contacts / Photos next), prints +// ONE compact JSON line to stdout, and exits 0. +// +// Handled errors are reported as {"ok":false,"error":...} inside that JSON, not via +// the exit code, so the Node invoker always reads the result from stdout and a +// permission denial is a normal result rather than a crash. Invoked as a child of the +// signed .app, the helper inherits the app's TCC identity, so the Info.plist usage +// strings (NSCalendarsFullAccessUsageDescription and friends) drive the OS prompts. + +func emit(_ object: [String: Any]) -> Never { + if let data = try? JSONSerialization.data(withJSONObject: object), + let json = String(data: data, encoding: .utf8) { + print(json) + } else { + print("{\"ok\":false,\"error\":\"failed to serialize response\"}") + } + exit(0) +} + +func fail(_ message: String) -> Never { emit(["ok": false, "error": message]) } +func ok(_ result: [String: Any]) -> Never { emit(["ok": true, "result": result]) } + +let iso = ISO8601DateFormatter() + +func parseDate(_ value: Any?) -> Date? { + guard let raw = value as? String else { return nil } + return iso.date(from: raw) +} + +// Request EventKit access synchronously. The completion handler runs off the calling +// thread, so block on it - this one-shot tool must have a decision before it can act. +func requestEventAccess(_ store: EKEventStore) -> (granted: Bool, error: String?) { + let semaphore = DispatchSemaphore(value: 0) + var granted = false + var errorMessage: String? + let handler: (Bool, Error?) -> Void = { allowed, err in + granted = allowed + if let err = err { errorMessage = err.localizedDescription } + semaphore.signal() + } + if #available(macOS 14.0, *) { + store.requestFullAccessToEvents(completion: handler) + } else { + store.requestAccess(to: .event, completion: handler) + } + semaphore.wait() + return (granted, errorMessage) +} + +func createEvent(_ args: [String: Any]) -> Never { + guard let title = args["title"] as? String, !title.isEmpty else { + fail("createEvent requires a non-empty title") + } + guard let start = parseDate(args["start"]) else { + fail("createEvent requires an ISO8601 start date") + } + let allDay = (args["allDay"] as? Bool) ?? false + let end = parseDate(args["end"]) ?? start.addingTimeInterval(3600) + + let store = EKEventStore() + let access = requestEventAccess(store) + if !access.granted { fail(access.error ?? "calendar access was not granted") } + + let event = EKEvent(eventStore: store) + event.title = title + event.startDate = start + event.endDate = end + event.isAllDay = allDay + if let notes = args["notes"] as? String { event.notes = notes } + if let calName = args["calendar"] as? String, + let cal = store.calendars(for: .event).first(where: { $0.title == calName }) { + event.calendar = cal + } else { + event.calendar = store.defaultCalendarForNewEvents + } + do { + try store.save(event, span: .thisEvent) + ok(["id": event.eventIdentifier ?? ""]) + } catch { + fail("failed to save event: \(error.localizedDescription)") + } +} + +func listEvents(_ args: [String: Any]) -> Never { + guard let start = parseDate(args["start"]), let end = parseDate(args["end"]) else { + fail("listEvents requires ISO8601 start and end dates") + } + let store = EKEventStore() + let access = requestEventAccess(store) + if !access.granted { fail(access.error ?? "calendar access was not granted") } + + let predicate = store.predicateForEvents(withStart: start, end: end, calendars: nil) + let events = store.events(matching: predicate).map { event -> [String: Any] in + [ + "id": event.eventIdentifier ?? "", + "title": event.title ?? "", + "start": iso.string(from: event.startDate), + "end": iso.string(from: event.endDate), + "allDay": event.isAllDay, + "calendar": event.calendar?.title ?? "" + ] + } + ok(["events": events]) +} + +let arguments = CommandLine.arguments +guard arguments.count >= 2 else { fail("no command provided") } +guard let data = arguments[1].data(using: .utf8), + let payload = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let command = payload["command"] as? String else { + fail("invalid command JSON") +} +let commandArgs = (payload["args"] as? [String: Any]) ?? [:] + +switch command { +case "calendar.createEvent": + createEvent(commandArgs) +case "calendar.listEvents": + listEvents(commandArgs) +default: + fail("unknown command: \(command)") +} diff --git a/scripts/build-actions-helper.sh b/scripts/build-actions-helper.sh new file mode 100755 index 00000000..79879fa4 --- /dev/null +++ b/scripts/build-actions-helper.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +# Compile the native actions helper (EventKit / Reminders / Contacts / Photos), the +# backend of the computer-use semantic rail. Output lands next to the source so dev +# mode finds it; CI copies it into resources/bin so extraResources bundles it at +# Contents/Resources/bin. Pinned to the same deployment target as every other bundled +# native binary (macOS 13) so it launches on the versions the app advertises. +set -euo pipefail +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SRC="$ROOT_DIR/actions-helper/main.swift" +OUT="$ROOT_DIR/actions-helper/actions-helper" +swiftc -O -target arm64-apple-macos13.0 -emit-executable "$SRC" -o "$OUT" +echo "built $OUT" diff --git a/src/main/actions/__tests__/native-helper-logic.test.ts b/src/main/actions/__tests__/native-helper-logic.test.ts new file mode 100644 index 00000000..c4fc46c1 --- /dev/null +++ b/src/main/actions/__tests__/native-helper-logic.test.ts @@ -0,0 +1,105 @@ +/** + * Unit tests for the native-helper invoker's pure logic. Guards the command/response + * contract the Swift helper (scripts/actions-helper/main.swift) and every semantic + * tool share, plus the packaged-vs-dev binary resolution that mirrors ocr.ts. The + * response parser must degrade every malformed shape to a reported { ok: false } so a + * broken helper never throws into the tool loop. + */ +import path from 'path' +import { describe, expect, it } from 'vitest' +import { serializeCommand, helperBinCandidates, parseHelperResponse } from '../native-helper-logic' + +describe('serializeCommand', () => { + it('encodes the command and args as a single JSON string', () => { + expect(serializeCommand({ command: 'calendar.createEvent', args: { title: 'Sync' } })).toBe( + '{"command":"calendar.createEvent","args":{"title":"Sync"}}' + ) + }) +}) + +describe('helperBinCandidates', () => { + it('prefers the bundled bin path in a packaged build', () => { + expect( + helperBinCandidates({ + isPackaged: true, + resourcesPath: '/App/Contents/Resources', + cwd: '/ignored', + appPath: '/ignored' + }) + ).toEqual([ + path.join('/App/Contents/Resources', 'bin', 'actions-helper'), + path.join('/App/Contents/Resources', 'actions-helper') + ]) + }) + + it('resolves next to the source in a dev build', () => { + expect( + helperBinCandidates({ + isPackaged: false, + resourcesPath: '/ignored', + cwd: '/repo', + appPath: '/app' + }) + ).toEqual([ + path.join('/repo', 'scripts', 'actions-helper', 'actions-helper'), + path.join('/app', 'scripts', 'actions-helper', 'actions-helper') + ]) + }) +}) + +describe('parseHelperResponse', () => { + it('parses a success response and preserves the result', () => { + expect(parseHelperResponse('{"ok":true,"result":{"id":"E1"}}')).toEqual({ + ok: true, + result: { id: 'E1' } + }) + }) + + it('parses an in-band error response', () => { + expect(parseHelperResponse('{"ok":false,"error":"calendar access was not granted"}')).toEqual({ + ok: false, + error: 'calendar access was not granted' + }) + }) + + it('reads the last non-empty line so a stray leading line does not break parsing', () => { + expect(parseHelperResponse('warming up\n\n{"ok":true,"result":null}\n')).toEqual({ + ok: true, + result: null + }) + }) + + it('reports empty output as an error rather than throwing', () => { + expect(parseHelperResponse(' \n ')).toEqual({ + ok: false, + error: 'actions helper returned no output' + }) + }) + + it('reports invalid JSON as an error and truncates the echoed text', () => { + const res = parseHelperResponse('not json at all') + expect(res.ok).toBe(false) + expect(res).toMatchObject({ error: expect.stringContaining('invalid JSON') }) + }) + + it('rejects a non-object JSON payload', () => { + expect(parseHelperResponse('42')).toEqual({ + ok: false, + error: 'actions helper returned a non-object response' + }) + }) + + it('rejects a recognized-shape-but-missing-ok payload', () => { + expect(parseHelperResponse('{"result":{"id":"E1"}}')).toEqual({ + ok: false, + error: 'actions helper returned an unrecognized response' + }) + }) + + it('substitutes a generic message when ok:false carries no error string', () => { + expect(parseHelperResponse('{"ok":false}')).toEqual({ + ok: false, + error: 'actions helper reported an error' + }) + }) +}) diff --git a/src/main/actions/native-helper-logic.ts b/src/main/actions/native-helper-logic.ts new file mode 100644 index 00000000..b05f4635 --- /dev/null +++ b/src/main/actions/native-helper-logic.ts @@ -0,0 +1,85 @@ +// Pure logic for the native actions helper invoker (no Electron, so it is unit +// testable). The Electron-bound wrapper in native-helper.ts resolves the binary and +// runs it; everything that can be reasoned about without spawning a process lives +// here: the command/response contract, binary-path candidates, and response parsing. + +import path from 'path' + +/** A command sent to the native helper: one namespaced action plus its arguments, + * e.g. { command: 'calendar.createEvent', args: { title, start, end } }. */ +export interface NativeActionCommand { + command: string + args: Record +} + +/** The helper's reply. It always exits 0 and reports handled failures in-band, so a + * denied permission or a bad argument is a normal { ok: false } result, not a throw. */ +export type NativeActionResponse = { ok: true; result: unknown } | { ok: false; error: string } + +export function serializeCommand(cmd: NativeActionCommand): string { + return JSON.stringify(cmd) +} + +export interface HelperPathContext { + isPackaged: boolean + resourcesPath: string + cwd: string + appPath: string +} + +/** Where the compiled helper can live, most-specific first. Packaged: bundled under + * Contents/Resources/bin (extraResources maps resources/ -> .). Dev: next to its + * source where build-actions-helper.sh emits it. Mirrors ocr.ts's resolution. */ +export function helperBinCandidates(ctx: HelperPathContext): string[] { + if (ctx.isPackaged) { + return [ + path.join(ctx.resourcesPath, 'bin', 'actions-helper'), + path.join(ctx.resourcesPath, 'actions-helper') + ] + } + return [ + path.join(ctx.cwd, 'scripts', 'actions-helper', 'actions-helper'), + path.join(ctx.appPath, 'scripts', 'actions-helper', 'actions-helper') + ] +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null +} + +function truncate(text: string): string { + return text.length > 200 ? `${text.slice(0, 200)}…` : text +} + +/** Parse the helper's stdout into a typed response. The helper prints one compact + * JSON line; we read the last non-empty line so a stray leading log line cannot + * break parsing. Any shape we do not recognize becomes an { ok: false } error + * rather than a throw, so a malformed helper degrades to a reported failure. */ +export function parseHelperResponse(stdout: string): NativeActionResponse { + const lines = stdout + .split('\n') + .map((l) => l.trim()) + .filter((l) => l.length > 0) + const last = lines[lines.length - 1] + if (!last) { + return { ok: false, error: 'actions helper returned no output' } + } + let parsed: unknown + try { + parsed = JSON.parse(last) + } catch { + return { ok: false, error: `actions helper returned invalid JSON: ${truncate(last)}` } + } + if (!isRecord(parsed)) { + return { ok: false, error: 'actions helper returned a non-object response' } + } + if (parsed.ok === true) { + return { ok: true, result: parsed.result } + } + if (parsed.ok === false) { + const error = + typeof parsed.error === 'string' ? parsed.error : 'actions helper reported an error' + return { ok: false, error } + } + return { ok: false, error: 'actions helper returned an unrecognized response' } +} diff --git a/src/main/actions/native-helper.ts b/src/main/actions/native-helper.ts new file mode 100644 index 00000000..d5365055 --- /dev/null +++ b/src/main/actions/native-helper.ts @@ -0,0 +1,64 @@ +// Electron-bound invoker for the native actions helper (macOS). Resolves the compiled +// helper binary and runs it as a one-shot child process, handing it one JSON command +// and parsing the one JSON line it prints back. This is the single seam every semantic +// native capability (calendar, reminders, contacts, photos) goes through, so the +// process/permission handling lives in one place. Mirrors ocr.ts. + +import { execFile } from 'child_process' +import { promisify } from 'util' +import fs from 'fs' +import { app } from 'electron' +import { + helperBinCandidates, + parseHelperResponse, + serializeCommand, + type NativeActionCommand, + type NativeActionResponse +} from './native-helper-logic' + +const execFileAsync = promisify(execFile) + +function helperBin(): string | null { + const candidates = helperBinCandidates({ + isPackaged: app.isPackaged, + resourcesPath: process.resourcesPath, + cwd: process.cwd(), + appPath: app.getAppPath() + }) + for (const candidate of candidates) { + try { + if (fs.existsSync(candidate)) { + return candidate + } + } catch { + /* ignore */ + } + } + return null +} + +/** Run one native action. Never throws: a missing helper, a spawn failure, a timeout, + * or a handled in-band error all resolve to an { ok: false } response so callers + * (tools, the approval executor) have a single shape to report. */ +export async function runNativeAction(cmd: NativeActionCommand): Promise { + const bin = helperBin() + if (!bin) { + return { ok: false, error: 'the native actions helper is not available in this build' } + } + try { + const { stdout } = await execFileAsync(bin, [serializeCommand(cmd)], { + maxBuffer: 8 * 1024 * 1024, + timeout: 20_000 + }) + return parseHelperResponse(stdout) + } catch (e) { + // execFile rejects on a non-zero exit, a timeout, or a spawn failure. The helper + // exits 0 even on handled errors, so reaching here means the process itself failed + // - but it may still have printed a response before dying, so prefer that. + const stdout = (e as { stdout?: string }).stdout + if (typeof stdout === 'string' && stdout.trim().length > 0) { + return parseHelperResponse(stdout) + } + return { ok: false, error: (e as Error).message } + } +} From acb0f0a062b9e8d55dd3d0ded0a9973f9f3a5137 Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Wed, 12 Aug 2026 13:18:21 +0530 Subject: [PATCH 12/75] feat(actions): wire calendar tools into the chat loop, gated + shipped Phase 1 semantic rail, made reachable. Registers a native-action tool extension (macOS-only) exposing calendar_create_event and calendar_list_events to the model; create is a mutate that offers itself to the shared approval seam (queues in pro, runs in the free build), list is a read that runs directly. Both route through runNativeAction to the EventKit helper. Adds 'native' to ActionKind so the approval UI and audit can label semantic OS actions apart from GUI computer use. The release workflow now builds + stages the helper into resources/bin, so a packaged build ships it (self-contained; if it fails the tools report 'not available' and nothing else breaks). Boundary injection mirrors the MCP extension, so the gate-then-run contract is unit tested end to end without a real EventKit call. Verified: tsc node, 90 tests across actions+tools+hooks, release.yml parses, eslint clean on new files. Co-Authored-By: Claude Fable 5 --- .github/workflows/release.yml | 11 ++ src/main/actions/approval.ts | 8 +- src/main/index.ts | 3 + .../nativeActionToolExtension-logic.test.ts | 48 +++++++ .../nativeActionToolExtension.test.ts | 127 ++++++++++++++++++ .../tools/nativeActionToolExtension-logic.ts | 100 ++++++++++++++ src/main/tools/nativeActionToolExtension.ts | 89 ++++++++++++ 7 files changed, 384 insertions(+), 2 deletions(-) create mode 100644 src/main/tools/__tests__/nativeActionToolExtension-logic.test.ts create mode 100644 src/main/tools/__tests__/nativeActionToolExtension.test.ts create mode 100644 src/main/tools/nativeActionToolExtension-logic.ts create mode 100644 src/main/tools/nativeActionToolExtension.ts diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 45d60443..94cdd884 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -160,6 +160,17 @@ jobs: mkdir -p resources/bin cp scripts/dictation-hotkey/dictation-hotkey resources/bin/dictation-hotkey chmod +x resources/bin/dictation-hotkey + # Compile the native actions helper (EventKit) and stage it into resources/bin so + # extraResources bundles it at Contents/Resources/bin — the path runNativeAction + # resolves. Self-contained: no committed binary, built fresh against the pinned + # target. If it ever fails to ship, the calendar tools report "not available" and + # the rest of the app is unaffected, so it can't break a release. + - name: Build native actions helper (computer use, semantic rail) + run: | + bash scripts/build-actions-helper.sh + mkdir -p resources/bin + cp scripts/actions-helper/actions-helper resources/bin/actions-helper + chmod +x resources/bin/actions-helper # Stage the Parakeet STT runtime (sherpa-onnx CLI + ONNX model) into # resources/bin/parakeet. Additive: with SHERPA_ONNX_URL / PARAKEET_MODEL_URL # unset this is a no-op and transcription stays on whisper, so it can't break a diff --git a/src/main/actions/approval.ts b/src/main/actions/approval.ts index dd196a49..c7af5d0b 100644 --- a/src/main/actions/approval.ts +++ b/src/main/actions/approval.ts @@ -22,8 +22,12 @@ import { callHook, hasHook, HOOKS } from '../bootstrap/hookRegistry' export type ActionRisk = 'read' | 'navigate' | 'mutate' | 'irreversible' /** Which executor raised the action — lets the approval UI and audit log group and - * label without branching on executor-specific fields. */ -export type ActionKind = 'mcp' | 'computer' | 'browser' + * label without branching on executor-specific fields. + * - mcp: a connector tool call + * - native: a semantic OS action (EventKit, AppleScript, Shortcuts) — the rail-1 path + * - browser: the embedded agent browser + * - computer: GUI automation (accessibility tree + synthetic input) */ +export type ActionKind = 'mcp' | 'native' | 'browser' | 'computer' export interface ActionApprovalRequest { kind: ActionKind diff --git a/src/main/index.ts b/src/main/index.ts index ceebdc46..85e358f2 100644 --- a/src/main/index.ts +++ b/src/main/index.ts @@ -19,6 +19,8 @@ import icon from '../../resources/icon.png?asset' import { setupIPC } from './ipc' // IMPORT FROM IPC ONLY import { setupRagIPC } from './rag-ipc' import { setupMcpIpc } from './mcp-ipc' +import { registerToolExtension } from './tools' +import { registerNativeActionTools } from './tools/nativeActionToolExtension' import { preloadPath } from './preload-path' import { rendererHtmlPath } from './renderer-path' import { startModelServer, stopModelServer } from './model-server' @@ -327,6 +329,7 @@ app.whenReady().then(() => { setupIPC() setupRagIPC() setupMcpIpc() // basic MCP connectors (management + chat tool extension) + registerNativeActionTools(registerToolExtension) // computer use: semantic rail (macOS-only) // one OpenAI-compatible local gateway (LLM + STT); auto-picks a free port. Async, so handle a // rejection on the promise (a try/catch around a fire-and-forget async call can't catch it). startModelServer().catch((e) => console.error('[model-server] start failed', e)) diff --git a/src/main/tools/__tests__/nativeActionToolExtension-logic.test.ts b/src/main/tools/__tests__/nativeActionToolExtension-logic.test.ts new file mode 100644 index 00000000..aaaf2a78 --- /dev/null +++ b/src/main/tools/__tests__/nativeActionToolExtension-logic.test.ts @@ -0,0 +1,48 @@ +import { describe, expect, it } from 'vitest' +import { + NATIVE_TOOL_SPECS, + findNativeToolSpec, + buildNativeToolSchemas +} from '../nativeActionToolExtension-logic' +import { shouldGate } from '../../actions/approval' + +describe('native tool specs', () => { + it('exposes calendar create and list with matching helper commands', () => { + expect(NATIVE_TOOL_SPECS.map((s) => s.name)).toEqual([ + 'calendar_create_event', + 'calendar_list_events' + ]) + expect(findNativeToolSpec('calendar_create_event')?.command).toBe('calendar.createEvent') + expect(findNativeToolSpec('calendar_list_events')?.command).toBe('calendar.listEvents') + }) + + it('returns undefined for an unknown tool name', () => { + expect(findNativeToolSpec('calendar_delete_everything')).toBeUndefined() + }) + + it('gates the mutating create tool and runs the read-only list tool freely', () => { + expect(shouldGate(findNativeToolSpec('calendar_create_event')!.risk)).toBe(true) + expect(shouldGate(findNativeToolSpec('calendar_list_events')!.risk)).toBe(false) + }) + + it('builds an approval title from the event title', () => { + expect(findNativeToolSpec('calendar_create_event')!.title({ title: 'Sync with Ali' })).toBe( + 'Create the calendar event "Sync with Ali"' + ) + }) + + it('formats a create result into a confirmation, with and without an id', () => { + const spec = findNativeToolSpec('calendar_create_event')! + expect(spec.formatResult({ id: 'E1' })).toBe('Created the calendar event (id E1).') + expect(spec.formatResult({})).toBe('Created the calendar event.') + }) + + it('builds OpenAI function schemas for every spec', () => { + const schemas = buildNativeToolSchemas() + expect(schemas).toHaveLength(NATIVE_TOOL_SPECS.length) + expect(schemas[0]).toMatchObject({ + type: 'function', + function: { name: 'calendar_create_event', parameters: { required: ['title', 'start'] } } + }) + }) +}) diff --git a/src/main/tools/__tests__/nativeActionToolExtension.test.ts b/src/main/tools/__tests__/nativeActionToolExtension.test.ts new file mode 100644 index 00000000..80387aa3 --- /dev/null +++ b/src/main/tools/__tests__/nativeActionToolExtension.test.ts @@ -0,0 +1,127 @@ +/** + * Execute-path tests for the native-action tool extension against a fake boundary + * (the same injection seam the MCP extension uses). Pins the gate-then-run contract: + * a mutating tool queues for approval and does NOT run when queued, runs directly when + * nothing gates it (free build), and a read tool never gates. Platform registration is + * asserted so the tools stay out of the grammar budget off macOS. + */ +import { describe, expect, it, beforeEach } from 'vitest' +import { + NativeActionToolExtension, + registerNativeActionTools, + type NativeActionToolBoundary +} from '../nativeActionToolExtension' +import type { ToolExtension } from '../../tools' +import type { ActionApprovalRequest } from '../../actions/approval' +import type { NativeActionCommand, NativeActionResponse } from '../../actions/native-helper-logic' + +class FakeBoundary implements NativeActionToolBoundary { + readonly commands: NativeActionCommand[] = [] + readonly approvals: ActionApprovalRequest[] = [] + queueApprovals = false + response: NativeActionResponse = { ok: true, result: { id: 'E1' } } + + async run(cmd: NativeActionCommand): Promise { + this.commands.push(cmd) + return this.response + } + + proposeApproval(request: ActionApprovalRequest): boolean { + this.approvals.push(request) + return this.queueApprovals + } +} + +let boundary: FakeBoundary +let ext: NativeActionToolExtension + +beforeEach(() => { + boundary = new FakeBoundary() + ext = new NativeActionToolExtension(boundary) +}) + +describe('NativeActionToolExtension', () => { + it('owns only its known tool names', () => { + expect(ext.canHandle('calendar_create_event')).toBe(true) + expect(ext.canHandle('calendar_list_events')).toBe(true) + expect(ext.canHandle('mcp__1__send')).toBe(false) + }) + + it('queues a create for approval and does not run the helper when queued', async () => { + boundary.queueApprovals = true + const out = await ext.execute('calendar_create_event', { + title: 'Sync', + start: '2026-08-13T15:00:00' + }) + + expect(out).toContain('Queued for the user') + expect(boundary.approvals).toEqual([ + expect.objectContaining({ + kind: 'native', + risk: 'mutate', + command: 'calendar.createEvent', + args: { title: 'Sync', start: '2026-08-13T15:00:00' } + }) + ]) + expect(boundary.commands).toEqual([]) + }) + + it('runs a create directly when nothing gates it (free build)', async () => { + boundary.queueApprovals = false + const out = await ext.execute('calendar_create_event', { + title: 'Sync', + start: '2026-08-13T15:00:00' + }) + + expect(out).toBe('Created the calendar event (id E1).') + expect(boundary.approvals).toHaveLength(1) // it was offered + expect(boundary.commands).toEqual([ + { command: 'calendar.createEvent', args: { title: 'Sync', start: '2026-08-13T15:00:00' } } + ]) + }) + + it('runs a read tool without ever offering it for approval', async () => { + boundary.response = { ok: true, result: { events: [] } } + const out = await ext.execute('calendar_list_events', { + start: '2026-08-13T00:00:00', + end: '2026-08-14T00:00:00' + }) + + expect(out).toBe('{"events":[]}') + expect(boundary.approvals).toEqual([]) + expect(boundary.commands).toEqual([ + { + command: 'calendar.listEvents', + args: { start: '2026-08-13T00:00:00', end: '2026-08-14T00:00:00' } + } + ]) + }) + + it('passes a helper failure back as an error string', async () => { + boundary.response = { ok: false, error: 'calendar access was not granted' } + expect(await ext.execute('calendar_list_events', { start: 'a', end: 'b' })).toBe( + 'Error: calendar access was not granted' + ) + }) + + it('rejects an unknown tool name', async () => { + expect(await ext.execute('calendar_delete_all', {})).toBe( + 'Error: unknown action calendar_delete_all' + ) + }) +}) + +describe('registerNativeActionTools', () => { + it('registers the extension on macOS', () => { + const registered: ToolExtension[] = [] + registerNativeActionTools((e) => registered.push(e), 'darwin') + expect(registered.map((e) => e.id)).toEqual(['native-actions']) + }) + + it('registers nothing off macOS', () => { + const registered: ToolExtension[] = [] + registerNativeActionTools((e) => registered.push(e), 'win32') + registerNativeActionTools((e) => registered.push(e), 'linux') + expect(registered).toEqual([]) + }) +}) diff --git a/src/main/tools/nativeActionToolExtension-logic.ts b/src/main/tools/nativeActionToolExtension-logic.ts new file mode 100644 index 00000000..9ce09663 --- /dev/null +++ b/src/main/tools/nativeActionToolExtension-logic.ts @@ -0,0 +1,100 @@ +// Pure logic for the native-action tool extension: the table of semantic tools the +// model can call (calendar today; reminders / contacts / photos add as rows), plus +// schema building, risk classification, argument mapping, and result formatting. No +// Electron or process I/O here, so it is unit testable; the extension shell wires it +// to runNativeAction + the approval seam. + +import type { ActionRisk } from '../actions/approval' + +export interface NativeToolSpec { + /** Model-facing tool name. */ + name: string + /** Model-facing description (kept plain, no marketing voice — this is a prompt). */ + description: string + /** JSON schema for the tool's arguments. */ + parameters: Record + /** The native helper command this tool invokes. */ + command: string + /** How consequential the action is — decides whether it gates for approval. */ + risk: ActionRisk + /** Map the model's tool arguments to the helper command's args. */ + buildArgs: (toolArgs: Record) => Record + /** One-line, user-facing approval-card title for a gated action. */ + title: (toolArgs: Record) => string + /** Turn a successful helper result into a string for the model. */ + formatResult: (result: unknown) => string +} + +function asString(value: unknown, fallback = ''): string { + return typeof value === 'string' ? value : fallback +} + +export const NATIVE_TOOL_SPECS: NativeToolSpec[] = [ + { + name: 'calendar_create_event', + description: + "Create an event in the user's macOS Calendar. Times are ISO 8601 (e.g. 2026-08-13T15:00:00). Needs the user to approve before it is written.", + parameters: { + type: 'object', + properties: { + title: { type: 'string', description: 'Event title' }, + start: { type: 'string', description: 'Start time, ISO 8601' }, + end: { + type: 'string', + description: 'End time, ISO 8601. Defaults to one hour after start.' + }, + notes: { type: 'string', description: 'Optional notes for the event' }, + allDay: { type: 'boolean', description: 'Whether the event lasts all day' }, + calendar: { type: 'string', description: 'Calendar name; defaults to the default calendar' } + }, + required: ['title', 'start'] + }, + command: 'calendar.createEvent', + risk: 'mutate', + buildArgs: (a) => a, + title: (a) => `Create the calendar event "${asString(a.title, 'Untitled')}"`, + formatResult: (result) => { + const id = + typeof result === 'object' && result !== null + ? asString((result as Record).id) + : '' + return id ? `Created the calendar event (id ${id}).` : 'Created the calendar event.' + } + }, + { + name: 'calendar_list_events', + description: + "List the user's macOS Calendar events between two ISO 8601 times. Read-only; runs without approval.", + parameters: { + type: 'object', + properties: { + start: { type: 'string', description: 'Range start, ISO 8601' }, + end: { type: 'string', description: 'Range end, ISO 8601' } + }, + required: ['start', 'end'] + }, + command: 'calendar.listEvents', + risk: 'read', + buildArgs: (a) => a, + title: (a) => `List calendar events from ${asString(a.start)} to ${asString(a.end)}`, + formatResult: (result) => JSON.stringify(result) + } +] + +const specsByName = new Map(NATIVE_TOOL_SPECS.map((s) => [s.name, s])) + +export function findNativeToolSpec(name: string): NativeToolSpec | undefined { + return specsByName.get(name) +} + +export interface NativeToolSchema { + type: 'function' + function: { name: string; description: string; parameters: Record } +} + +export function buildNativeToolSchemas(): NativeToolSchema[] { + return NATIVE_TOOL_SPECS.map((s) => ({ + type: 'function', + function: { name: s.name, description: s.description, parameters: s.parameters } + })) +} diff --git a/src/main/tools/nativeActionToolExtension.ts b/src/main/tools/nativeActionToolExtension.ts new file mode 100644 index 00000000..f584cc3e --- /dev/null +++ b/src/main/tools/nativeActionToolExtension.ts @@ -0,0 +1,89 @@ +// Native semantic actions as a chat tool extension (core, macOS). Registered into the +// chat tool loop via registerToolExtension. Exposes calendar (and, as rows are added, +// reminders / contacts / photos) as model tools that run through the native actions +// helper, with mutating tools gated through the shared approval seam - the same +// open-core seam the MCP extension uses. Free build: no approval hook, so a write runs +// directly. Pro: the write queues for approval and the pro executor runs it on approve. + +import type { ToolExtension } from '../tools' +import { proposeActionApproval, shouldGate, type ActionApprovalRequest } from '../actions/approval' +import { runNativeAction } from '../actions/native-helper' +import type { NativeActionCommand, NativeActionResponse } from '../actions/native-helper-logic' +import { + buildNativeToolSchemas, + findNativeToolSpec, + NATIVE_TOOL_SPECS +} from './nativeActionToolExtension-logic' + +export interface NativeActionToolBoundary { + run: (cmd: NativeActionCommand) => Promise + proposeApproval: (request: ActionApprovalRequest) => boolean | undefined +} + +const productionBoundary: NativeActionToolBoundary = { + run: runNativeAction, + proposeApproval: proposeActionApproval +} + +export class NativeActionToolExtension implements ToolExtension { + id = 'native-actions' + + constructor(private readonly boundary: NativeActionToolBoundary = productionBoundary) {} + + schemas(): unknown[] { + return buildNativeToolSchemas() + } + + canHandle(name: string): boolean { + return findNativeToolSpec(name) !== undefined + } + + systemHint(): string { + return "You can act on the user's Mac: create and read calendar events with calendar_create_event and calendar_list_events. Use ISO 8601 for all times. Creating an event needs the user's approval; tell them it is pending until they approve." + } + + async execute(name: string, args: Record): Promise { + const spec = findNativeToolSpec(name) + if (!spec) { + return `Error: unknown action ${name}` + } + // Mutating actions offer themselves for approval first; pro queues them. + if (shouldGate(spec.risk)) { + const queued = this.boundary.proposeApproval({ + kind: 'native', + title: spec.title(args), + detail: `Requested from chat. Arguments: ${JSON.stringify(args)}`, + risk: spec.risk, + command: spec.command, + args, + source: 'chat' + }) + if (queued) { + return `Queued for the user's approval — ${spec.title(args)} will run only after they approve it. Do not assume it has happened; tell the user it's pending approval.` + } + } + const res = await this.boundary.run({ command: spec.command, args: spec.buildArgs(args) }) + if (!res.ok) { + return `Error: ${res.error}` + } + return spec.formatResult(res.result) + } +} + +export const nativeActionToolExtension = new NativeActionToolExtension() + +/** Register the native-action tools. macOS-only: the helper is an EventKit binary and + * simply reports "not available" elsewhere, so gate registration on the platform to + * keep the tools out of the grammar budget where they cannot work. */ +export function registerNativeActionTools( + register: (ext: ToolExtension) => void, + platform: NodeJS.Platform = process.platform +): void { + if (platform !== 'darwin') { + return + } + if (NATIVE_TOOL_SPECS.length === 0) { + return + } + register(nativeActionToolExtension) +} From 45b7283fbc553ac3eb557195041564c48011876c Mon Sep 17 00:00:00 2001 From: siddWednesday Date: Wed, 12 Aug 2026 13:22:36 +0530 Subject: [PATCH 13/75] feat(actions): add Reminders to the semantic rail reminders_create (mutate, gated) and reminders_list (read) alongside the calendar tools, via EventKit reminder access in the helper. Extracts a shared formatCreated() so each create tool reuses one confirmation shape instead of re-encoding it. Verified: swiftc build, tsc node, 16 tool tests, eslint clean. Co-Authored-By: Claude Fable 5 --- scripts/actions-helper/main.swift | 70 +++++++++++++++++++ .../nativeActionToolExtension-logic.test.ts | 23 +++++- .../tools/nativeActionToolExtension-logic.ts | 49 +++++++++++-- src/main/tools/nativeActionToolExtension.ts | 2 +- 4 files changed, 134 insertions(+), 10 deletions(-) diff --git a/scripts/actions-helper/main.swift b/scripts/actions-helper/main.swift index 989de8e6..44e7247d 100644 --- a/scripts/actions-helper/main.swift +++ b/scripts/actions-helper/main.swift @@ -86,6 +86,72 @@ func createEvent(_ args: [String: Any]) -> Never { } } +// Reminders share EKEventStore with calendar but need their own access grant. +func requestReminderAccess(_ store: EKEventStore) -> (granted: Bool, error: String?) { + let semaphore = DispatchSemaphore(value: 0) + var granted = false + var errorMessage: String? + let handler: (Bool, Error?) -> Void = { allowed, err in + granted = allowed + if let err = err { errorMessage = err.localizedDescription } + semaphore.signal() + } + if #available(macOS 14.0, *) { + store.requestFullAccessToReminders(completion: handler) + } else { + store.requestAccess(to: .reminder, completion: handler) + } + semaphore.wait() + return (granted, errorMessage) +} + +func createReminder(_ args: [String: Any]) -> Never { + guard let title = args["title"] as? String, !title.isEmpty else { + fail("createReminder requires a non-empty title") + } + let store = EKEventStore() + let access = requestReminderAccess(store) + if !access.granted { fail(access.error ?? "reminders access was not granted") } + + let reminder = EKReminder(eventStore: store) + reminder.title = title + reminder.calendar = store.defaultCalendarForNewReminders() + if let notes = args["notes"] as? String { reminder.notes = notes } + if let due = parseDate(args["due"]) { + reminder.dueDateComponents = Calendar.current.dateComponents( + [.year, .month, .day, .hour, .minute], from: due) + } + do { + try store.save(reminder, commit: true) + ok(["id": reminder.calendarItemIdentifier]) + } catch { + fail("failed to save reminder: \(error.localizedDescription)") + } +} + +func listReminders(_ args: [String: Any]) -> Never { + let store = EKEventStore() + let access = requestReminderAccess(store) + if !access.granted { fail(access.error ?? "reminders access was not granted") } + + let predicate = store.predicateForIncompleteReminders( + withDueDateStarting: nil, ending: nil, calendars: nil) + let semaphore = DispatchSemaphore(value: 0) + var out: [[String: Any]] = [] + store.fetchReminders(matching: predicate) { reminders in + for reminder in reminders ?? [] { + var item: [String: Any] = ["id": reminder.calendarItemIdentifier, "title": reminder.title ?? ""] + if let due = reminder.dueDateComponents, let date = Calendar.current.date(from: due) { + item["due"] = iso.string(from: date) + } + out.append(item) + } + semaphore.signal() + } + semaphore.wait() + ok(["reminders": out]) +} + func listEvents(_ args: [String: Any]) -> Never { guard let start = parseDate(args["start"]), let end = parseDate(args["end"]) else { fail("listEvents requires ISO8601 start and end dates") @@ -122,6 +188,10 @@ case "calendar.createEvent": createEvent(commandArgs) case "calendar.listEvents": listEvents(commandArgs) +case "reminders.create": + createReminder(commandArgs) +case "reminders.list": + listReminders(commandArgs) default: fail("unknown command: \(command)") } diff --git a/src/main/tools/__tests__/nativeActionToolExtension-logic.test.ts b/src/main/tools/__tests__/nativeActionToolExtension-logic.test.ts index aaaf2a78..25a8922e 100644 --- a/src/main/tools/__tests__/nativeActionToolExtension-logic.test.ts +++ b/src/main/tools/__tests__/nativeActionToolExtension-logic.test.ts @@ -7,13 +7,32 @@ import { import { shouldGate } from '../../actions/approval' describe('native tool specs', () => { - it('exposes calendar create and list with matching helper commands', () => { + it('exposes calendar and reminder tools with matching helper commands', () => { expect(NATIVE_TOOL_SPECS.map((s) => s.name)).toEqual([ 'calendar_create_event', - 'calendar_list_events' + 'calendar_list_events', + 'reminders_create', + 'reminders_list' ]) expect(findNativeToolSpec('calendar_create_event')?.command).toBe('calendar.createEvent') expect(findNativeToolSpec('calendar_list_events')?.command).toBe('calendar.listEvents') + expect(findNativeToolSpec('reminders_create')?.command).toBe('reminders.create') + expect(findNativeToolSpec('reminders_list')?.command).toBe('reminders.list') + }) + + it('classifies every create tool as a gating mutate and every list tool as a read', () => { + for (const name of ['calendar_create_event', 'reminders_create']) { + expect(shouldGate(findNativeToolSpec(name)!.risk)).toBe(true) + } + for (const name of ['calendar_list_events', 'reminders_list']) { + expect(shouldGate(findNativeToolSpec(name)!.risk)).toBe(false) + } + }) + + it('formats a created reminder with the shared confirmation shape', () => { + expect(findNativeToolSpec('reminders_create')!.formatResult({ id: 'R1' })).toBe( + 'Created the reminder (id R1).' + ) }) it('returns undefined for an unknown tool name', () => { diff --git a/src/main/tools/nativeActionToolExtension-logic.ts b/src/main/tools/nativeActionToolExtension-logic.ts index 9ce09663..755a328b 100644 --- a/src/main/tools/nativeActionToolExtension-logic.ts +++ b/src/main/tools/nativeActionToolExtension-logic.ts @@ -29,6 +29,18 @@ function asString(value: unknown, fallback = ''): string { return typeof value === 'string' ? value : fallback } +/** Shared "Created the