Add semantic act-wait-observe execution

CHANGELOG.md

@@ -1,6 +1,7 @@
 # Changelog
 
 ## Unreleased
+- Added a compatibility-preserving semantic execution loop for AgentTab v2. Accessibility snapshots now include `semantic_ref` for uniquely named actionable roles; actions resolve those refs against the live accessibility tree so same-document SPA rerenders do not stale an otherwise stable target, while missing or newly ambiguous targets fail before execution with bounded recovery candidates. `browser_wait` now wakes from Chrome tab events, CDP network/download events, or a page MutationObserver with a 500 ms ownership/policy heartbeat instead of polling every 100 ms. TypeScript `actWaitObserve` and Python `act_wait_observe` helpers compose the existing act, deterministic wait, and fresh snapshot methods without changing Core RPC v1.
 - Replaced the Chrome Bridge v1 runtime with the AgentTab 2.0 release candidate: a Rust production host over OS-native local IPC, seven task-scoped Standard methods, explicit resumable capabilities, a developer-only eighth method, TypeScript and Python SDKs, MCP and OMP adapters, a transactional installer, and a minimal extension. Consequential controls now use a two-party Commit flow: `browser_act` stages an exact effect, the popup approves the durable review record without executing it, and the requesting task must consume its private one-use token through `browser_commit`.
 - Added task-owned background window creation to Standard `browser_open` through `placement: "new_window"`. It is operation-specific rather than a general window-control grant: only an otherwise empty task can request it, the extension creates an unfocused normal window, `background: false` is rejected, ownership is derived from persisted task state, and a failed visible group grant removes the new tab. Focus, state changes, and closure of unrelated windows remain unavailable.
 - Added explicit restricted-origin routing for task tabs. `browser_open` and `browser_tabs` now report `automation_route: "full" | "tab_only"`; Chrome system pages, extension pages, DevTools, and the Chrome Web Store retain task-owned explicit navigation, reload, close, and bounded waits, while page inspection, interaction, and raw CDP fail before execution with the stable `browser_restricted_origin` / `not_started` result and non-retry recovery. History movement remains available without managed origin constraints and fails closed when constraints are configured because Chrome does not expose its destination for pre-navigation authorization.

README.md

@@ -23,7 +23,7 @@ The command has no path, token, or shell-specific argument and is suitable for P
 ## A task-owned workflow
 
 1. An agent calls `browser_open` with `mode: "create"`. AgentTab creates a background tab for that task and returns its task, tab, window, page-revision, and automation-route identifiers. `placement: "new_window"` may create the task's first tab in a separate unfocused normal window.
-2. On a normal web origin, the agent calls `browser_snapshot`, works from revisioned accessibility references, then calls `browser_act` with the expected page revision. It cannot act on unrelated tabs.
+2. On a normal web origin, the agent calls `browser_snapshot`, prefers a unique accessibility `semantic_ref`, then calls `browser_act` with the expected page revision. AgentTab resolves the semantic target against the live accessibility tree, waits on browser/page events rather than requiring arbitrary sleeps, and rejects ambiguous targets with bounded candidates. It cannot act on unrelated tabs.
 3. If a site requires human-only input, the agent calls `browser_handoff`. AgentTab focuses that tab, pauses automation, and blocks browser observation until the declared completion condition or **I'm done**.
 4. If AgentTab recognizes a send, publish, purchase, delete, upload, authorization, or permission-grant control, `browser_act` can return `commit_required`. The extension shows the staged effect in its popup. A human must approve it there before the agent can call `browser_commit` with the one-use staged token.
 5. The task can list only its own tabs with `browser_tabs`. A separate client gets a separate task unless it proves its durable resume capability.
@@ -111,6 +111,7 @@ Generated extension assets live only in `packages/extension/dist/`; they are not
 - [Setup and local paths](docs/setup.md)
 - [Command reference](docs/commands.md)
 - [MCP adapter and Core RPC](docs/mcp.md)
+- [Semantic automation and deterministic waits](docs/semantic-automation.md)
 - [Security and trust boundary](docs/security.md)
 - [Multi-agent behavior](docs/multi-agent.md)
 - [Runtime architecture decision](docs/adr/0001-agenttab-runtime.md)

docs/mcp.md

@@ -63,15 +63,17 @@ For MCP, the capability store namespace is `mcp`; OMP uses `omp`; Pi uses `pi`.
 | Tool | Required input and behavior |
 |---|---|
 | `browser_open` | `mode: "create"` optionally accepts an `http`, `https`, or `about` URL, `background`, and `placement`. The default `placement: "task"` creates a tab in the task's existing window when possible. `placement: "new_window"` creates the first tab of an otherwise empty task in a separate unfocused normal window and rejects `background: false`. `mode: "adopt_active"` explicitly adopts only the currently active tab. The result includes task, tab, window, page-revision, and `automation_route` identifiers. |
-| `browser_snapshot` | Requires `tab_id`. Modes are `accessibility`, `text`, `html`, and `screenshot`. Only accessibility snapshots return revisioned node references. Screenshot requests may select `png`, `jpeg`, or `webp`, set JPEG/WebP `quality`, bound dimensions with `max_width`/`max_height`, and cap compressed image size with `max_bytes`. Snapshots require the `full` automation route. |
-| `browser_act` | Requires `tab_id`, `expected_page_revision`, and one to 64 typed actions. Actions are click, type, fill, select, scroll, drag, navigate, history movement, reload, close, dialog decision, and staged file upload. No coordinate action exists in Standard mode. A `tab_only` route accepts explicit navigation, history movement, reload, and close only. Managed origin constraints disable history movement because Chrome does not expose its destination for authorization before navigation; use explicit navigation to an allowed URL instead. |
-| `browser_wait` | Requires `tab_id` and one load, URL, text, selector, network-idle, or download condition. `timeout_ms` is at most 120 seconds. A `tab_only` route accepts load and URL conditions only; network-idle and download attribution require the tab-scoped debugger connection available on the `full` route. |
+| `browser_snapshot` | Requires `tab_id`. Modes are `accessibility`, `text`, `html`, and `screenshot`. Accessibility nodes retain their revisioned `ref`; uniquely named actionable nodes also return a stable `semantic_ref` that can be passed anywhere a ref is accepted. Screenshot requests may select `png`, `jpeg`, or `webp`, set JPEG/WebP `quality`, bound dimensions with `max_width`/`max_height`, and cap compressed image size with `max_bytes`. Snapshots require the `full` automation route. |
+| `browser_act` | Requires `tab_id`, `expected_page_revision`, and one to 64 typed actions. Actions are click, type, fill, select, scroll, drag, navigate, history movement, reload, close, dialog decision, and staged file upload. Prefer an available `semantic_ref`: AgentTab resolves its accessibility role and name against the live tree, so an SPA can replace the DOM node without invalidating the target. If the target disappears or becomes ambiguous, the action does not start and the error returns recovery plus bounded candidate refs. No coordinate action exists in Standard mode. A `tab_only` route accepts explicit navigation, history movement, reload, and close only. Managed origin constraints disable history movement because Chrome does not expose its destination for authorization before navigation; use explicit navigation to an allowed URL instead. |
+| `browser_wait` | Requires `tab_id` and one load, URL, text, selector, network-idle, or download condition. `timeout_ms` is at most 120 seconds. Load and URL waits wake on Chrome tab events; network and download waits wake on tab-scoped debugger events; text and selector waits use a bounded page `MutationObserver`. A 500 ms ownership/policy heartbeat covers missed events and guarantees cleanup at the deadline. A `tab_only` route accepts load and URL conditions only. |
 | `browser_tabs` | Takes an empty object and lists only the current task's tabs, including each tab's `automation_route`. |
 | `browser_handoff` | Requires a task tab, expected page revision, prompt, completion condition, and optional timeout. Completion can be navigation, manual completion, a URL, or a selector. It remains available on a `tab_only` route because AgentTab blocks agent observation while the human controls the tab, but selector completion requires the `full` route. |
 | `browser_commit` | Requires the staged token returned by a prior `commit_required` action and executes that one staged operation. On a `tab_only` route, only a staged close can execute; page-dependent staged actions require the `full` route. |
 
 Every existing-page mutation carries its expected page revision. If navigation or document replacement makes that revision stale, AgentTab rejects the operation rather than selecting a new target.
 
+Raw SDK callers can use `AgentTabClient.actWaitObserve` (TypeScript) or `act_wait_observe` (Python) for the standard act → wait → observe loop. The helper infers `load` after navigation/history/reload and `network_idle` after click/select/drag/dialog/upload, accepts an explicit wait condition for a deterministic verification, then returns a fresh accessibility snapshot by default. Its result exposes the original action `outcome`; waits and observations run only after `completed`, while `commit_required` and `needs_user` return immediately. It composes the existing v1 methods; no protocol field or server upgrade is required. See [Semantic automation](semantic-automation.md).
+
 `automation_route` is `full` for ordinary HTTP, HTTPS, and `about:blank` tabs. It is `tab_only` with `route_reason: "browser_restricted_origin"` for Chrome system pages, extension pages, DevTools, the Chrome Web Store, malformed URLs, and unknown schemes. Page inspection or interaction requested on a `tab_only` tab returns `browser_restricted_origin` with `outcome: "not_started"` and recovery that explicitly says not to retry the same AgentTab route. This is a browser platform boundary, not a policy denial and not permission that can be granted through AgentTab.
 
 ### Developer mode
@@ -139,4 +141,5 @@ Commit reduces recognizable risk only. It requires both the popup's human approv
 - [Handoff parameters](../schemas/rpc/v1/browser-handoff.schema.json)
 - [Commit parameters](../schemas/rpc/v1/browser-commit.schema.json)
 - [Commands](commands.md)
+- [Semantic automation](semantic-automation.md)
 - [Setup](setup.md)

docs/semantic-automation.md

@@ -0,0 +1,65 @@
+# Semantic automation
+
+AgentTab can run reliable browser loops without fixed post-action sleeps. This is additive to Core RPC v1: existing revisioned refs and separate `browser_act`, `browser_wait`, and `browser_snapshot` calls continue to work unchanged.
+
+## Prefer semantic refs
+
+An accessibility snapshot still returns an exact node `ref` such as `r7-204`. For a uniquely named actionable role, it also returns a `semantic_ref` such as `a7:button:Continue`.
+
+- The page revision remains part of the ref, so navigation cannot silently retarget an old action.
+- The role and accessible name are resolved against the current accessibility tree. A same-document SPA rerender may replace backend node `204` and the semantic ref can still resolve.
+- A semantic ref is emitted only for a unique, named actionable target in the captured tree. Long names that cannot fit the protocol's 256-character ref bound retain only their exact ref.
+- If a later tree has no match, AgentTab returns `target_not_found` with `outcome: not_started`.
+- If a later tree has multiple matches, AgentTab returns `ambiguous_target` with `outcome: not_started`. `error.details.candidates` contains at most eight exact refs from a fresh live-tree resolution; take a new snapshot and choose using surrounding context.
+
+Exact refs remain useful when two controls intentionally share a role and name. Semantic resolution does not guess by DOM order, fuzzy text, coordinates, or the previously matching backend node.
+
+## Wait on state, not time
+
+`browser_wait` uses a hybrid event engine:
+
+| Condition | Primary wake-up | Bounded fallback |
+|---|---|---|
+| `load`, `url` | `chrome.tabs.onUpdated` | 500 ms ownership and policy revalidation |
+| `network_idle`, `download` | tab-scoped CDP network/download events | quiet-window deadline and 500 ms revalidation |
+| `text`, `selector` | page `MutationObserver` | 500 ms observer slice and revalidation |
+
+Every timer and page observer is removed on match, navigation, error, or timeout. The requested `timeout_ms` is a real deadline, not a retry count. The heartbeat handles a browser event that arrives between the state check and listener registration while keeping ownership revocation and route changes responsive.
+
+Network idle means the task tab has no tracked in-flight request and has remained quiet for 500 ms. It is not a claim that application work has completed. Prefer a specific selector, text, or URL condition when the page exposes one.
+
+## Act → wait → observe
+
+The TypeScript and Python SDKs provide an optional convenience workflow. It issues ordinary Core v1 requests in order, so results, idempotency, Commit, handoff, and recovery semantics stay visible.
+
+```ts
+const result = await client.actWaitObserve({
+  act: {
+    tab_id: tabId,
+    expected_page_revision: pageRevision,
+    actions: [{ kind: "click", ref: continueButton.semantic_ref }],
+  },
+  wait: { kind: "selector", value: "[data-step='shipping']" },
+  waitTimeoutMs: 15_000,
+  observe: { mode: "accessibility", max_nodes: 500 },
+});
+```
+
+```python
+result = client.act_wait_observe(
+    {
+        "tab_id": tab_id,
+        "expected_page_revision": page_revision,
+        "actions": [{"kind": "click", "ref": continue_button["semantic_ref"]}],
+    },
+    wait={"kind": "selector", "value": "[data-step='shipping']"},
+    wait_timeout_ms=15_000,
+    observe={"mode": "accessibility", "max_nodes": 500},
+)
+```
+
+When `wait` is omitted, the helper uses `load` after navigation/history/reload, `network_idle` after click/select/drag/dialog/upload, and no wait after local input or scrolling. It returns a fresh accessibility snapshot unless `observe=False`. Pass `wait=false` in TypeScript or `wait=False` in Python to observe immediately. Closing a tab returns after the action and skips wait and observation.
+
+The returned object always includes `outcome`, copied from the `browser_act` Core response, and `action`, containing that response's result payload. `wait` and `observation` are present only when the action outcome is `completed`. A successful `commit_required` or `needs_user` response is returned immediately so callers can commit or hand off without accidentally starting a wait against an action that did not run.
+
+An explicit page postcondition is preferable to the inferred default. The helper is intentionally not one opaque server transaction: failed action responses and unknown transport outcomes still raise the SDK's normal errors before any wait or observation begins.

packages/extension/src/background.ts

@@ -44,6 +44,8 @@ const PRE_DISPATCH_ERRORS: Record<string, true> = {
   permissions_required: true,
   stale_revision: true,
   stale_ref: true,
+  target_not_found: true,
+  ambiguous_target: true,
   paused: true,
   developer_mode_required: true,
   invalid_staged_token: true,
@@ -334,6 +336,15 @@ function errorRecovery(error: unknown): string | undefined {
   return isRecord(error) && typeof error.recovery === "string" ? error.recovery : undefined;
 }
 
+function errorDetails(error: unknown): Record<string, unknown> | undefined {
+  if (!isRecord(error)) return undefined;
+  const details = isRecord(error.details) ? { ...error.details } : {};
+  if (typeof error.currentPageRevision === "number") {
+    details.current_page_revision = error.currentPageRevision;
+  }
+  return Object.keys(details).length > 0 ? details : undefined;
+}
+
 function errorOutcome(error: unknown, mutating: boolean, code: string): Outcome {
   if (isRecord(error) && typeof error.outcome === "string") {
     const outcome = error.outcome;
@@ -390,7 +401,7 @@ async function dispatch(command: NativeDispatchCommand): Promise<NativeResponse>
       return completed(command.request_id, await scheduler.enqueueGlobal(() => ownership.open(command.task_id, params)));
     }
     if (command.method === "browser_tabs") {
-      const result = await scheduler.enqueueGlobal(() => ownership.inventory());
+      const result = await scheduler.readAfterAllWrites(() => ownership.inventory());
       return completed(command.request_id, {
         tabs: result
           .filter((tab) => tab.task_id === command.task_id)
@@ -487,9 +498,7 @@ async function dispatch(command: NativeDispatchCommand): Promise<NativeResponse>
       normalized instanceof Error ? normalized.message : String(normalized),
       errorOutcome(normalized, mutating, code),
       errorRecovery(normalized),
-      isRecord(normalized) && typeof normalized.currentPageRevision === "number"
-        ? { current_page_revision: normalized.currentPageRevision }
-        : undefined,
+      errorDetails(normalized),
     );
   }
 }