diff --git a/CHANGELOG.md b/CHANGELOG.md index f60ac7b..2aa0ad1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,7 @@ # Changelog ## Unreleased +- Added a compatibility-preserving semantic execution loop for AgentTab v2. Accessibility snapshots now include `semantic_ref` for uniquely named actionable roles; actions resolve those refs against the live accessibility tree so same-document SPA rerenders do not stale an otherwise stable target, while missing or newly ambiguous targets fail before execution with bounded recovery candidates. `browser_wait` now wakes from Chrome tab events, CDP network/download events, or a page MutationObserver with a 500 ms ownership/policy heartbeat instead of polling every 100 ms. TypeScript `actWaitObserve` and Python `act_wait_observe` helpers compose the existing act, deterministic wait, and fresh snapshot methods without changing Core RPC v1. - Replaced the Chrome Bridge v1 runtime with the AgentTab 2.0 release candidate: a Rust production host over OS-native local IPC, seven task-scoped Standard methods, explicit resumable capabilities, a developer-only eighth method, TypeScript and Python SDKs, MCP and OMP adapters, a transactional installer, and a minimal extension. Consequential controls now use a two-party Commit flow: `browser_act` stages an exact effect, the popup approves the durable review record without executing it, and the requesting task must consume its private one-use token through `browser_commit`. - Added task-owned background window creation to Standard `browser_open` through `placement: "new_window"`. It is operation-specific rather than a general window-control grant: only an otherwise empty task can request it, the extension creates an unfocused normal window, `background: false` is rejected, ownership is derived from persisted task state, and a failed visible group grant removes the new tab. Focus, state changes, and closure of unrelated windows remain unavailable. - Added explicit restricted-origin routing for task tabs. `browser_open` and `browser_tabs` now report `automation_route: "full" | "tab_only"`; Chrome system pages, extension pages, DevTools, and the Chrome Web Store retain task-owned explicit navigation, reload, close, and bounded waits, while page inspection, interaction, and raw CDP fail before execution with the stable `browser_restricted_origin` / `not_started` result and non-retry recovery. History movement remains available without managed origin constraints and fails closed when constraints are configured because Chrome does not expose its destination for pre-navigation authorization. diff --git a/README.md b/README.md index 186700d..4b8fa9f 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ The command has no path, token, or shell-specific argument and is suitable for P ## A task-owned workflow 1. An agent calls `browser_open` with `mode: "create"`. AgentTab creates a background tab for that task and returns its task, tab, window, page-revision, and automation-route identifiers. `placement: "new_window"` may create the task's first tab in a separate unfocused normal window. -2. On a normal web origin, the agent calls `browser_snapshot`, works from revisioned accessibility references, then calls `browser_act` with the expected page revision. It cannot act on unrelated tabs. +2. On a normal web origin, the agent calls `browser_snapshot`, prefers a unique accessibility `semantic_ref`, then calls `browser_act` with the expected page revision. AgentTab resolves the semantic target against the live accessibility tree, waits on browser/page events rather than requiring arbitrary sleeps, and rejects ambiguous targets with bounded candidates. It cannot act on unrelated tabs. 3. If a site requires human-only input, the agent calls `browser_handoff`. AgentTab focuses that tab, pauses automation, and blocks browser observation until the declared completion condition or **I'm done**. 4. If AgentTab recognizes a send, publish, purchase, delete, upload, authorization, or permission-grant control, `browser_act` can return `commit_required`. The extension shows the staged effect in its popup. A human must approve it there before the agent can call `browser_commit` with the one-use staged token. 5. The task can list only its own tabs with `browser_tabs`. A separate client gets a separate task unless it proves its durable resume capability. @@ -111,6 +111,7 @@ Generated extension assets live only in `packages/extension/dist/`; they are not - [Setup and local paths](docs/setup.md) - [Command reference](docs/commands.md) - [MCP adapter and Core RPC](docs/mcp.md) +- [Semantic automation and deterministic waits](docs/semantic-automation.md) - [Security and trust boundary](docs/security.md) - [Multi-agent behavior](docs/multi-agent.md) - [Runtime architecture decision](docs/adr/0001-agenttab-runtime.md) diff --git a/docs/mcp.md b/docs/mcp.md index 650cb97..1e8e46a 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -63,15 +63,17 @@ For MCP, the capability store namespace is `mcp`; OMP uses `omp`; Pi uses `pi`. | Tool | Required input and behavior | |---|---| | `browser_open` | `mode: "create"` optionally accepts an `http`, `https`, or `about` URL, `background`, and `placement`. The default `placement: "task"` creates a tab in the task's existing window when possible. `placement: "new_window"` creates the first tab of an otherwise empty task in a separate unfocused normal window and rejects `background: false`. `mode: "adopt_active"` explicitly adopts only the currently active tab. The result includes task, tab, window, page-revision, and `automation_route` identifiers. | -| `browser_snapshot` | Requires `tab_id`. Modes are `accessibility`, `text`, `html`, and `screenshot`. Only accessibility snapshots return revisioned node references. Screenshot requests may select `png`, `jpeg`, or `webp`, set JPEG/WebP `quality`, bound dimensions with `max_width`/`max_height`, and cap compressed image size with `max_bytes`. Snapshots require the `full` automation route. | -| `browser_act` | Requires `tab_id`, `expected_page_revision`, and one to 64 typed actions. Actions are click, type, fill, select, scroll, drag, navigate, history movement, reload, close, dialog decision, and staged file upload. No coordinate action exists in Standard mode. A `tab_only` route accepts explicit navigation, history movement, reload, and close only. Managed origin constraints disable history movement because Chrome does not expose its destination for authorization before navigation; use explicit navigation to an allowed URL instead. | -| `browser_wait` | Requires `tab_id` and one load, URL, text, selector, network-idle, or download condition. `timeout_ms` is at most 120 seconds. A `tab_only` route accepts load and URL conditions only; network-idle and download attribution require the tab-scoped debugger connection available on the `full` route. | +| `browser_snapshot` | Requires `tab_id`. Modes are `accessibility`, `text`, `html`, and `screenshot`. Accessibility nodes retain their revisioned `ref`; uniquely named actionable nodes also return a stable `semantic_ref` that can be passed anywhere a ref is accepted. Screenshot requests may select `png`, `jpeg`, or `webp`, set JPEG/WebP `quality`, bound dimensions with `max_width`/`max_height`, and cap compressed image size with `max_bytes`. Snapshots require the `full` automation route. | +| `browser_act` | Requires `tab_id`, `expected_page_revision`, and one to 64 typed actions. Actions are click, type, fill, select, scroll, drag, navigate, history movement, reload, close, dialog decision, and staged file upload. Prefer an available `semantic_ref`: AgentTab resolves its accessibility role and name against the live tree, so an SPA can replace the DOM node without invalidating the target. If the target disappears or becomes ambiguous, the action does not start and the error returns recovery plus bounded candidate refs. No coordinate action exists in Standard mode. A `tab_only` route accepts explicit navigation, history movement, reload, and close only. Managed origin constraints disable history movement because Chrome does not expose its destination for authorization before navigation; use explicit navigation to an allowed URL instead. | +| `browser_wait` | Requires `tab_id` and one load, URL, text, selector, network-idle, or download condition. `timeout_ms` is at most 120 seconds. Load and URL waits wake on Chrome tab events; network and download waits wake on tab-scoped debugger events; text and selector waits use a bounded page `MutationObserver`. A 500 ms ownership/policy heartbeat covers missed events and guarantees cleanup at the deadline. A `tab_only` route accepts load and URL conditions only. | | `browser_tabs` | Takes an empty object and lists only the current task's tabs, including each tab's `automation_route`. | | `browser_handoff` | Requires a task tab, expected page revision, prompt, completion condition, and optional timeout. Completion can be navigation, manual completion, a URL, or a selector. It remains available on a `tab_only` route because AgentTab blocks agent observation while the human controls the tab, but selector completion requires the `full` route. | | `browser_commit` | Requires the staged token returned by a prior `commit_required` action and executes that one staged operation. On a `tab_only` route, only a staged close can execute; page-dependent staged actions require the `full` route. | Every existing-page mutation carries its expected page revision. If navigation or document replacement makes that revision stale, AgentTab rejects the operation rather than selecting a new target. +Raw SDK callers can use `AgentTabClient.actWaitObserve` (TypeScript) or `act_wait_observe` (Python) for the standard act → wait → observe loop. The helper infers `load` after navigation/history/reload and `network_idle` after click/select/drag/dialog/upload, accepts an explicit wait condition for a deterministic verification, then returns a fresh accessibility snapshot by default. Its result exposes the original action `outcome`; waits and observations run only after `completed`, while `commit_required` and `needs_user` return immediately. It composes the existing v1 methods; no protocol field or server upgrade is required. See [Semantic automation](semantic-automation.md). + `automation_route` is `full` for ordinary HTTP, HTTPS, and `about:blank` tabs. It is `tab_only` with `route_reason: "browser_restricted_origin"` for Chrome system pages, extension pages, DevTools, the Chrome Web Store, malformed URLs, and unknown schemes. Page inspection or interaction requested on a `tab_only` tab returns `browser_restricted_origin` with `outcome: "not_started"` and recovery that explicitly says not to retry the same AgentTab route. This is a browser platform boundary, not a policy denial and not permission that can be granted through AgentTab. ### Developer mode @@ -139,4 +141,5 @@ Commit reduces recognizable risk only. It requires both the popup's human approv - [Handoff parameters](../schemas/rpc/v1/browser-handoff.schema.json) - [Commit parameters](../schemas/rpc/v1/browser-commit.schema.json) - [Commands](commands.md) +- [Semantic automation](semantic-automation.md) - [Setup](setup.md) diff --git a/docs/semantic-automation.md b/docs/semantic-automation.md new file mode 100644 index 0000000..b35f08f --- /dev/null +++ b/docs/semantic-automation.md @@ -0,0 +1,65 @@ +# Semantic automation + +AgentTab can run reliable browser loops without fixed post-action sleeps. This is additive to Core RPC v1: existing revisioned refs and separate `browser_act`, `browser_wait`, and `browser_snapshot` calls continue to work unchanged. + +## Prefer semantic refs + +An accessibility snapshot still returns an exact node `ref` such as `r7-204`. For a uniquely named actionable role, it also returns a `semantic_ref` such as `a7:button:Continue`. + +- The page revision remains part of the ref, so navigation cannot silently retarget an old action. +- The role and accessible name are resolved against the current accessibility tree. A same-document SPA rerender may replace backend node `204` and the semantic ref can still resolve. +- A semantic ref is emitted only for a unique, named actionable target in the captured tree. Long names that cannot fit the protocol's 256-character ref bound retain only their exact ref. +- If a later tree has no match, AgentTab returns `target_not_found` with `outcome: not_started`. +- If a later tree has multiple matches, AgentTab returns `ambiguous_target` with `outcome: not_started`. `error.details.candidates` contains at most eight exact refs from a fresh live-tree resolution; take a new snapshot and choose using surrounding context. + +Exact refs remain useful when two controls intentionally share a role and name. Semantic resolution does not guess by DOM order, fuzzy text, coordinates, or the previously matching backend node. + +## Wait on state, not time + +`browser_wait` uses a hybrid event engine: + +| Condition | Primary wake-up | Bounded fallback | +|---|---|---| +| `load`, `url` | `chrome.tabs.onUpdated` | 500 ms ownership and policy revalidation | +| `network_idle`, `download` | tab-scoped CDP network/download events | quiet-window deadline and 500 ms revalidation | +| `text`, `selector` | page `MutationObserver` | 500 ms observer slice and revalidation | + +Every timer and page observer is removed on match, navigation, error, or timeout. The requested `timeout_ms` is a real deadline, not a retry count. The heartbeat handles a browser event that arrives between the state check and listener registration while keeping ownership revocation and route changes responsive. + +Network idle means the task tab has no tracked in-flight request and has remained quiet for 500 ms. It is not a claim that application work has completed. Prefer a specific selector, text, or URL condition when the page exposes one. + +## Act → wait → observe + +The TypeScript and Python SDKs provide an optional convenience workflow. It issues ordinary Core v1 requests in order, so results, idempotency, Commit, handoff, and recovery semantics stay visible. + +```ts +const result = await client.actWaitObserve({ + act: { + tab_id: tabId, + expected_page_revision: pageRevision, + actions: [{ kind: "click", ref: continueButton.semantic_ref }], + }, + wait: { kind: "selector", value: "[data-step='shipping']" }, + waitTimeoutMs: 15_000, + observe: { mode: "accessibility", max_nodes: 500 }, +}); +``` + +```python +result = client.act_wait_observe( + { + "tab_id": tab_id, + "expected_page_revision": page_revision, + "actions": [{"kind": "click", "ref": continue_button["semantic_ref"]}], + }, + wait={"kind": "selector", "value": "[data-step='shipping']"}, + wait_timeout_ms=15_000, + observe={"mode": "accessibility", "max_nodes": 500}, +) +``` + +When `wait` is omitted, the helper uses `load` after navigation/history/reload, `network_idle` after click/select/drag/dialog/upload, and no wait after local input or scrolling. It returns a fresh accessibility snapshot unless `observe=False`. Pass `wait=false` in TypeScript or `wait=False` in Python to observe immediately. Closing a tab returns after the action and skips wait and observation. + +The returned object always includes `outcome`, copied from the `browser_act` Core response, and `action`, containing that response's result payload. `wait` and `observation` are present only when the action outcome is `completed`. A successful `commit_required` or `needs_user` response is returned immediately so callers can commit or hand off without accidentally starting a wait against an action that did not run. + +An explicit page postcondition is preferable to the inferred default. The helper is intentionally not one opaque server transaction: failed action responses and unknown transport outcomes still raise the SDK's normal errors before any wait or observation begins. diff --git a/packages/extension/src/background.ts b/packages/extension/src/background.ts index c860ecf..9dffaa6 100644 --- a/packages/extension/src/background.ts +++ b/packages/extension/src/background.ts @@ -44,6 +44,8 @@ const PRE_DISPATCH_ERRORS: Record = { permissions_required: true, stale_revision: true, stale_ref: true, + target_not_found: true, + ambiguous_target: true, paused: true, developer_mode_required: true, invalid_staged_token: true, @@ -334,6 +336,15 @@ function errorRecovery(error: unknown): string | undefined { return isRecord(error) && typeof error.recovery === "string" ? error.recovery : undefined; } +function errorDetails(error: unknown): Record | undefined { + if (!isRecord(error)) return undefined; + const details = isRecord(error.details) ? { ...error.details } : {}; + if (typeof error.currentPageRevision === "number") { + details.current_page_revision = error.currentPageRevision; + } + return Object.keys(details).length > 0 ? details : undefined; +} + function errorOutcome(error: unknown, mutating: boolean, code: string): Outcome { if (isRecord(error) && typeof error.outcome === "string") { const outcome = error.outcome; @@ -390,7 +401,7 @@ async function dispatch(command: NativeDispatchCommand): Promise return completed(command.request_id, await scheduler.enqueueGlobal(() => ownership.open(command.task_id, params))); } if (command.method === "browser_tabs") { - const result = await scheduler.enqueueGlobal(() => ownership.inventory()); + const result = await scheduler.readAfterAllWrites(() => ownership.inventory()); return completed(command.request_id, { tabs: result .filter((tab) => tab.task_id === command.task_id) @@ -487,9 +498,7 @@ async function dispatch(command: NativeDispatchCommand): Promise normalized instanceof Error ? normalized.message : String(normalized), errorOutcome(normalized, mutating, code), errorRecovery(normalized), - isRecord(normalized) && typeof normalized.currentPageRevision === "number" - ? { current_page_revision: normalized.currentPageRevision } - : undefined, + errorDetails(normalized), ); } } diff --git a/packages/extension/src/browser.ts b/packages/extension/src/browser.ts index 9c5de57..657b9a0 100644 --- a/packages/extension/src/browser.ts +++ b/packages/extension/src/browser.ts @@ -17,6 +17,28 @@ const DEBUGGER_IDLE_MS = 30_000; // the 1 MiB host-to-client frame. Screenshot base64 data is separately capped at // 1,000,000 characters (750,000 decoded bytes). const SNAPSHOT_RESULT_BUDGET_BYTES = 1_032_000; +const WAIT_REVALIDATE_MS = 500; +const NETWORK_IDLE_MS = 500; +const DOM_OBSERVER_SCAN_INTERVAL_MS = 50; +const SEMANTIC_REF_MAX_CHARS = 256; +const SEMANTIC_ROLES = new Set([ + "button", + "checkbox", + "combobox", + "link", + "listbox", + "menuitem", + "option", + "radio", + "searchbox", + "slider", + "spinbutton", + "switch", + "tab", + "textbox", + "treeitem", +]); +const LAZY_DEBUGGER_DOMAINS = new Set(["DOM", "Accessibility", "Runtime"]); const TAB_ONLY_ACTIONS: Readonly> = { navigate: true, go_back: true, @@ -66,6 +88,8 @@ interface DebugSession { pageLoadInFlight: boolean; downloads: Map; dialogGeneration: number; + enabledDomains: Set; + domainEnableTail: Promise; dialog?: JavaScriptDialog; pendingWindowOpen?: PendingWindowOpen; } @@ -123,6 +147,12 @@ function base64ByteLength(value: string): number | null { return (value.length / 4) * 3 - padding; } +interface SemanticTarget { + pageRevision: number; + role: string; + name: string; +} + type CloseTab = (tabId: number) => Promise; type EventSink = (event: string, payload: Record) => void; type AuthorizeDebuggerUse = (tabId: number) => Promise; @@ -151,6 +181,7 @@ export class StandardBrowserRuntime { private readonly sessions = new Map(); private readonly expectedDetaches = new Map(); private readonly debuggerCandidates = new Set(); + private readonly waitSignals = new Map void>>(); constructor( private readonly revisions: RevisionTracker, @@ -172,6 +203,7 @@ export class StandardBrowserRuntime { const session = this.sessions.get(source.tabId); if (session?.idleTimer) clearTimeout(session.idleTimer); this.sessions.delete(source.tabId); + this.signalWaiters(source.tabId); void this.invalidateStagedDialogs(source.tabId); }); chrome.debugger.onEvent.addListener( @@ -180,6 +212,7 @@ export class StandardBrowserRuntime { const session = this.sessions.get(source.tabId); if (!session) return; const params = isRecord(rawParams) ? rawParams : {}; + let shouldSignalWaiters = false; if (method === "Network.requestWillBeSent" && typeof params.requestId === "string") { session.inflight.add(params.requestId); session.lastNetworkActivity = Date.now(); @@ -189,6 +222,10 @@ export class StandardBrowserRuntime { ) { session.inflight.delete(params.requestId); session.lastNetworkActivity = Date.now(); + // A network-idle waiter cannot match while any request remains. Wake + // it only when the burst drains instead of revalidating on every + // resource completion. + shouldSignalWaiters = session.inflight.size === 0; // chrome.downloads has no initiator tab. Debugger events are scoped by source.tabId, // so a matching Page GUID is the only completion proof accepted here. } else if (method === "Page.downloadWillBegin" && typeof params.guid === "string") { @@ -197,6 +234,7 @@ export class StandardBrowserRuntime { const download = session.downloads.get(params.guid); if (params.state === "completed" && download) { download.completedAt = Date.now(); + shouldSignalWaiters = true; } else if (params.state === "canceled") { session.downloads.delete(params.guid); } @@ -227,8 +265,18 @@ export class StandardBrowserRuntime { session.dialog = undefined; void this.invalidateStagedDialogs(source.tabId); } + if (shouldSignalWaiters) this.signalWaiters(source.tabId); }, ); + chrome.tabs.onUpdated.addListener((tabId, changeInfo) => { + if ( + changeInfo.status === "loading" || + changeInfo.status === "complete" || + typeof changeInfo.url === "string" + ) { + this.signalWaiters(tabId); + } + }); this.revisions.onChange((tabId) => this.invalidateStagedDialogs(tabId)); } @@ -372,7 +420,7 @@ export class StandardBrowserRuntime { ); const result = typeof params.root_ref === "string" ? await this.send(tabId, "Accessibility.getPartialAXTree", { - backendNodeId: this.backendNodeId(pageRevision, params.root_ref), + backendNodeId: await this.resolveBackendNodeId(tabId, pageRevision, params.root_ref), fetchRelatives: true, }) : await this.send(tabId, "Accessibility.getFullAXTree", { depth: maxDepth }); @@ -389,16 +437,39 @@ export class StandardBrowserRuntime { }); } const nodes = (Array.isArray(result.nodes) ? result.nodes.filter(isRecord) : []) as AxNode[]; - const encoded = nodes.slice(0, maxNodes).map((node) => ({ - ...(node.backendDOMNodeId - ? { ref: `r${pageRevision}-${node.backendDOMNodeId}` } - : {}), - role: typeof node.role?.value === "string" ? node.role.value : "unknown", - name: typeof node.name?.value === "string" ? node.name.value : "", - ...(node.value?.value !== undefined ? { value: node.value.value } : {}), - ...(node.description?.value !== undefined ? { description: node.description.value } : {}), - ...(node.ignored ? { ignored: true } : {}), - })); + const semanticCounts = new Map(); + for (const node of nodes) { + const role = this.axText(node.role?.value); + const name = this.axText(node.name?.value); + if (!node.ignored && node.backendDOMNodeId && SEMANTIC_ROLES.has(role) && name.length > 0) { + const key = this.semanticKey(role, name); + semanticCounts.set(key, (semanticCounts.get(key) ?? 0) + 1); + } + } + const encoded = nodes.slice(0, maxNodes).map((node) => { + const role = this.axText(node.role?.value) || "unknown"; + const name = this.axText(node.name?.value); + const nodeRef = node.backendDOMNodeId + ? `r${pageRevision}-${node.backendDOMNodeId}` + : undefined; + const semanticRef = + !node.ignored && + nodeRef !== undefined && + SEMANTIC_ROLES.has(role) && + name.length > 0 && + semanticCounts.get(this.semanticKey(role, name)) === 1 + ? this.encodeSemanticRef(pageRevision, role, name) + : undefined; + return { + ...(nodeRef !== undefined ? { ref: nodeRef } : {}), + ...(semanticRef !== undefined ? { semantic_ref: semanticRef } : {}), + role, + name, + ...(node.value?.value !== undefined ? { value: node.value.value } : {}), + ...(node.description?.value !== undefined ? { description: node.description.value } : {}), + ...(node.ignored ? { ignored: true } : {}), + }; + }); return assertDeliverableSnapshot({ tab_id: tabId, page_revision: pageRevision, @@ -446,7 +517,13 @@ export class StandardBrowserRuntime { }); } await this.revisions.assertExpected(tabId, pageRevision); - const stagedConsequence = await this.consequence(tabId, pageRevision, action); + const resolvedTargets = new Map(); + const stagedConsequence = await this.consequence( + tabId, + pageRevision, + action, + resolvedTargets, + ); if (stagedConsequence) { const staged: StagedCommit = { native_token: randomToken(), @@ -492,7 +569,12 @@ export class StandardBrowserRuntime { }, }; } - completedActions.push(await this.performAction(tabId, pageRevision, action)); + completedActions.push(await this.performAction( + tabId, + pageRevision, + action, + resolvedTargets, + )); } return { result: { @@ -659,8 +741,15 @@ export class StandardBrowserRuntime { code: "staged_commit_mismatch", }); } + const resolvedTargets = new Map(); const currentTarget = typeof action.ref === "string" - ? await this.targetDescriptor(staged.tab_id, staged.page_revision, action.ref, action) + ? await this.targetDescriptor( + staged.tab_id, + staged.page_revision, + action.ref, + action, + resolvedTargets, + ) : { kind: action.kind }; if ( await this.stageFingerprint(taskId, staged.tab_id, staged.page_revision, action, currentTarget) !== @@ -678,7 +767,12 @@ export class StandardBrowserRuntime { }); const result = action.kind === "dialog" && action.decision === "accept" ? await this.acceptStagedDialog(staged.tab_id, action, staged.dialog) - : await this.performAction(staged.tab_id, staged.page_revision, action); + : await this.performAction( + staged.tab_id, + staged.page_revision, + action, + resolvedTargets, + ); return { tab_id: staged.tab_id, page_revision: await this.revisions.current(staged.tab_id), @@ -749,9 +843,34 @@ export class StandardBrowserRuntime { matched: true, }; } - const delay = Promise.withResolvers(); - setTimeout(delay.resolve, 100); - await delay.promise; + const remainingMs = deadline - Date.now(); + if (remainingMs <= 0) break; + if (conditionKind === "text" || conditionKind === "selector") { + const observerStartedAt = Date.now(); + const observerSliceMs = Math.min(remainingMs, WAIT_REVALIDATE_MS); + const observed = await this.waitForDomCondition( + tabId, + condition, + observerSliceMs, + ); + if (observed) continue; + const unusedSliceMs = observerSliceMs - (Date.now() - observerStartedAt); + const observerRemainingMs = deadline - Date.now(); + if (unusedSliceMs > 0 && observerRemainingMs > 0) { + await this.waitForTabSignal( + tabId, + Math.min(unusedSliceMs, observerRemainingMs), + ); + } + continue; + } + await this.waitForTabSignal( + tabId, + Math.min( + remainingMs, + this.nextWaitDelay(conditionKind, debuggerSession, waitStartedAtMs), + ), + ); } while (Date.now() < deadline); throw Object.assign(new Error(`Timed out waiting for ${String(condition.kind)}`), { code: "wait_timeout", @@ -771,7 +890,14 @@ export class StandardBrowserRuntime { }); } await this.requireFullAutomationRoute(tabId, `run ${action}`); - return this.send(tabId, action, params); + const result = await this.send(tabId, action, params); + if (action.endsWith(".disable")) { + // Developer mode can invalidate the Standard runtime's domain cache. + // Recycle the attachment so the next Standard call re-establishes a + // known Page, Network, and lazy-domain baseline. + await this.detach(tabId); + } + return result; } private async scriptSnapshot( @@ -955,6 +1081,7 @@ export class StandardBrowserRuntime { tabId: number, pageRevision: number, action: Record, + resolvedTargets: Map, ): Promise { if (action.kind === "close") { return { @@ -967,7 +1094,7 @@ export class StandardBrowserRuntime { return { effect: `Upload ${count} ${count === 1 ? "file" : "files"} to the page`, target: typeof action.ref === "string" - ? await this.targetDescriptor(tabId, pageRevision, action.ref) + ? await this.targetDescriptor(tabId, pageRevision, action.ref, undefined, resolvedTargets) : { kind: action.kind }, }; } @@ -986,7 +1113,13 @@ export class StandardBrowserRuntime { ) { return null; } - const target = await this.targetDescriptor(tabId, pageRevision, action.ref, action); + const target = await this.targetDescriptor( + tabId, + pageRevision, + action.ref, + action, + resolvedTargets, + ); const label = [ target.role, target.text, @@ -1090,13 +1223,19 @@ export class StandardBrowserRuntime { pageRevision: number, ref: unknown, action?: Record, + resolvedTargets?: Map, ): Promise> { const requestedValue = action?.kind === "select" ? String(action.value ?? "") : action?.kind === "fill" || action?.kind === "type" ? String(action.text ?? "") : null; - const backendNodeId = this.backendNodeId(pageRevision, ref); + const backendNodeId = await this.resolveBackendNodeId( + tabId, + pageRevision, + ref, + resolvedTargets, + ); const resolved = await this.send(tabId, "DOM.resolveNode", { backendNodeId }); if (!isRecord(resolved.object) || typeof resolved.object.objectId !== "string") { throw Object.assign(new Error("Snapshot ref no longer resolves"), { code: "stale_ref" }); @@ -1124,6 +1263,7 @@ export class StandardBrowserRuntime { tabId: number, pageRevision: number, action: Record, + resolvedTargets: Map = new Map(), ): Promise> { await this.authorizeDebuggerUse(tabId); const kind = action.kind; @@ -1170,7 +1310,12 @@ export class StandardBrowserRuntime { }); return { kind, completed: true }; } - const backendNodeId = this.backendNodeId(pageRevision, action.ref); + const backendNodeId = await this.resolveBackendNodeId( + tabId, + pageRevision, + action.ref, + resolvedTargets, + ); if (kind === "click") { const [activeTab, existingTabs] = await Promise.all([ chrome.tabs.query({ active: true, lastFocusedWindow: true }).then(([tab]) => tab), @@ -1241,7 +1386,12 @@ export class StandardBrowserRuntime { [{ value: Number(action.delta_x ?? 0) }, { value: Number(action.delta_y ?? 0) }], ); } else if (kind === "drag") { - const targetBackendNodeId = this.backendNodeId(pageRevision, action.target_ref); + const targetBackendNodeId = await this.resolveBackendNodeId( + tabId, + pageRevision, + action.target_ref, + resolvedTargets, + ); const [source, target] = await Promise.all([ this.nodeCenter(tabId, backendNodeId), this.nodeCenter(tabId, targetBackendNodeId), @@ -1307,7 +1457,8 @@ export class StandardBrowserRuntime { session.pageLoadInFlight = false; session.lastNetworkActivity = Date.now(); } - return session.inflight.size === 0 && Date.now() - session.lastNetworkActivity >= 500; + const quietSince = Math.max(session.lastNetworkActivity, waitStartedAtMs); + return session.inflight.size === 0 && Date.now() - quietSince >= NETWORK_IDLE_MS; } if (kind === "download") { const session = debuggerSession; @@ -1323,14 +1474,261 @@ export class StandardBrowserRuntime { code: "invalid_request", }); } - private backendNodeId(pageRevision: number, ref: unknown): number { + + private async waitForDomCondition( + tabId: number, + condition: Record, + timeoutMs: number, + ): Promise { + try { + const [{ result }] = await chrome.scripting.executeScript({ + target: { tabId }, + func: async (kind: string, value: string, timeout: number, scanInterval: number) => + new Promise((resolve) => { + const matches = () => + kind === "text" + ? (document.documentElement.textContent ?? "").includes(value) + : document.querySelector(value) !== null; + if (matches()) { + resolve(true); + return; + } + let settled = false; + let scanTimer: ReturnType | undefined; + const finish = (matched: boolean) => { + if (settled) return; + settled = true; + clearTimeout(timer); + if (scanTimer !== undefined) clearTimeout(scanTimer); + observer.disconnect(); + removeEventListener("pagehide", pageHidden); + resolve(matched); + }; + const observer = new MutationObserver(() => { + // A dynamic page may deliver mutations at animation-frame rate. + // Coalesce them so a wait never rescans the whole DOM at that rate. + if (settled || scanTimer !== undefined) return; + scanTimer = setTimeout(() => { + scanTimer = undefined; + if (matches()) finish(true); + }, scanInterval); + }); + const pageHidden = () => finish(false); + const timer = setTimeout(() => finish(false), timeout); + observer.observe(document.documentElement, { + // Attribute changes can affect a CSS selector, but cannot change + // the textContent tested by a text wait. + attributes: kind === "selector", + childList: true, + characterData: kind === "text", + subtree: true, + }); + addEventListener("pagehide", pageHidden, { once: true }); + // Close the check/observe race without falling back to a poll. + if (matches()) finish(true); + }), + args: [ + String(condition.kind), + String(condition.value ?? ""), + timeoutMs, + DOM_OBSERVER_SCAN_INTERVAL_MS, + ], + }); + return result === true; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + if (/execution context was destroyed|frame was removed|cannot access contents of url/i.test(message)) { + return false; + } + throw error; + } + } + + private nextWaitDelay( + conditionKind: string, + session?: DebugSession, + waitStartedAtMs = 0, + ): number { + if ( + conditionKind === "network_idle" && + session && + !session.pageLoadInFlight && + session.inflight.size === 0 + ) { + return Math.max( + 1, + Math.min( + WAIT_REVALIDATE_MS, + NETWORK_IDLE_MS - (Date.now() - Math.max(session.lastNetworkActivity, waitStartedAtMs)), + ), + ); + } + return WAIT_REVALIDATE_MS; + } + + private waitForTabSignal(tabId: number, timeoutMs: number): Promise { + if (timeoutMs <= 0) return Promise.resolve(); + return new Promise((resolve) => { + const waiters = this.waitSignals.get(tabId) ?? new Set<() => void>(); + this.waitSignals.set(tabId, waiters); + let timer: ReturnType; + const wake = () => { + clearTimeout(timer); + waiters.delete(wake); + if (waiters.size === 0) this.waitSignals.delete(tabId); + resolve(); + }; + waiters.add(wake); + timer = setTimeout(wake, timeoutMs); + }); + } + + private signalWaiters(tabId: number): void { + const waiters = this.waitSignals.get(tabId); + if (!waiters) return; + for (const wake of [...waiters]) wake(); + } + + private axText(value: unknown): string { + return typeof value === "string" ? value.replace(/\s+/g, " ").trim() : ""; + } + + private semanticKey(role: string, name: string): string { + return `${role}\u0000${name}`; + } + + private encodeSemanticRef(pageRevision: number, role: string, name: string): string | undefined { + const ref = `a${pageRevision}:${encodeURIComponent(role)}:${encodeURIComponent(name)}`; + return ref.length <= SEMANTIC_REF_MAX_CHARS ? ref : undefined; + } + + private decodeSemanticRef(ref: unknown): SemanticTarget | null { + const match = /^a(\d+):([^:]+):(.*)$/.exec(String(ref ?? "")); + if (!match) return null; + try { + const role = decodeURIComponent(match[2]); + const name = decodeURIComponent(match[3]); + if (!SEMANTIC_ROLES.has(role) || name.length === 0) return null; + return { pageRevision: Number(match[1]), role, name }; + } catch { + return null; + } + } + + private async resolveBackendNodeId( + tabId: number, + pageRevision: number, + ref: unknown, + resolvedTargets?: Map, + ): Promise { + const cacheKey = String(ref ?? ""); + const cached = resolvedTargets?.get(cacheKey); + if (cached !== undefined) return cached; const match = /^r(\d+)-(\d+)$/.exec(String(ref ?? "")); - if (!match || Number(match[1]) !== pageRevision) { + if (match) { + if (Number(match[1]) !== pageRevision) { + throw Object.assign(new Error("Snapshot ref belongs to a stale page revision"), { + code: "stale_ref", + }); + } + const backendNodeId = Number(match[2]); + resolvedTargets?.set(cacheKey, backendNodeId); + return backendNodeId; + } + const semantic = this.decodeSemanticRef(ref); + if (!semantic || semantic.pageRevision !== pageRevision) { throw Object.assign(new Error("Snapshot ref belongs to a stale page revision"), { code: "stale_ref", }); } - return Number(match[2]); + const before = await this.pageIdentity(tabId); + const beforeRevision = await this.revisions.observeDocument( + tabId, + before.documentId, + before.loaderId, + ); + if (beforeRevision !== pageRevision) { + throw Object.assign(new Error("Semantic ref belongs to a stale page document"), { + code: "stale_ref", + currentPageRevision: beforeRevision, + recovery: "Take a fresh accessibility snapshot and retry.", + }); + } + const document = await this.send(tabId, "DOM.getDocument", { depth: 0 }); + if (!isRecord(document.root) || typeof document.root.nodeId !== "number") { + throw Object.assign(new Error("Could not inspect the semantic target document"), { + code: "target_not_found", + recovery: "Take a fresh accessibility snapshot and retry.", + }); + } + const result = await this.send(tabId, "Accessibility.queryAXTree", { + nodeId: document.root.nodeId, + accessibleName: semantic.name, + role: semantic.role, + }); + const after = await this.pageIdentity(tabId); + const afterRevision = await this.revisions.observeDocument( + tabId, + after.documentId, + after.loaderId, + ); + if ( + before.documentId !== after.documentId || + before.loaderId !== after.loaderId || + afterRevision !== pageRevision + ) { + throw Object.assign(new Error("Page changed while resolving the semantic ref"), { + code: "stale_ref", + currentPageRevision: afterRevision, + recovery: "Take a fresh accessibility snapshot and retry.", + }); + } + const nodes = (Array.isArray(result.nodes) ? result.nodes.filter(isRecord) : []) as AxNode[]; + const matches = nodes.filter((node) => + !node.ignored && + typeof node.backendDOMNodeId === "number" && + this.axText(node.role?.value) === semantic.role && + this.axText(node.name?.value) === semantic.name, + ); + if (matches.length === 1) { + const backendNodeId = matches[0].backendDOMNodeId as number; + resolvedTargets?.set(cacheKey, backendNodeId); + return backendNodeId; + } + const candidates = matches.slice(0, 8).map((node) => ({ + ref: `r${pageRevision}-${String(node.backendDOMNodeId)}`, + role: this.axText(node.role?.value), + name: this.axText(node.name?.value), + ...(node.description?.value !== undefined + ? { description: String(node.description.value).slice(0, 160) } + : {}), + })); + if (matches.length === 0) { + throw Object.assign( + new Error(`Semantic target no longer exists: ${semantic.role} “${semantic.name}”`), + { + code: "target_not_found", + recovery: "Take a fresh accessibility snapshot and retry with its semantic_ref or ref.", + details: { role: semantic.role, name: semantic.name, match_count: 0 }, + }, + ); + } + throw Object.assign( + new Error( + `Semantic target is ambiguous: ${matches.length} ${semantic.role} elements are named “${semantic.name}”`, + ), + { + code: "ambiguous_target", + recovery: "Use one of details.candidates.ref from a fresh accessibility snapshot.", + details: { + role: semantic.role, + name: semantic.name, + match_count: matches.length, + candidates, + candidates_truncated: matches.length > candidates.length, + }, + }, + ); } private async callOnNode( @@ -1472,6 +1870,8 @@ export class StandardBrowserRuntime { pageLoadInFlight: false, downloads: new Map(), dialogGeneration: 0, + enabledDomains: new Set(), + domainEnableTail: Promise.resolve(), }; this.sessions.set(tabId, session); } @@ -1492,20 +1892,11 @@ export class StandardBrowserRuntime { await chrome.debugger.attach({ tabId }, DEBUGGER_VERSION); session.attached = true; try { - for (const method of [ - "Page.enable", - "DOM.enable", - "Accessibility.enable", - "Runtime.enable", - "Network.enable", - ]) { - await this.authorizeDebuggerUse(tabId); - await chrome.debugger.sendCommand({ tabId }, method, {}); - if (method === "Network.enable") { - session.pageLoadInFlight = (await chrome.tabs.get(tabId)).status === "loading"; - session.lastNetworkActivity = Date.now(); - } - } + // Page and Network events must be observed for the lifetime of an + // attachment so dialogs, popups, downloads, and in-flight requests are + // not missed. Command-only domains are enabled on first use below. + await this.enableDebuggerDomain(tabId, session, "Page"); + await this.enableDebuggerDomain(tabId, session, "Network"); } catch (error) { try { await this.detachTrackedSession(tabId, session); @@ -1560,9 +1951,61 @@ export class StandardBrowserRuntime { }, DEBUGGER_IDLE_MS); } + private async enableDebuggerDomain( + tabId: number, + session: DebugSession, + domain: string, + ): Promise { + if (session.enabledDomains.has(domain)) return; + const enabling = session.domainEnableTail.then(async () => { + if (session.enabledDomains.has(domain)) return; + if (this.sessions.get(tabId) !== session || !session.attached) { + throw Object.assign(new Error("Debugger detached before its domain could be enabled"), { + code: "debugger_detached", + }); + } + await this.authorizeDebuggerUse(tabId); + await chrome.debugger.sendCommand({ tabId }, `${domain}.enable`, {}); + session.enabledDomains.add(domain); + if (domain === "Network") { + session.pageLoadInFlight = (await chrome.tabs.get(tabId)).status === "loading"; + session.lastNetworkActivity = Date.now(); + } + }); + session.domainEnableTail = enabling.then( + () => undefined, + () => undefined, + ); + await enabling; + } + + private async enableCommandDomain( + tabId: number, + session: DebugSession, + method: string, + ): Promise { + const separator = method.indexOf("."); + const domain = separator > 0 ? method.slice(0, separator) : ""; + if (!LAZY_DEBUGGER_DOMAINS.has(domain)) return; + try { + await this.enableDebuggerDomain(tabId, session, domain); + } catch (error) { + try { + await this.detachTrackedSession(tabId, session); + } catch (detachError) { + throw new AggregateError( + [error, detachError], + "Debugger initialization and cleanup both failed", + ); + } + throw error; + } + } + private async send(tabId: number, method: string, params: Record): Promise> { const session = await this.acquireDebuggerBusyLease(tabId); try { + await this.enableCommandDomain(tabId, session, method); await this.authorizeDebuggerUse(tabId); const result: unknown = await chrome.debugger.sendCommand({ tabId }, method, params); return isRecord(result) ? result : {}; diff --git a/packages/extension/src/scheduler.ts b/packages/extension/src/scheduler.ts index 949e678..f615339 100644 --- a/packages/extension/src/scheduler.ts +++ b/packages/extension/src/scheduler.ts @@ -102,25 +102,10 @@ export class MutationScheduler { return result; } - readAfterWrites(tabId: number | undefined, work: Work): Promise { + readAfterWrites(tabId: number, work: Work): Promise { if (!this.accepting) return Promise.reject(this.notStarted()); const admissionEpoch = this.admissionEpoch; const priorGlobal = this.globalTail; - if (tabId === undefined) { - const result = priorGlobal.then(async () => { - if (!this.accepting || this.admissionEpoch !== admissionEpoch) { - throw this.notStarted("AgentTab paused before this observation was dispatched"); - } - return work(); - }); - this.globalTail = result.then( - () => undefined, - () => undefined, - ); - this.track(result); - return result; - } - const priorTab = this.tabTails.get(tabId) ?? Promise.resolve(); const result = priorGlobal.then(() => priorTab).then(async () => { if (!this.accepting || this.admissionEpoch !== admissionEpoch) { @@ -133,6 +118,23 @@ export class MutationScheduler { return result; } + readAfterAllWrites(work: Work): Promise { + if (!this.accepting) return Promise.reject(this.notStarted()); + const admissionEpoch = this.admissionEpoch; + const priorGlobal = this.globalTail; + const priorTabs = [...this.tabTails.values()]; + const result = priorGlobal.then(() => Promise.all(priorTabs)).then(async () => { + if (!this.accepting || this.admissionEpoch !== admissionEpoch) { + throw this.notStarted("AgentTab paused before this observation was dispatched"); + } + return work(); + }); + // Observations wait for writes admitted before them, but do not become a + // global write barrier for independent work admitted afterward. + this.track(result); + return result; + } + revokeTab(tabId: number): void { this.invalidateQueuedTab(tabId, { code: "ownership_revoked", diff --git a/packages/extension/test/extension.test.ts b/packages/extension/test/extension.test.ts index da6a015..88ab5bf 100644 --- a/packages/extension/test/extension.test.ts +++ b/packages/extension/test/extension.test.ts @@ -72,6 +72,12 @@ let nextTabId: number; let nextGroupId: number; let scriptResult: unknown; let scriptingCallCount: number; +let scriptingExecuteOverride: + | ((options: { + func?: (...args: unknown[]) => unknown; + args?: unknown[]; + }) => unknown | Promise) + | null; let alarmCreates: Array<{ name: string; when: number }>; let alarmClears: string[]; let alarmListeners: Array<(alarm: { name: string }) => void>; @@ -318,6 +324,7 @@ function installChromeMock(): void { nextGroupId = 50; scriptResult = true; scriptingCallCount = 0; + scriptingExecuteOverride = null; alarmCreates = []; alarmClears = []; alarmListeners = []; @@ -558,9 +565,16 @@ function installChromeMock(): void { async reload() { }, }, scripting: { - async executeScript() { + async executeScript(options: { + func?: (...args: unknown[]) => unknown; + args?: unknown[]; + }) { scriptingCallCount += 1; - return [{ result: scriptResult }]; + return [{ + result: scriptingExecuteOverride + ? await scriptingExecuteOverride(options) + : scriptResult, + }]; }, }, windows: { @@ -1059,6 +1073,43 @@ describe("mutation scheduler", () => { expect(order).toEqual(["writer-start", "other-tab-read", "writer-end", "same-tab-read"]); }); + test("a global observation waits for prior writes without becoming a later write barrier", async () => { + const scheduler = new MutationScheduler(); + const writerGate = Promise.withResolvers(); + const writerStarted = Promise.withResolvers(); + const observationGate = Promise.withResolvers(); + const observationStarted = Promise.withResolvers(); + const order: string[] = []; + const writer = scheduler.enqueueTab(TASK_A, 41, async () => { + order.push("writer-start"); + writerStarted.resolve(); + await writerGate.promise; + order.push("writer-end"); + }); + const observation = scheduler.readAfterAllWrites(async () => { + order.push("observation-start"); + observationStarted.resolve(); + await observationGate.promise; + order.push("observation-end"); + }); + const laterWriter = scheduler.enqueueTab(TASK_B, 42, async () => { + order.push("later-writer"); + }); + + await writerStarted.promise; + await laterWriter; + expect(order).toEqual(["writer-start", "later-writer"]); + + writerGate.resolve(); + await writer; + await observationStarted.promise; + expect(order).toEqual(["writer-start", "later-writer", "writer-end", "observation-start"]); + + observationGate.resolve(); + await observation; + expect(order.at(-1)).toBe("observation-end"); + }); + test("prunes idle tab generations and invalidation reasons", async () => { const scheduler = new MutationScheduler(); const internal = scheduler as unknown as { @@ -1337,6 +1388,192 @@ describe("page revision monotonicity", () => { await runtime.detach(61); }); + test("publishes stable semantic refs only for unique actionable accessibility targets", async () => { + debuggerCommandOverride = (method) => { + if (method !== "Accessibility.getFullAXTree") return undefined; + return { + nodes: [ + { backendDOMNodeId: 22, role: { value: "button" }, name: { value: "Continue" } }, + { backendDOMNodeId: 23, role: { value: "link" }, name: { value: "Documentation" } }, + { backendDOMNodeId: 24, role: { value: "button" }, name: { value: "Delete" } }, + { backendDOMNodeId: 25, role: { value: "button" }, name: { value: "Delete" } }, + { backendDOMNodeId: 26, role: { value: "paragraph" }, name: { value: "Introduction" } }, + ], + }; + }; + const runtime = new StandardBrowserRuntime( + new RevisionTracker(), + async () => undefined, + () => undefined, + async () => undefined, + ); + + const snapshot = await runtime.snapshot(61, { mode: "accessibility" }); + const nodes = snapshot.nodes as Array>; + + expect(nodes[0]).toMatchObject({ + ref: "r1-22", + semantic_ref: "a1:button:Continue", + role: "button", + name: "Continue", + }); + expect(nodes[1]).toMatchObject({ + ref: "r1-23", + semantic_ref: "a1:link:Documentation", + }); + expect(nodes[2].semantic_ref).toBeUndefined(); + expect(nodes[3].semantic_ref).toBeUndefined(); + expect(nodes[4].semantic_ref).toBeUndefined(); + await runtime.detach(61); + }); + + test("resolves a semantic ref after an SPA replaces the underlying DOM node", async () => { + debuggerCommandOverride = (method, params) => { + if (method === "Accessibility.queryAXTree") { + return { + nodes: [ + { backendDOMNodeId: 77, role: { value: "button" }, name: { value: "Continue" } }, + ], + }; + } + if ( + method === "Runtime.callFunctionOn" && + String(params.functionDeclaration).includes("const f=this.form") + ) { + return { result: { value: { tag: "BUTTON", text: "Continue" } } }; + } + return undefined; + }; + const revisions = new RevisionTracker(); + const runtime = new StandardBrowserRuntime( + revisions, + async () => undefined, + () => undefined, + async () => undefined, + ); + const pageRevision = await revisions.ensure(61); + + await expect(runtime.act( + TASK_A, + 61, + pageRevision, + [{ kind: "click", ref: `a${pageRevision}:button:Continue` }], + )).resolves.toMatchObject({ + result: { actions: [{ kind: "click", completed: true }] }, + }); + expect(debuggerCommands.filter( + ({ method, params }) => method === "DOM.resolveNode" && params.backendNodeId === 77, + )).not.toHaveLength(0); + expect(debuggerCommands.filter(({ method }) => method === "Accessibility.queryAXTree")) + .toEqual([{ + method: "Accessibility.queryAXTree", + params: { nodeId: 1, accessibleName: "Continue", role: "button" }, + }]); + await runtime.detach(61); + }); + + test("rejects a semantic ref when its document changes during live resolution", async () => { + let documentReads = 0; + let frameTreeReads = 0; + debuggerCommandOverride = (method) => { + if (method === "DOM.getDocument") { + documentReads += 1; + const backendNodeId = documentReads === 1 ? 1 : 2; + return { root: { nodeId: backendNodeId, backendNodeId } }; + } + if (method === "Page.getFrameTree") { + frameTreeReads += 1; + return { + frameTree: { + frame: { loaderId: frameTreeReads === 1 ? "loader-a" : "loader-b" }, + }, + }; + } + if (method === "Accessibility.queryAXTree") { + return { + nodes: [ + { backendDOMNodeId: 77, role: { value: "button" }, name: { value: "Continue" } }, + ], + }; + } + return undefined; + }; + const revisions = new RevisionTracker(); + const pageRevision = await revisions.observeDocument(61, "backend:1", "loader-a"); + const runtime = new StandardBrowserRuntime( + revisions, + async () => undefined, + () => undefined, + async () => undefined, + ); + + await expect(runtime.act( + TASK_A, + 61, + pageRevision, + [{ kind: "click", ref: `a${pageRevision}:button:Continue` }], + )).rejects.toMatchObject({ + code: "stale_ref", + currentPageRevision: pageRevision + 1, + }); + expect(debuggerCommands.some(({ method }) => + method === "DOM.resolveNode" || method === "Runtime.callFunctionOn", + )).toBe(false); + await runtime.detach(61); + }); + + test("returns actionable candidates when a semantic target becomes ambiguous", async () => { + debuggerCommandOverride = (method) => { + if (method !== "Accessibility.queryAXTree") return undefined; + return { + nodes: [ + { backendDOMNodeId: 40, role: { value: "button" }, name: { value: "Continue" } }, + { backendDOMNodeId: 41, role: { value: "button" }, name: { value: "Continue" } }, + ], + }; + }; + const revisions = new RevisionTracker(); + const runtime = new StandardBrowserRuntime( + revisions, + async () => undefined, + () => undefined, + async () => undefined, + ); + const pageRevision = await revisions.ensure(61); + + await expect(runtime.act( + TASK_A, + 61, + pageRevision, + [{ kind: "click", ref: `a${pageRevision}:button:Continue` }], + )).rejects.toMatchObject({ + code: "ambiguous_target", + details: { + role: "button", + name: "Continue", + match_count: 2, + candidates: [ + { ref: `r${pageRevision}-40` }, + { ref: `r${pageRevision}-41` }, + ], + }, + }); + debuggerCommandOverride = (method) => method === "Accessibility.queryAXTree" + ? { nodes: [] } + : undefined; + await expect(runtime.act( + TASK_A, + 61, + pageRevision, + [{ kind: "click", ref: `a${pageRevision}:button:Continue` }], + )).rejects.toMatchObject({ + code: "target_not_found", + recovery: expect.stringContaining("fresh accessibility snapshot"), + details: { role: "button", name: "Continue", match_count: 0 }, + }); + await runtime.detach(61); + }); + test("retries a content match when navigation replaces the probed document", async () => { tabStore.set(61, { id: 61, @@ -1364,11 +1601,220 @@ describe("page revision monotonicity", () => { timeout_ms: 500, }); - expect(scriptingCallCount).toBe(2); + expect(scriptingCallCount).toBe(3); expect(result).toMatchObject({ matched: true, page_revision: 2 }); await runtime.detach(61); }); + test("wakes a load wait from the Chrome tab event without waiting for its heartbeat", async () => { + vi.useFakeTimers(); + try { + tabStore.set(61, { + id: 61, + windowId: 1, + groupId: -1, + url: "https://example.test/next", + status: "loading", + }); + const runtime = new StandardBrowserRuntime( + new RevisionTracker(), + async () => undefined, + () => undefined, + async () => undefined, + ); + let settled = false; + const waiting = runtime.wait(61, { + condition: { kind: "load" }, + timeout_ms: 1_000, + }).then((result) => { + settled = true; + return result; + }); + await flushPromiseQueue(); + expect(settled).toBe(false); + + const tab = tabStore.get(61); + if (!tab) throw new Error("missing tab"); + tab.status = "complete"; + for (const listener of tabUpdatedListeners) listener(61, { status: "complete" }); + for (let turn = 0; turn < 50; turn += 1) await Promise.resolve(); + + expect(settled).toBe(true); + await expect(waiting).resolves.toMatchObject({ condition: "load", matched: true }); + await runtime.detach(61); + } finally { + vi.useRealTimers(); + } + }); + + test("ignores unrelated tab-update storms while still waking for URL changes", async () => { + vi.useFakeTimers(); + try { + tabStore.set(61, { + id: 61, + windowId: 1, + groupId: -1, + url: "https://example.test/before", + status: "complete", + }); + const runtime = new StandardBrowserRuntime( + new RevisionTracker(), + async () => undefined, + () => undefined, + async () => undefined, + ); + let revalidations = 0; + const waiting = runtime.wait( + 61, + { + condition: { kind: "url", value: "https://example.test/next" }, + timeout_ms: 1_000, + }, + async () => { + revalidations += 1; + }, + ); + for (let turn = 0; turn < 50 && revalidations < 2; turn += 1) { + await Promise.resolve(); + } + const beforeStorm = revalidations; + + for (let index = 0; index < 100; index += 1) { + for (const listener of tabUpdatedListeners) { + listener(61, { title: `Loading ${index}` }); + } + } + await flushPromiseQueue(); + expect(revalidations).toBe(beforeStorm); + + const tab = tabStore.get(61); + if (!tab) throw new Error("missing tab"); + tab.url = "https://example.test/next"; + for (const listener of tabUpdatedListeners) listener(61, { url: tab.url }); + for (let turn = 0; turn < 50 && revalidations === beforeStorm; turn += 1) { + await Promise.resolve(); + } + + expect(revalidations).toBeGreaterThan(beforeStorm); + await expect(waiting).resolves.toMatchObject({ condition: "url", matched: true }); + await runtime.detach(61); + } finally { + vi.useRealTimers(); + } + }); + + test("bounds quiet DOM waits to two mutation-observer slices per second", async () => { + vi.useFakeTimers(); + try { + scriptResult = false; + const runtime = new StandardBrowserRuntime( + new RevisionTracker(), + async () => undefined, + () => undefined, + async () => undefined, + ); + const waiting = runtime.wait(61, { + condition: { kind: "selector", value: "#never" }, + timeout_ms: 1_000, + }); + await flushPromiseQueue(); + await advanceTimers(1_100); + + await expect(waiting).rejects.toMatchObject({ code: "wait_timeout" }); + expect(scriptingCallCount).toBeLessThanOrEqual(5); + await runtime.detach(61); + } finally { + vi.useRealTimers(); + } + }); + + test("coalesces mutation bursts and observes only condition-relevant DOM changes", async () => { + vi.useFakeTimers(); + const priorGlobals = new Map(); + for (const key of ["document", "MutationObserver", "addEventListener", "removeEventListener"]) { + priorGlobals.set(key, { + present: key in globalThis, + value: Reflect.get(globalThis, key), + }); + } + try { + for (const kind of ["text", "selector"] as const) { + let ready = false; + let scans = 0; + let mutationCallback: (() => void) | undefined; + let observerOptions: MutationObserverInit | undefined; + const documentElement = { + get textContent(): string { + scans += 1; + return ready ? "ready" : "pending"; + }, + }; + Reflect.set(globalThis, "document", { + documentElement, + querySelector: () => { + scans += 1; + return ready ? documentElement : null; + }, + }); + Reflect.set(globalThis, "MutationObserver", class { + constructor(callback: () => void) { + mutationCallback = callback; + } + + observe(_target: unknown, options: MutationObserverInit): void { + observerOptions = options; + } + + disconnect(): void { } + }); + Reflect.set(globalThis, "addEventListener", () => undefined); + Reflect.set(globalThis, "removeEventListener", () => undefined); + + scriptingExecuteOverride = async (options) => { + if (options.args?.length !== 4 || !options.func) return ready; + const observerResult = Promise.resolve(options.func(...options.args)); + expect(mutationCallback).toBeDefined(); + for (let index = 0; index < 100; index += 1) mutationCallback?.(); + expect(scans).toBe(2); + + ready = true; + vi.advanceTimersByTime(49); + await flushPromiseQueue(); + expect(scans).toBe(2); + vi.advanceTimersByTime(1); + await flushPromiseQueue(); + expect(scans).toBe(3); + return observerResult; + }; + + const runtime = new StandardBrowserRuntime( + new RevisionTracker(), + async () => undefined, + () => undefined, + async () => undefined, + ); + await expect(runtime.wait(61, { + condition: { kind, value: kind === "text" ? "ready" : "#ready" }, + timeout_ms: 1_000, + })).resolves.toMatchObject({ condition: kind, matched: true }); + expect(observerOptions).toMatchObject({ + attributes: kind === "selector", + characterData: kind === "text", + childList: true, + subtree: true, + }); + await runtime.detach(61); + } + } finally { + scriptingExecuteOverride = null; + for (const [key, prior] of priorGlobals) { + if (prior.present) Reflect.set(globalThis, key, prior.value); + else Reflect.deleteProperty(globalThis, key); + } + vi.useRealTimers(); + } + }); + test("serializes concurrent attachment and keeps a failed detach recoverable", async () => { const runtime = new StandardBrowserRuntime( new RevisionTracker(), @@ -1381,6 +1827,10 @@ describe("page revision monotonicity", () => { runtime.snapshot(62, { mode: "accessibility" }), ]); expect(debuggerCalls.filter((call) => call === "attach")).toHaveLength(1); + for (const domain of ["Page", "Network", "DOM", "Accessibility"]) { + expect(debuggerCalls.filter((call) => call === `${domain}.enable`)).toHaveLength(1); + } + expect(debuggerCalls).not.toContain("Runtime.enable"); debuggerDetachFailures = 1; await expect(runtime.detach(62)).rejects.toThrow("debugger detach failed"); @@ -1447,7 +1897,7 @@ describe("page revision monotonicity", () => { await Promise.resolve(); } expect(debuggerCalls).toContain("Network.enable"); - await advanceTimers(0); + await advanceTimers(1); emitDebuggerEvent(61, "Network.loadingFinished", { requestId: "racing-enable" }); await advanceTimers(550); @@ -1461,7 +1911,7 @@ describe("page revision monotonicity", () => { await advanceTimers(450); expect(settled).toBe(false); - await advanceTimers(100); + await advanceTimers(500); await expect(waiting).resolves.toMatchObject({ tab_id: 61, condition: "network_idle", @@ -1586,6 +2036,85 @@ describe("page revision monotonicity", () => { expect(forgotten).toEqual([66]); }); + test("enables command-only debugger domains once, on first use", async () => { + const runtime = new StandardBrowserRuntime( + new RevisionTracker(), + async () => undefined, + () => undefined, + ); + + await runtime.snapshot(67, { mode: "text" }); + expect(debuggerCommands.filter(({ method }) => method.endsWith(".enable")).map(({ method }) => method)).toEqual([ + "Page.enable", + "Network.enable", + "DOM.enable", + ]); + + await runtime.snapshot(67, { mode: "accessibility" }); + await runtime.snapshot(67, { mode: "accessibility" }); + expect(debuggerCommands.filter(({ method }) => method.endsWith(".enable")).map(({ method }) => method)).toEqual([ + "Page.enable", + "Network.enable", + "DOM.enable", + "Accessibility.enable", + ]); + + await runtime.developer(67, "Runtime.evaluate", { expression: "document.title" }); + expect(debuggerCommands.filter(({ method }) => method.endsWith(".enable")).map(({ method }) => method)).toEqual([ + "Page.enable", + "Network.enable", + "DOM.enable", + "Accessibility.enable", + "Runtime.enable", + ]); + await runtime.detach(67); + }); + + test("recycles the debugger session after a Developer domain disable", async () => { + const runtime = new StandardBrowserRuntime( + new RevisionTracker(), + async () => undefined, + () => undefined, + ); + + await runtime.snapshot(67, { mode: "accessibility" }); + await runtime.developer(67, "Accessibility.disable", {}); + expect(debuggerCalls.filter((call) => call === "detach")).toHaveLength(1); + + await runtime.snapshot(67, { mode: "accessibility" }); + expect(debuggerCalls.filter((call) => call === "attach")).toHaveLength(2); + expect(debuggerCalls.filter((call) => call === "Accessibility.enable")).toHaveLength(2); + await runtime.detach(67); + }); + + test("detaches and recovers when a lazy domain fails to enable", async () => { + let failDomEnable = true; + debuggerCommandOverride = (method) => { + if (method === "DOM.enable" && failDomEnable) { + failDomEnable = false; + throw new Error("DOM domain unavailable"); + } + return undefined; + }; + const runtime = new StandardBrowserRuntime( + new RevisionTracker(), + async () => undefined, + () => undefined, + ); + + await expect(runtime.snapshot(68, { mode: "text" })).rejects.toThrow("DOM domain unavailable"); + expect(debuggerAttachedTabIds.has(68)).toBe(false); + expect(debuggerCalls.filter((call) => call === "detach")).toHaveLength(1); + + await expect(runtime.snapshot(68, { mode: "text" })).resolves.toMatchObject({ + tab_id: 68, + mode: "text", + }); + expect(debuggerCalls.filter((call) => call === "attach")).toHaveLength(2); + expect(debuggerCalls.filter((call) => call === "DOM.enable")).toHaveLength(2); + await runtime.detach(68); + }); + test("authorizes every debugger initialization command", async () => { let authorizationChecks = 0; const runtime = new StandardBrowserRuntime( @@ -1883,6 +2412,95 @@ describe("page revision monotonicity", () => { } }); + test("anchors network idle to a new wait on an already quiet debugger session", async () => { + vi.useFakeTimers(); + try { + const runtime = new StandardBrowserRuntime( + new RevisionTracker(), + async () => undefined, + () => undefined, + async () => undefined, + ); + await runtime.snapshot(61, { mode: "accessibility" }); + await advanceTimers(5_000); + + let settled = false; + const waiting = runtime.wait(61, { + condition: { kind: "network_idle" }, + timeout_ms: 2_000, + }).then((result) => { + settled = true; + return result; + }); + await flushPromiseQueue(); + + await advanceTimers(100); + expect(settled).toBe(false); + emitDebuggerEvent(61, "Network.requestWillBeSent", { requestId: "post-action" }); + await advanceTimers(100); + emitDebuggerEvent(61, "Network.loadingFinished", { requestId: "post-action" }); + + await advanceTimers(450); + expect(settled).toBe(false); + await advanceTimers(100); + await expect(waiting).resolves.toMatchObject({ + tab_id: 61, + condition: "network_idle", + matched: true, + }); + await runtime.detach(61); + } finally { + vi.useRealTimers(); + } + }); + + test("bounds network-event wakeups until the active request burst drains", async () => { + vi.useFakeTimers(); + try { + const runtime = new StandardBrowserRuntime( + new RevisionTracker(), + async () => undefined, + () => undefined, + async () => undefined, + ); + await runtime.snapshot(61, { mode: "accessibility" }); + let revalidations = 0; + const waiting = runtime.wait( + 61, + { condition: { kind: "network_idle" }, timeout_ms: 2_000 }, + async () => { + revalidations += 1; + }, + ); + await flushPromiseQueue(); + for (let turn = 0; turn < 50 && revalidations < 2; turn += 1) { + await Promise.resolve(); + } + const beforeBurst = revalidations; + + for (let index = 0; index < 100; index += 1) { + emitDebuggerEvent(61, "Network.requestWillBeSent", { requestId: `burst-${index}` }); + } + for (let index = 0; index < 99; index += 1) { + emitDebuggerEvent(61, "Network.loadingFinished", { requestId: `burst-${index}` }); + } + await flushPromiseQueue(); + expect(revalidations).toBe(beforeBurst); + + emitDebuggerEvent(61, "Network.loadingFinished", { requestId: "burst-99" }); + await flushPromiseQueue(); + expect(revalidations).toBeGreaterThan(beforeBurst); + await advanceTimers(600); + await expect(waiting).resolves.toMatchObject({ + condition: "network_idle", + matched: true, + }); + await runtime.detach(61); + } finally { + vi.useRealTimers(); + } + }); + test("schedules idle detach after a newly attached network-idle wait completes", async () => { vi.useFakeTimers(); try { @@ -4408,6 +5026,38 @@ describe("extension entrypoint admission boundaries", () => { ), ).toBe(true); + debuggerCommandOverride = (method) => method === "Accessibility.queryAXTree" + ? { + nodes: [ + { backendDOMNodeId: 40, role: { value: "button" }, name: { value: "Continue" } }, + { backendDOMNodeId: 41, role: { value: "button" }, name: { value: "Continue" } }, + ], + } + : undefined; + const ambiguousTarget = await sendNativeCommand( + "018f47b8-2f80-7c20-9c77-f8a38c9e6401", + TASK_A, + "browser_act", + { + tab_id: 100, + expected_page_revision: 1, + actions: [{ kind: "click", ref: "a1:button:Continue" }], + }, + ); + expect(ambiguousTarget).toMatchObject({ + outcome: "not_started", + error: { + code: "ambiguous_target", + details: { + role: "button", + name: "Continue", + match_count: 2, + candidates: [{ ref: "r1-40" }, { ref: "r1-41" }], + }, + }, + }); + debuggerCommandOverride = null; + const staged = await sendNativeCommand( "018f47b8-2f80-7c20-9c77-f8a38c9e6229", diff --git a/packages/mcp/src/server.ts b/packages/mcp/src/server.ts index 6905d94..be2fc25 100644 --- a/packages/mcp/src/server.ts +++ b/packages/mcp/src/server.ts @@ -156,17 +156,17 @@ export const STANDARD_TOOLS: readonly Tool[] = [ }, { name: "browser_snapshot", - description: "Read an accessibility snapshot, bounded text or HTML, or a screenshot from a task-owned tab.", + description: "Read an accessibility snapshot with stable semantic refs, bounded text or HTML, or a screenshot from a task-owned tab.", inputSchema: schema(snapshotSchema), }, { name: "browser_act", - description: "Run an ordered batch of typed actions against one task-owned tab and page revision.", + description: "Run typed actions against one task-owned tab and page revision; prefer a snapshot semantic_ref when available.", inputSchema: schema(actSchema), }, { name: "browser_wait", - description: "Wait for one schema-defined load, URL, text, selector, network-idle, or download condition.", + description: "Wait event-first for one load, URL, text, selector, network-idle, or download condition instead of sleeping.", inputSchema: schema(waitSchema), }, { diff --git a/packages/omp/src/index.ts b/packages/omp/src/index.ts index bbcb4a2..e52fc47 100644 --- a/packages/omp/src/index.ts +++ b/packages/omp/src/index.ts @@ -158,7 +158,7 @@ const DEFINITIONS: ReadonlyArray<{ { name: "browser_snapshot", label: "Browser Snapshot", - description: "Read an accessibility snapshot, bounded text or HTML, or a screenshot from a task-owned tab.", + description: "Read an accessibility snapshot with stable semantic refs, bounded text or HTML, or a screenshot from a task-owned tab.", approval: "read", schema: (z) => z.union([ z.object({ @@ -190,7 +190,7 @@ const DEFINITIONS: ReadonlyArray<{ { name: "browser_act", label: "Browser Act", - description: "Run an ordered batch of typed actions against one task-owned tab and page revision.", + description: "Run typed actions against one task-owned tab and page revision; prefer a snapshot semantic_ref when available.", approval: "write", schema: (z) => { const ref = z.string().min(1).max(256); @@ -231,7 +231,7 @@ const DEFINITIONS: ReadonlyArray<{ { name: "browser_wait", label: "Browser Wait", - description: "Wait for one schema-defined load, URL, text, selector, network-idle, or download condition.", + description: "Wait event-first for one load, URL, text, selector, network-idle, or download condition instead of sleeping.", approval: "read", schema: (z) => z.object({ tab_id: z.number().int().min(0), diff --git a/packages/sdk-python/agenttab/client.py b/packages/sdk-python/agenttab/client.py index cf7efc0..82a56fc 100644 --- a/packages/sdk-python/agenttab/client.py +++ b/packages/sdk-python/agenttab/client.py @@ -59,6 +59,22 @@ def resolve_transport_timeout( return max(request_timeout, operation_timeout + LONG_OPERATION_TRANSPORT_GRACE) +def _inferred_post_action_wait( + actions: object, +) -> JsonObject | None: + if not isinstance(actions, list) or not actions: + return None + last = actions[-1] + if not isinstance(last, Mapping): + return None + kind = last.get("kind") + if kind in {"navigate", "go_back", "go_forward", "reload"}: + return {"kind": "load"} + if kind in {"click", "select", "drag", "dialog", "upload_file"}: + return {"kind": "network_idle"} + return None + + class AgentTabError(RuntimeError): def __init__(self, response: Mapping[str, Any]) -> None: error = response.get("error") @@ -985,6 +1001,59 @@ def call( raise AgentTabError(response) return response.get("result") + def act_wait_observe( + self, + act: Mapping[str, Any], + *, + wait: Mapping[str, Any] | Literal[False] | None = None, + wait_timeout_ms: int | None = None, + observe: Mapping[str, Any] | Literal[False] | None = None, + ) -> JsonObject: + """Act, await a deterministic postcondition, then return a fresh observation.""" + action_response = self.request("browser_act", act) + if not action_response.get("ok"): + raise AgentTabError(action_response) + outcome = str(action_response.get("outcome", "unknown")) + result: JsonObject = { + "outcome": outcome, + "action": action_response.get("result"), + } + if outcome != "completed": + return result + + actions = act.get("actions") + last = actions[-1] if isinstance(actions, list) and actions else None + if isinstance(last, Mapping) and last.get("kind") == "close": + return result + + if wait is False: + condition = None + elif isinstance(wait, Mapping): + condition = dict(wait) + else: + condition = _inferred_post_action_wait(actions) + if condition is not None: + wait_params: JsonObject = { + "tab_id": act["tab_id"], + "condition": condition, + } + if wait_timeout_ms is not None: + wait_params["timeout_ms"] = wait_timeout_ms + result["wait"] = self.call("browser_wait", wait_params) + + if observe is False: + observation = None + elif isinstance(observe, Mapping): + observation = dict(observe) + else: + observation = {"mode": "accessibility"} + if observation is not None: + result["observation"] = self.call( + "browser_snapshot", + {"tab_id": act["tab_id"], **observation}, + ) + return result + def close(self) -> None: if self._closed: return diff --git a/packages/sdk-python/tests/test_client.py b/packages/sdk-python/tests/test_client.py index c25c870..f784df9 100644 --- a/packages/sdk-python/tests/test_client.py +++ b/packages/sdk-python/tests/test_client.py @@ -75,6 +75,81 @@ def test_host_response_frames_remain_capped_at_1_mib(self) -> None: @unittest.skipIf(not hasattr(socket, "AF_UNIX"), "requires Unix sockets") class ClientTests(unittest.TestCase): + def test_act_wait_observe_uses_semantic_settle_loop(self) -> None: + client = object.__new__(AgentTabClient) + calls: list[tuple[str, dict[str, object]]] = [] + + def request(method: str, params: object, **_options: object) -> dict[str, object]: + self.assertIsInstance(params, dict) + calls.append((method, dict(params))) # type: ignore[arg-type] + return { + "ok": True, + "outcome": "completed", + "result": {"method": method}, + } + + def call(method: str, params: object, **_options: object) -> dict[str, object]: + self.assertIsInstance(params, dict) + calls.append((method, dict(params))) # type: ignore[arg-type] + return {"method": method} + + client.request = request # type: ignore[method-assign] + client.call = call # type: ignore[method-assign] + result = client.act_wait_observe({ + "tab_id": 7, + "expected_page_revision": 3, + "actions": [{"kind": "click", "ref": "a3:button:Continue"}], + }) + + self.assertEqual(result, { + "outcome": "completed", + "action": {"method": "browser_act"}, + "wait": {"method": "browser_wait"}, + "observation": {"method": "browser_snapshot"}, + }) + self.assertEqual([method for method, _ in calls], [ + "browser_act", + "browser_wait", + "browser_snapshot", + ]) + self.assertEqual(calls[1][1], { + "tab_id": 7, + "condition": {"kind": "network_idle"}, + }) + self.assertEqual(calls[2][1], {"tab_id": 7, "mode": "accessibility"}) + + def test_act_wait_observe_stops_on_non_completed_success(self) -> None: + for outcome in ("commit_required", "needs_user"): + with self.subTest(outcome=outcome): + client = object.__new__(AgentTabClient) + + def request( + _method: str, + _params: object, + **_options: object, + ) -> dict[str, object]: + return { + "ok": True, + "outcome": outcome, + "result": {"stopped": outcome}, + } + + def call(*_args: object, **_options: object) -> object: + self.fail("wait or observation must not start") + + client.request = request # type: ignore[method-assign] + client.call = call # type: ignore[method-assign] + + result = client.act_wait_observe({ + "tab_id": 7, + "expected_page_revision": 3, + "actions": [{"kind": "click", "ref": "a3:button:Continue"}], + }) + self.assertEqual(result, { + "outcome": outcome, + "action": {"stopped": outcome}, + }) + def test_long_operation_transport_deadlines_follow_protocol_timeouts(self) -> None: self.assertEqual( resolve_transport_timeout( diff --git a/packages/sdk-typescript/src/index.ts b/packages/sdk-typescript/src/index.ts index b0cb289..3f97f4f 100644 --- a/packages/sdk-typescript/src/index.ts +++ b/packages/sdk-typescript/src/index.ts @@ -71,6 +71,37 @@ export interface BrowserWaitParams { timeout_ms?: number; } +export type BrowserObservation = + | { mode: "accessibility"; root_ref?: string; max_depth?: number; max_nodes?: number } + | { mode: "text" | "html"; selector?: string; max_bytes?: number } + | { + mode: "screenshot"; + selector?: string; + full_page?: boolean; + format?: "png" | "jpeg" | "webp"; + quality?: number; + max_width?: number; + max_height?: number; + max_bytes?: number; + }; + +export interface ActWaitObserveOptions { + act: BrowserActParams; + /** Omit for an action-aware default, or pass false to observe immediately. */ + wait?: BrowserWaitCondition | false; + waitTimeoutMs?: number; + /** Defaults to a fresh accessibility snapshot; pass false to skip observation. */ + observe?: BrowserObservation | false; +} + +export interface ActWaitObserveResult { + /** Core outcome from browser_act. Wait and observation run only when this is completed. */ + outcome: Outcome; + action: unknown; + wait?: unknown; + observation?: unknown; +} + export interface BrowserHandoffParams { tab_id: number; expected_page_revision: number; @@ -102,6 +133,29 @@ export interface MethodParams { "agenttab.status": Record; } +function inferredPostActionWait(actions: readonly BrowserAction[]): BrowserWaitCondition | undefined { + const last = actions.at(-1); + if (!last || last.kind === "close") return undefined; + if ( + last.kind === "navigate" || + last.kind === "go_back" || + last.kind === "go_forward" || + last.kind === "reload" + ) { + return { kind: "load" }; + } + if ( + last.kind === "click" || + last.kind === "select" || + last.kind === "drag" || + last.kind === "dialog" || + last.kind === "upload_file" + ) { + return { kind: "network_idle" }; + } + return undefined; +} + export type RpcMethod = keyof MethodParams; export type MutationMethod = | "browser_open" @@ -798,6 +852,48 @@ export class AgentTabClient { return response.result as T; } + /** + * Run the common semantic automation loop without caller-authored sleeps: + * act, wait for a deterministic postcondition, then return a fresh observation. + */ + async actWaitObserve(options: ActWaitObserveOptions): Promise { + const actionResponse = await this.request("browser_act", options.act); + if (!actionResponse.ok) throw new AgentTabError(actionResponse); + const actionResult: ActWaitObserveResult = { + outcome: actionResponse.outcome, + action: actionResponse.result, + }; + if (actionResponse.outcome !== "completed") return actionResult; + + const lastAction = options.act.actions.at(-1); + if (lastAction?.kind === "close") return actionResult; + + const condition = options.wait === false + ? undefined + : options.wait ?? inferredPostActionWait(options.act.actions); + const waited = condition === undefined + ? undefined + : await this.call("browser_wait", { + tab_id: options.act.tab_id, + condition, + ...(options.waitTimeoutMs === undefined ? {} : { timeout_ms: options.waitTimeoutMs }), + }); + const observationSpec = options.observe === false + ? undefined + : options.observe ?? { mode: "accessibility" as const }; + const observation = observationSpec === undefined + ? undefined + : await this.call("browser_snapshot", { + tab_id: options.act.tab_id, + ...observationSpec, + } as BrowserSnapshotParams); + return { + ...actionResult, + ...(waited === undefined ? {} : { wait: waited }), + ...(observation === undefined ? {} : { observation }), + }; + } + close(): void { if (this.#closed) return; this.#closed = true; diff --git a/packages/sdk-typescript/test/client.test.ts b/packages/sdk-typescript/test/client.test.ts index e8eae49..270c619 100644 --- a/packages/sdk-typescript/test/client.test.ts +++ b/packages/sdk-typescript/test/client.test.ts @@ -157,6 +157,77 @@ describe("Core RPC transport deadlines", () => { }); }); +test("actWaitObserve replaces post-action sleeps with wait and fresh observation", async () => { + const requests: Array<{ method: string; params: unknown }> = []; + const client = Object.create(AgentTabClient.prototype) as AgentTabClient; + client.request = (async (method: string, params: unknown) => { + requests.push({ method, params }); + return { + protocol: "agenttab.rpc", + version: 1, + request_id: "act-request", + ok: true, + outcome: "completed", + result: { method }, + }; + }) as AgentTabClient["request"]; + client.call = (async (method: string, params: unknown) => { + requests.push({ method, params }); + return { method }; + }) as AgentTabClient["call"]; + + await expect(client.actWaitObserve({ + act: { + tab_id: 7, + expected_page_revision: 3, + actions: [{ kind: "click", ref: "a3:button:Continue" }], + }, + })).resolves.toEqual({ + outcome: "completed", + action: { method: "browser_act" }, + wait: { method: "browser_wait" }, + observation: { method: "browser_snapshot" }, + }); + expect(requests.map(({ method }) => method)).toEqual([ + "browser_act", + "browser_wait", + "browser_snapshot", + ]); + expect(requests[1].params).toEqual({ + tab_id: 7, + condition: { kind: "network_idle" }, + }); + expect(requests[2].params).toEqual({ tab_id: 7, mode: "accessibility" }); +}); + +test("actWaitObserve stops on non-completed successful action outcomes", async () => { + for (const outcome of ["commit_required", "needs_user"] as const) { + const client = Object.create(AgentTabClient.prototype) as AgentTabClient; + client.request = (async () => ({ + protocol: "agenttab.rpc", + version: 1, + request_id: `${outcome}-request`, + ok: true, + outcome, + result: { stopped: outcome }, + })) as AgentTabClient["request"]; + client.call = (async () => { + throw new Error("wait or observation must not start"); + }) as AgentTabClient["call"]; + + await expect(client.actWaitObserve({ + act: { + tab_id: 7, + expected_page_revision: 3, + actions: [{ kind: "click", ref: "a3:button:Continue" }], + }, + })).resolves.toEqual({ + outcome, + action: { stopped: outcome }, + }); + } +}); + if (false) { // @ts-expect-error Standard browser actions never expose an agent-controlled focus transition. const forbiddenFocusAction: BrowserAction = { kind: "focus" };