diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b1294fe7a..77f4ed8f0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -388,6 +388,7 @@ jobs: src/tools/__tests__/background-full-page.browser.test.ts src/session-manager/__tests__/incognito.browser.test.ts src/transport/__tests__/remote-connection.browser.test.ts + src/entrypoints/__tests__/runtime-injection.browser.test.ts - name: Run headed background execution, screenshot and window-routing regressions run: | sudo apt-get update diff --git a/CHANGELOG.md b/CHANGELOG.md index a27a2af9c..ea4d2441a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,7 @@ determines which versions can connect. ### Changed +- **Zero-tax Chrome MV3 `registration: "runtime"` dual-tier content script injection, SW-startup orphan Agent Window reaping, and shared `content/dom-primitives.ts`**: Migrated all four MV3 content-script entrypoints (`content`, `cooperation.content`, `record-frame.content`, and `long-screenshot-page.content`) out of static `manifest.json` `content_scripts` into idempotent runtime injection (`claimIdempotentContentScript`) with `noScriptStartedPostMessage: true` so ordinary browsing incurs zero content-script, `window.postMessage`, or React 19 overhead; injected Tier A (`content.js` and `long-screenshot-page.js`) per `tabId`/`documentId` via idempotent `ensureOverlayInjected` across controlled task tabs, `webNavigation.onCommitted` (including Speculation Rules prerender outermost frames, sequenced strictly after Tier B `cooperation.js` arming/repair), pre-confirmation `tab_borrow` candidate user tabs, `request_help`, `record`, and active `video` overlays while skipping uninjected tabs on `hidden` / `CAPTURE_SUPPRESS` / `INPUT_PASSTHROUGH`; dynamically registered/unregistered pure-TypeScript Tier B (`cooperation.js` and `record-frame.js`) with `persistAcrossSessions: false` when active sessions or recordings transition between `0` and `>= 1`, loading silently at `document_start` with 0 DOM listeners and 0 startup IPC until armed by Background (`COOPERATION_ARM_FRAME` / `RECORD_FRAME_START` plus 50ms per-tab subframe coalescing, `cooperation.js` co-injection in `injectRecordFrameAllFrames`, and non-destructive per-frame `injectCooperationAllFrames` repair across `tab_borrow`, `ack === undefined` frames, and dynamically inserted light-DOM or open/closed `ShadowRoot` `about:blank` / `srcdoc` subframes); tracked dedicated Agent Window `ownedTabIds` in `chrome.storage.session` (`bsk.orphan_agent_windows`) to reap orphaned pure Agent Windows on Service Worker restart without ever closing windows containing user, borrowed, or human-preserved tabs; and extracted `apps/extension/src/content/dom-primitives.ts`. - **Single daemon `tool_pipeline`, 4-module `daemon/ipc` decomposition, non-blocking persistence, and shared `cli::orchestration` kernel**: Decomposed 3,489-line `daemon/ipc.rs` into `daemon/tool_pipeline.rs` (single unified `tool.*` pipeline with RAII `AuditTicketGuard`, `SessionActivityGuard`, `ToolInflightRegistry`, `IntentWal`, and post-settlement scope/run-log reconciliation), `daemon/ipc_sessions.rs`, `daemon/ipc_transport.rs`, and a slim `daemon/ipc.rs` router; promoted exhaustive `OutcomeKind` (`bsk-protocol`) / `Outcome` (`daemon/tool_policy.rs`) routing in `daemon/queue.rs`; moved `TransferRegistry` staging/import/session-teardown I/O, `AuditStore` JSONL `sync_data()` writes (including `session.user_interrupt` markers), and monotonic-sequence `IntentWal` flushes/atomic `sync_data()` compactions out of global state `Mutex`es and Tokio worker threads via `spawn_blocking`; and extracted `cli/orchestration.rs` to share Canvas screenshot point-click validation, `open`/`sync`/action/positive timeout bounds, explicit `--session` + `--browser` conflict checks, cached VOM hydration, and 2-second `CANCEL_RESPONSE_GRACE` settlement across `bsk` CLI and `bsk mcp`. - **100% strongly typed `bsk-protocol` contracts, typed `EventFrame` payloads, out-of-band WS correlation, and native `ActionMetadata` SSOT**: Replaced all 19 remaining `serde_json::Value` RPC contracts in `Method::contract_meta` (`ToolCooperate`, `LocalAuth*`, `AuditRequest`, `LedgerRequest`, `Session*`, `BrowserList`, `ToolOperationAwait`, and `Transfer*`) with strict Rust/JSON Schema structs, replaced untyped `EventFrame.payload: Value` across Rust and TypeScript with an exhaustive 10-variant `EventFrame` discriminated union, moved `_audit_id` and `_operation_id` out of business `params` into top-level `RequestFrame` envelope fields (deleting in-place `params` mutation and `stripEnvelopeParams`), embedded `ActionMetadata` / passive `context_id` natively with `#[serde(deny_unknown_fields)]` across all 23 `Full` and 9 passive-read `*Params` structs (eliminating post-hoc `catalog.rs` JSON Schema surgery and manual 4-key copy/re-insertion in `daemon/ipc.rs`), and unified `MethodSpec.action_metadata` (`Full` / `ContextOnly` / `TabBindingContext` / `Dynamic` / `None`) as the single source of truth across `bsk-protocol` and `bsk-cli`. - **Unified daemon session teardown, strongly typed `IpcEnvelope`, and MCP host-scope lifecycle**: Converged all 6 session teardown paths across `state.rs`, `ws.rs`, `ipc.rs`, `start.rs`, `session_requests.rs`, and `sessions.rs` into `DaemonState::teardown_session(session_id, reason)` covering all 12 session-associated state tables, promoted `HOST_AGENT_ENV_KEYS` ahead of transient `mcp::` scopes so `bsk mcp` and `bsk` CLI within the same host agent thread share one implicit session (while immediately stopping orphan sessions on `stdio` EOF or disconnect read errors only for exclusive transient `mcp:` scopes), added 256-entry LRU eviction for inactive scope tombstones and idle `start_locks`, and replaced private `_`-prefixed IPC JSON fields with strongly typed `IpcEnvelope` metadata (`IpcRequestMeta` / `IpcResponseMeta`) negotiated via `StatusResult.ipc_envelope` and `DaemonInfo.ipc_envelope` (preserving bidirectional rolling-upgrade compatibility with pre-envelope `"2.0"` `bsk mcp`, CLI, and daemon processes) plus explicit extension-WS `WsCorrelation`. @@ -41,6 +42,7 @@ determines which versions can connect. ### Fixed +- Ignored internal `data-bsk-*` attributes and `` (`[data-bsk-overlay]`) ShadowHost mutations in `cooperation-capture.ts` so active recording and sensitive-field markers no longer advance `FrameState.structure`, excluded extension ShadowRoot UI controls (`HelpRequestOverlay`, `RecordOverlay`, `BorrowConfirmationOverlay`) across `record-capture.ts`, `record-hover-surface.ts`, and `cooperation-capture.ts` so overlay interactions and hover surfaces are never mis-recorded as webpage steps, and re-evaluated tab control ownership after awaiting `ensureOverlayInjected` so mid-injection tab detachments never flash stale `control` state. - VOM token estimation now handles unpaired UTF-16 surrogates without combining them with the following symbol or throwing a `RangeError`; valid emoji pairs retain their existing estimates. - Fixed non-`start` `tool.video` (`status`/`stop`/`list`/`get`/`delete`), `tool.operation_await`, and scope-bound `tool.wait_ms` bypassing `ToolInflightRegistry` cancellation (and scope-bound `tool.wait_ms` omitting `SessionActivityGuard` idle leases or `ToolDialogAccept`/`ToolDialogDismiss` `decision_lock` waits masking `UserAborted`/`SessionEnded` reasons) on `session.user_interrupt` and `DaemonState::teardown_session` (`session.window_closed` / idle reap / disconnect), fixed `IntentWal` `flush_pending_async` returning before concurrently drained `DISPATCHED` records hit disk and `on_session_ended` leaving un-reconciled `dispatched`/`unknown` records on disk across daemon restarts, fixed `queue.rs` misattributing deadline cancel-cleanup timeouts on non-transfer tools (`navigate`, `screenshot_full_page`, `window_resize`, etc.) as `"file transfer timed out"` (`transfer_outcome_unknown`) and stripping extension `effect_state` / `cancelled_before_dispatch` metadata on cancelled file transfers (including when terminal session teardown reasons are present), fixed Canvas screenshot point-click and zero-timeout validation drift across `bsk` CLI and MCP (`bsk_interact`, `bsk_page`, `bsk_inspect`, `bsk_tabs`, `bsk_assist`, `bsk sync --wait-for/--wait-not --timeout-ms` up to `120000`ms, and structured `scope_browser_conflict` errors on MCP `bsk_session(open)`), and fixed MCP `notifications/cancelled` dropping in-flight IPC futures at `0`ms without awaiting `CANCEL_RESPONSE_GRACE` (recovering real `effect_state` receipts and releasing staged upload files on `cancelled_before_dispatch`). @@ -100,6 +102,7 @@ determines which versions can connect. ### Removed +- Removed dead write-only `apps/extension/src/lib/sessions-live-flag.ts` (`bh.sessions_live` / `SESSIONS_LIVE_FLAG_KEY`) and ineffective Isolated World `history.pushState` / `history.replaceState` monkey-patching in `apps/extension/src/content/record-capture.ts`. - Removed `reattachIfSoftDetached`, `reconcileSoftDetachedFrames`, and `resolveTarget` compensation hooks from `CdpRunner`, `RefStore.remapChildFrameSessions`, partial hand-written `CdpRunner` wrapper object literals (replaced with full `delegateCdpRunner` delegation), and isolated default `new ChromiumCdp()` fallback instances outside `background.ts`. - Removed deprecated `TraceV2` recording types/converters (retaining `TraceV3` only), all 9 legacy `1.x` `Feature::*` compatibility gates, extension/daemon `"1.6"`/`LEGACY_MIN_COMPATIBLE_PEER` shims, duplicate `DragParams.observation_id` / legacy drag modes, and duplicate `wire_*` runtime validators from generated TypeScript artifacts. - Retired and removed `packages/dsh-plugin-browserskill`, its standalone release diff --git a/CHANGELOG.zh-CN.md b/CHANGELOG.zh-CN.md index 1a93e676f..2926d026c 100644 --- a/CHANGELOG.zh-CN.md +++ b/CHANGELOG.zh-CN.md @@ -17,6 +17,7 @@ CLI 和扩展可以分别发布,能否连接由协议兼容性决定。更新 ### 变更 +- **Chrome MV3 `registration: "runtime"` 零税双层按需内容脚本注入、SW 启动期孤儿 Agent 窗口兜底回收与共享 `content/dom-primitives.ts`**:将全部 4 个 MV3 内容脚本入口(`content`、`cooperation.content`、`record-frame.content` 与 `long-screenshot-page.content`)以 `noScriptStartedPostMessage: true` 与 `claimIdempotentContentScript` 幂等守卫迁出静态 `manifest.json` `content_scripts`,彻底消除日常浏览普通网页时的全局 `` 内容脚本、`window.postMessage` 广播与 React 19 加载税;通过幂等 `ensureOverlayInjected` 对受控任务标签页、`webNavigation.onCommitted`(含 Speculation Rules 预渲染顶层帧,且严格排在 Tier B `cooperation.js` 激活/补注入之后)、确认前 `tab_borrow` 候选用户标签页、`request_help`、`record` 与活跃 `video` 浮层按 `tabId`/`documentId` 单点注入 Tier A(`content.js` 与 `long-screenshot-page.js`),并在 `hidden` / `CAPTURE_SUPPRESS` / `INPUT_PASSTHROUGH` 时跳过未注入标签页;在活跃会话或录制于 `0` 与 `>= 1` 间切换时以 `persistAcrossSessions: false` 动态启停纯 TS 轻量 Tier B(`cooperation.js` 与 `record-frame.js`),使其在 `document_start` 零 DOM 监听、零启动唤醒 IPC 静默加载并由后台按需点名激活(`COOPERATION_ARM_FRAME` / `RECORD_FRAME_START` 配合单标签页 50ms 子帧合并去抖、`injectRecordFrameAllFrames` 协同注入 `cooperation.js`,以及在 `tab_borrow`、`ack === undefined` 帧与动态插入的普通 DOM 或 open/closed `ShadowRoot` 内同源 `about:blank` / `srcdoc` 子帧上进行无损按帧探测补注入);通过 `chrome.storage.session`(`bsk.orphan_agent_windows`)追踪独占 Agent 窗口创建的标签页集合,在 Service Worker 重启后精准回收无活跃会话的纯孤儿 Agent 窗口(严格保留含用户、借用或人工接管标签页的窗口);并抽离共享 DOM 原语模块 `apps/extension/src/content/dom-primitives.ts`。 - **守护进程单一 `tool_pipeline` 工具管线、`daemon/ipc` 四模块拆解、无锁阻塞持久化与 `cli::orchestration` 客户端共享编排内核**:将 3,489 行的 `daemon/ipc.rs` 拆解为 `daemon/tool_pipeline.rs`(统一收口全部 `tool.*` 执行路径的 RAII `AuditTicketGuard`、`SessionActivityGuard`、`ToolInflightRegistry`、`IntentWal` 与作用域/运行日志结算)、`daemon/ipc_sessions.rs`、`daemon/ipc_transport.rs` 及精简路由层 `daemon/ipc.rs`;在 `bsk-protocol` 与 `daemon/tool_policy.rs` / `daemon/queue.rs` 中建立穷尽式 `OutcomeKind` / `Outcome` 结果与超时归因路由;通过局部状态分离与 `spawn_blocking` 将 `TransferRegistry` 大文件暂存/导入复制/会话销毁目录清理、`AuditStore` JSONL `sync_data()` 刷盘(含 `session.user_interrupt` 标记)与单调序号 `IntentWal` 刷盘及 `sync_data()` 原子压缩彻底移出全局状态 `Mutex` 与 Tokio 工作线程阻塞路径;并提取 `cli/orchestration.rs` 统一 `bsk` CLI 与 `bsk mcp` 的 Canvas 截图坐标点击校验、`open`/`sync`/动作/正数超时边界、显式 `--session` + `--browser` 冲突校验、缓存 VOM 回填以及 2 秒 `CANCEL_RESPONSE_GRACE` 优雅取消结算。 - **`bsk-protocol` 100% 强类型化、强类型 `EventFrame` 事件载荷、WS 信封相关性解耦与原生 `ActionMetadata` 单一事实源**:将 `Method::contract_meta` 中剩余全部 19 个 `serde_json::Value` RPC 契约(`ToolCooperate`、`LocalAuth*`、`AuditRequest`、`LedgerRequest`、`Session*`、`BrowserList`、`ToolOperationAwait` 与 `Transfer*`)收口为编译期强类型 Rust/JSON Schema 结构;将跨 Rust 与 TypeScript 的裸 `EventFrame.payload: Value` 升级为穷尽覆盖全部 10 类事件的强类型 `EventFrame` 判别联合;将扩展 WS 请求上的 `_audit_id` 与 `_operation_id` 迁出业务 `params` 并提升至 `RequestFrame` 顶层可选字段(彻底移除原地篡改 `params` 与扩展侧 `stripEnvelopeParams` 补偿剥离);在全部 23 个 `Full` 与 9 个被动只读 `*Params` 结构中原生内嵌 `ActionMetadata` / `context_id` 并统一启用 `#[serde(deny_unknown_fields)]`(净删 `catalog.rs` 事后 JSON Schema 拼接补丁与 `daemon/ipc.rs` 手工拷出再塞回 4 个元字段的胶水代码);并将 `MethodSpec.action_metadata`(`Full` / `ContextOnly` / `TabBindingContext` / `Dynamic` / `None`)打通为 `bsk-protocol` 与 `bsk-cli` 共享的唯一真源。 - **统一守护进程会话销毁入口、强类型 `IpcEnvelope` 与 MCP 宿主作用域生命周期**:将 `state.rs`、`ws.rs`、`ipc.rs`、`start.rs`、`session_requests.rs` 与 `sessions.rs` 中分散的 6 处会话清理路径统一收口至覆盖全部 12 张会话状态表的 `DaemonState::teardown_session(session_id, reason)`;在 `ImplicitScopeRegistry` 中将宿主子代理环境变量(`HOST_AGENT_ENV_KEYS`)优先级提升至瞬态 `mcp::` 之前,使同一宿主线程内的 `bsk mcp` 与 `bsk` CLI 自动共享隐式会话,同时仅对无宿主环境变量兜底的独占瞬态 `mcp:` 连接在 `stdio` EOF 或读取异常断开时立即停止并回收孤儿会话;为非活跃作用域墓碑与空闲 `start_locks` 增加 256 容量上限的 LRU 淘汰;并以通过 `StatusResult.ipc_envelope` 与 `DaemonInfo.ipc_envelope` 协商的强类型 `IpcEnvelope`(`IpcRequestMeta` / `IpcResponseMeta`,保留与升级期间未重启的 `"2.0"` 旧版 `bsk mcp`、CLI 及守护进程间的双向滚动升级兼容)及显式扩展 WS `WsCorrelation` 替代本地 IPC 载荷混装的私有 `_` 字段。 @@ -28,6 +29,7 @@ CLI 和扩展可以分别发布,能否连接由协议兼容性决定。更新 ### 修复 +- 在 `cooperation-capture.ts` 中忽略扩展内部私有 `data-bsk-*` 属性与 ``(`[data-bsk-overlay]`)ShadowHost 变更以消除录制/敏感字段标记对 `FrameState.structure` 的自污染,在 `record-capture.ts`、`record-hover-surface.ts` 与 `cooperation-capture.ts` 中统一过滤扩展自身 ShadowRoot 浮层控件(`HelpRequestOverlay`、`RecordOverlay`、`BorrowConfirmationOverlay`)防止浮层内点击/输入或悬停面被误录为网页操作,并在 `ensureOverlayInjected` 异步注入完成后二次校验标签页受控归属以杜绝注入途中断开控制导致的陈旧浮层闪现。 - VOM Token 估算现在能处理未配对的 UTF-16 代理字符,不会再误合并后续符号或抛出 `RangeError`;有效 emoji 代理对的估算保持不变。 - 修复非 `start` 的 `tool.video`(`status`/`stop`/`list`/`get`/`delete`)、`tool.operation_await` 与作用域绑定的 `tool.wait_ms` 未接入 `ToolInflightRegistry` 而在 `session.user_interrupt` 及 `DaemonState::teardown_session`(`session.window_closed` / 空闲超时回收 / 断连)时无法立即唤醒取消与转发 WS `cancel` 的问题(同时补齐作用域绑定 `tool.wait_ms` 的 `SessionActivityGuard` 空闲租约,并修复 `ToolDialogAccept`/`ToolDialogDismiss` 等待 `decision_lock` 被中断或销毁时丢失 `UserAborted`/`SessionEnded` 错误原因的问题),修复 `IntentWal` `flush_pending_async` 在并发已被取出的 `DISPATCHED` 批次落盘前提前返回以及 `on_session_ended` 未将会话结束对账持久化导致 daemon 重启后误冻已关闭会话的竞态,修复 `queue.rs` 在非文件传输工具(`navigate`、`screenshot_full_page`、`window_resize` 等)发生超时取消清理挂起时被兜底分支误报为 `"file transfer timed out"`(`transfer_outcome_unknown`)以及在文件传输取消清理完成(含携带会话终止 `reason` 时)丢弃扩展侧 `effect_state` / `cancelled_before_dispatch` 元数据的缺陷,修复 `bsk` CLI 与 MCP(`bsk_interact`、`bsk_page`、`bsk_inspect`、`bsk_tabs`、`bsk_assist`)间的 Canvas 截图坐标点击与零值超时校验漂移(同时对齐 `bsk sync --wait-for/--wait-not --timeout-ms` `120000`ms 上限与 MCP `bsk_session(open)` 的结构化 `scope_browser_conflict` 错误),并修复 MCP `notifications/cancelled` 在 `0`ms 直接丢弃在途 IPC 请求而丢失真实 `effect_state` 及在 `cancelled_before_dispatch` 时泄漏已暂存上传文件的缺陷。 @@ -48,6 +50,7 @@ CLI 和扩展可以分别发布,能否连接由协议兼容性决定。更新 ### 移除 +- 移除已沦为只写不读死代码的 `apps/extension/src/lib/sessions-live-flag.ts`(`bh.sessions_live` / `SESSIONS_LIVE_FLAG_KEY`)以及 `apps/extension/src/content/record-capture.ts` 中在隔离世界无效的 `history.pushState` / `history.replaceState` 重写死代码。 - 从 `CdpRunner` 及上层调用栈净删 `reattachIfSoftDetached`、`reconcileSoftDetachedFrames` 与 `resolveTarget` 三大外泄补偿钩子,移除 `RefStore.remapChildFrameSessions`、手写残缺局部 `CdpRunner` 对象字面量(统一收口为 `delegateCdpRunner` 全量委托代理)以及 `background.ts` 以外所有孤立默认的 `new ChromiumCdp()` 野实例。 - 移除已废弃的 `TraceV2` 录制协议类型与转换器(仅保留 `TraceV3`)、全部 9 套 `1.x` `Feature::*` 历史版本协商门禁、扩展/守护进程 `"1.6"`/`LEGACY_MIN_COMPATIBLE_PEER` 兼容兜底、重复字段 `DragParams.observation_id` / 旧拖拽模式以及生成 TS 产物中重复导出的 `wire_*` 运行时校验器。 - 下线并整体移除 `packages/dsh-plugin-browserskill`、独立发版流水线及专属测试桩;DeepSeek Harness diff --git a/apps/extension/src/content/__tests__/cooperation-capture.browser.test.ts b/apps/extension/src/content/__tests__/cooperation-capture.browser.test.ts index 050ce0411..2ab420ca1 100644 --- a/apps/extension/src/content/__tests__/cooperation-capture.browser.test.ts +++ b/apps/extension/src/content/__tests__/cooperation-capture.browser.test.ts @@ -375,14 +375,15 @@ globalThis.httpCooperationReady = true;`, const repaired = await collectCooperationFrames(delayed.id, COOPERATION_READ, "late"); expect(executeScript).toHaveBeenCalledOnce(); const repairedDocument = repaired.frames.get(child.frameId)!.document; - // Let the delayed declared script run after the repair. WXT must - // dispose the previous collector rather than leave both listening. + // Let the delayed declared script run after the repair. The idempotent + // mount guard must preserve the active collector rather than wipe its + // document/lease or leave two collectors listening. await evaluate( `chrome.scripting.executeScript({target:{tabId:${delayed.id},documentIds:[${JSON.stringify(child.documentId)}]},func:()=>globalThis.fixtureStartCapture()})`, ); const declared = await collectCooperationFrames(delayed.id, COOPERATION_READ, "late"); const declaredDocument = declared.frames.get(child.frameId)!.document; - expect(declaredDocument).not.toBe(repairedDocument); + expect(declaredDocument).toBe(repairedDocument); await evaluate("globalThis.frameUpdates = []"); await clickFrame("/late-frame"); const afterInput = await collectCooperationFrames(delayed.id, COOPERATION_READ, "late"); diff --git a/apps/extension/src/content/__tests__/cooperation-capture.test.ts b/apps/extension/src/content/__tests__/cooperation-capture.test.ts index c0f89dfdf..7f34fb959 100644 --- a/apps/extension/src/content/__tests__/cooperation-capture.test.ts +++ b/apps/extension/src/content/__tests__/cooperation-capture.test.ts @@ -2,9 +2,10 @@ import { afterEach, expect, it, vi } from "vitest"; import { guardedRunner } from "@/cooperation/action-guard"; import { COOPERATION_ARM, + COOPERATION_ARM_FRAME, COOPERATION_DISABLE, COOPERATION_READ, - COOPERATION_READY, + COOPERATION_SUBFRAME_READY, type FrameState, type InputExpectation, } from "@/cooperation/types"; @@ -918,10 +919,141 @@ it("preserves editing across a focus transition into a shadow field", async () = expect(read()).toMatchObject({ editing: false }); }); -it("announces renderer readiness once after the collection listener exists", () => { +it("loads silently with zero startup messages and zero DOM listeners until explicitly armed", async () => { + const closedRoots = new WeakMap(); + document.body.innerHTML = + '
'; + const preHost = document.querySelector("#preexisting-closed")!; + const preClosed = preHost.attachShadow({ mode: "closed" }); + closedRoots.set(preHost, preClosed); + + const read = collector((el) => closedRoots.get(el) ?? null); + expect(chrome.runtime.sendMessage).not.toHaveBeenCalled(); + + // Unarmed mutations and human pointer events are ignored before COOPERATION_ARM_FRAME / COOPERATION_READ. + const field = document.querySelector("input")!; + const press = new MouseEvent("pointerdown", { bubbles: true, clientX: 5, clientY: 5 }); + Object.defineProperty(press, "isTrusted", { value: true }); + field.dispatchEvent(press); + document.body.appendChild(document.createElement("span")).textContent = "unarmed"; + + expect(read(COOPERATION_ARM_FRAME)).toEqual({ ok: true }); + + // Once armed (even before any `COOPERATION_READ` grants a lease or traverses `controls()`), + // dynamically inserted light-DOM and closed-ShadowRoot subframes emit `COOPERATION_SUBFRAME_READY`. + const firstFrame = document.createElement("iframe"); + const secondFrame = document.createElement("iframe"); + document.body.append(firstFrame, secondFrame); + await vi.waitFor( + () => { + expect(chrome.runtime.sendMessage).toHaveBeenCalledExactlyOnceWith({ + kind: COOPERATION_SUBFRAME_READY, + }); + }, + { interval: 5 }, + ); + + // Pre-existing closed ShadowRoot observed solely by `COOPERATION_ARM_FRAME` reports dynamic iframes. + vi.mocked(chrome.runtime.sendMessage).mockClear(); + preClosed.appendChild(document.createElement("iframe")); + await vi.waitFor( + () => { + expect(chrome.runtime.sendMessage).toHaveBeenCalledExactlyOnceWith({ + kind: COOPERATION_SUBFRAME_READY, + }); + }, + { interval: 5 }, + ); + + // Dynamically inserted closed ShadowRoot host containing an iframe reports `COOPERATION_SUBFRAME_READY` + // AND continues observing subsequent iframe insertions inside that closed ShadowRoot. + vi.mocked(chrome.runtime.sendMessage).mockClear(); + const dynamicHost = document.createElement("div"); + const dynamicClosed = dynamicHost.attachShadow({ mode: "closed" }); + closedRoots.set(dynamicHost, dynamicClosed); + dynamicClosed.appendChild(document.createElement("iframe")); + document.body.appendChild(dynamicHost); + await vi.waitFor( + () => { + expect(chrome.runtime.sendMessage).toHaveBeenCalledExactlyOnceWith({ + kind: COOPERATION_SUBFRAME_READY, + }); + }, + { interval: 5 }, + ); + + vi.mocked(chrome.runtime.sendMessage).mockClear(); + dynamicClosed.appendChild(document.createElement("iframe")); + await vi.waitFor( + () => { + expect(chrome.runtime.sendMessage).toHaveBeenCalledExactlyOnceWith({ + kind: COOPERATION_SUBFRAME_READY, + }); + }, + { interval: 5 }, + ); + + expect(read()).toMatchObject({ + human: 0, + structure: 0, + }); +}); + +it("ignores internal data-bsk-* attributes and browser-skill-overlay mutations and UI interactions", async () => { + document.body.innerHTML = + ''; const read = collector(); - expect(chrome.runtime.sendMessage).toHaveBeenCalledWith({ kind: COOPERATION_READY }); - expect(read()).toHaveProperty("document"); + const initial = read()!; + + // Recording / sensitive-field private markers and overlay host mount must not advance `structure` or `revision`. + document.documentElement.setAttribute("data-bsk-record-document", "doc-1"); + const secret = document.querySelector("#secret")!; + secret.setAttribute("data-bsk-record-password", ""); + const overlayHost = document.createElement("browser-skill-overlay"); + overlayHost.setAttribute("data-bsk-overlay", ""); + const shadow = overlayHost.attachShadow({ mode: "open" }); + shadow.innerHTML = + '
'; + document.documentElement.appendChild(overlayHost); + + await Promise.resolve(); + expect(read()).toMatchObject({ + document: initial.document, + revision: initial.revision, + structure: 0, + human: 0, + editing: false, + }); + + // Clicking, focusing, and typing inside the extension's own ShadowRoot UI must not advance `human` or `editing`. + const helpNote = shadow.querySelector("#help-note")!; + const helpSend = shadow.querySelector("#help-send")!; + for (const target of [helpNote, helpSend]) { + const down = new MouseEvent("pointerdown", { + bubbles: true, + composed: true, + clientX: 4, + clientY: 4, + }); + Object.defineProperty(down, "isTrusted", { value: true }); + target.dispatchEvent(down); + } + helpNote.focus(); + helpNote.value = "User typed note inside HelpRequestOverlay"; + const inputEvent = new InputEvent("input", { bubbles: true, composed: true }); + Object.defineProperty(inputEvent, "isTrusted", { value: true }); + helpNote.dispatchEvent(inputEvent); + + const afterOverlay = read()!; + expect(afterOverlay).toMatchObject({ + revision: initial.revision, + structure: 0, + human: 0, + editing: false, + }); + expect(afterOverlay.fields).toEqual([expect.objectContaining({ label: "Search", value: "ok" })]); + overlayHost.remove(); + document.documentElement.removeAttribute("data-bsk-record-document"); }); it("classifies a non-submitted media timeline without masking business fields", () => { diff --git a/apps/extension/src/content/__tests__/record-capture.test.ts b/apps/extension/src/content/__tests__/record-capture.test.ts index 05ac377ae..696f3c1dd 100644 --- a/apps/extension/src/content/__tests__/record-capture.test.ts +++ b/apps/extension/src/content/__tests__/record-capture.test.ts @@ -1,6 +1,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; import type { RecordStepPayload } from "@/lib/record-bridge"; import { startRecordCapture } from "../record-capture"; +import { collectHoverSurfaceStates } from "../record-hover-surface"; vi.stubGlobal("chrome", { runtime: { @@ -1686,4 +1687,65 @@ describe("record-capture semantic", () => { }), ]); }); + + it("ignores interactions inside browser-skill-overlay ShadowRoot and leaves history.pushState untouched", () => { + const originalPushState = window.history.pushState; + const originalReplaceState = window.history.replaceState; + const overlayHost = document.createElement("browser-skill-overlay"); + overlayHost.setAttribute("data-bsk-overlay", ""); + overlayHost.setAttribute("role", "dialog"); + mockRect(overlayHost, { left: 20, top: 20, width: 240, height: 120 }); + const shadow = overlayHost.attachShadow({ mode: "open" }); + shadow.innerHTML = ` +
+ + + + +
+ `; + document.documentElement.appendChild(overlayHost); + + try { + const capture = startRecordCapture("rec-overlay-ignore", (step) => steps.push(step)); + expect(window.history.pushState).toBe(originalPushState); + expect(window.history.replaceState).toBe(originalReplaceState); + expect(collectHoverSurfaceStates()).toEqual([]); + + const overlayInput = shadow.querySelector("#overlay-note")!; + const overlaySelect = shadow.querySelector("#overlay-select")!; + const overlayButton = shadow.querySelector("#overlay-finish")!; + + overlayButton.dispatchEvent(new MouseEvent("mouseover", { bubbles: true, composed: true })); + overlayInput.dispatchEvent(new FocusEvent("focusin", { bubbles: true, composed: true })); + overlayInput.dispatchEvent( + new CompositionEvent("compositionstart", { bubbles: true, composed: true }), + ); + overlayInput.value = "help explanation"; + overlayInput.dispatchEvent(new Event("input", { bubbles: true, composed: true })); + overlayInput.dispatchEvent( + new CompositionEvent("compositionend", { bubbles: true, composed: true }), + ); + overlayInput.dispatchEvent( + new KeyboardEvent("keydown", { key: "Enter", bubbles: true, composed: true }), + ); + overlayInput.dispatchEvent(new FocusEvent("focusout", { bubbles: true, composed: true })); + overlaySelect.value = "b"; + overlaySelect.dispatchEvent(new Event("change", { bubbles: true, composed: true })); + overlayButton.dispatchEvent( + new MouseEvent("click", { bubbles: true, composed: true, button: 0, detail: 1 }), + ); + + expect(steps).toEqual([]); + + // Real webpage controls on the same document continue recording normally. + const pageButton = document.querySelector("button")!; + click(pageButton); + capture.dispose(); + + expect(steps.map((s) => s.op)).toEqual(["click"]); + } finally { + overlayHost.remove(); + } + }); }); diff --git a/apps/extension/src/content/__tests__/record-frame-agent.test.ts b/apps/extension/src/content/__tests__/record-frame-agent.test.ts index 7c55de3dd..39c490766 100644 --- a/apps/extension/src/content/__tests__/record-frame-agent.test.ts +++ b/apps/extension/src/content/__tests__/record-frame-agent.test.ts @@ -2,7 +2,7 @@ import { afterEach, describe, expect, it, vi } from "vitest"; import type { RecordFramePortMessage } from "@/lib/recording/frame-bridge"; import { RECORD_FRAME_START } from "@/lib/recording/frame-bridge"; import { RECORD_DOCUMENT_ATTRIBUTE } from "@/shared/recording-document-identity"; -import { RecordFrameAgent } from "../recording/frame-agent"; +import { attachRecordFrameAgent, RecordFrameAgent } from "../recording/frame-agent"; class PortListeners unknown> { readonly values = new Set(); @@ -121,4 +121,89 @@ describe("RecordFrameAgent", () => { expect(sendMessage).toHaveBeenCalledTimes(3); expect(document.documentElement.hasAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe(false); }); + + it("loads with zero startup wakeup IPC and reports dynamically added subframes (including closed ShadowRoots) while recording", async () => { + const closedRoots = new WeakMap(); + const preHost = document.createElement("div"); + const preClosed = preHost.attachShadow({ mode: "closed" }); + closedRoots.set(preHost, preClosed); + document.body.appendChild(preHost); + + const harness = portHarness(); + const onMessage = new PortListeners< + ( + message: unknown, + sender: chrome.runtime.MessageSender, + sendResponse: (response: unknown) => void, + ) => boolean + >(); + const sendMessage = vi.fn(); + vi.stubGlobal("chrome", { + dom: { + openOrClosedShadowRoot: (el: HTMLElement) => closedRoots.get(el) ?? null, + }, + runtime: { + onMessage, + connect: vi.fn(() => harness.port), + sendMessage, + }, + }); + const dispose = attachRecordFrameAgent(); + try { + expect(sendMessage).not.toHaveBeenCalled(); + expect(onMessage.values.size).toBe(1); + + // Appending an iframe before recording starts emits nothing. + document.body.appendChild(document.createElement("iframe")); + await Promise.resolve(); + expect(harness.outbound).toEqual([]); + + // Arm recording via RECORD_FRAME_START, then dynamically append two iframes. + let startReply: unknown; + for (const listener of onMessage.values) { + listener( + { type: RECORD_FRAME_START, requestId: "rec-subframe", startedAtMs: 20 }, + {} as chrome.runtime.MessageSender, + (reply) => { + startReply = reply; + }, + ); + } + await vi.waitFor(() => expect(startReply).toEqual({ ok: true })); + harness.outbound.length = 0; + + document.body.append(document.createElement("iframe"), document.createElement("iframe")); + await vi.waitFor( + () => { + expect(harness.outbound).toEqual([ + { + type: "subframe_added", + requestId: "rec-subframe", + producerId: expect.any(String), + }, + ]); + }, + { interval: 5 }, + ); + + // Pre-existing closed ShadowRoot and newly inserted closed ShadowRoot hosts also emit `subframe_added`. + harness.outbound.length = 0; + preClosed.appendChild(document.createElement("iframe")); + await vi.waitFor(() => expect(harness.outbound).toHaveLength(1), { interval: 5 }); + + harness.outbound.length = 0; + const dynamicHost = document.createElement("div"); + const dynamicClosed = dynamicHost.attachShadow({ mode: "closed" }); + closedRoots.set(dynamicHost, dynamicClosed); + dynamicClosed.appendChild(document.createElement("iframe")); + document.body.appendChild(dynamicHost); + await vi.waitFor(() => expect(harness.outbound).toHaveLength(1), { interval: 5 }); + + harness.outbound.length = 0; + dynamicClosed.appendChild(document.createElement("iframe")); + await vi.waitFor(() => expect(harness.outbound).toHaveLength(1), { interval: 5 }); + } finally { + dispose(); + } + }); }); diff --git a/apps/extension/src/content/cooperation-capture.ts b/apps/extension/src/content/cooperation-capture.ts index c958eb1a6..cfc267927 100644 --- a/apps/extension/src/content/cooperation-capture.ts +++ b/apps/extension/src/content/cooperation-capture.ts @@ -2,21 +2,32 @@ import { describeFrameOwners } from "@/cooperation/hidden-frames"; import { playbackProgressMedia } from "@/cooperation/media-progress"; import { COOPERATION_ARM, + COOPERATION_ARM_FRAME, COOPERATION_DISABLE, COOPERATION_MESSAGE, COOPERATION_OWNERS, COOPERATION_READ, - COOPERATION_READY, COOPERATION_RESUME, + COOPERATION_SUBFRAME_READY, type FieldChange, type FrameState, type HumanActivity, type InputExpectation, type TextChange, } from "@/cooperation/types"; -import { OVERLAY_HOST_SELECTOR } from "@/lib/overlay-bridge"; import { createRandomUuid } from "@/lib/random-uuid"; -import { sensitiveAutocomplete } from "@/lib/sensitive-autocomplete"; +import { + hasAddedSubframeElement, + isExtensionOverlayEvent, + isExtensionOverlayNode, + isInternalBskAttribute, + isInternalBskMutation, + isPasswordTypeMutation, + isSensitiveFormField, + openOrClosedShadowRoot, + primaryEventTarget, + resolveDeepActiveElement, +} from "./dom-primitives"; const MAX_CHANGES = 64; // CDP's native clock conversion and the DOM clock round independently. @@ -51,6 +62,8 @@ const isText = (node: Node | undefined): node is Text => node?.nodeType === 3; /** Isolated-world collection also covers frames; no page-world control API is exposed. */ export function attachCooperationCapture() { let enabled = false; + let listenersAttached = false; + let subframeNotifyScheduled = false; let lease: string | undefined; const retired = new Set(); const state: FrameState = { @@ -94,12 +107,10 @@ export function attachCooperationCapture() { let eventTimer: ReturnType | undefined; let inputTimer: ReturnType | undefined; let scrollTimer: ReturnType | undefined; - const overlay = (target: EventTarget | null) => - target instanceof Element && !!target.closest(OVERLAY_HOST_SELECTOR); - const formerPasswords = new WeakSet(); + const formerPasswords = new WeakSet(); const markFormerPasswords = (records: MutationRecord[]) => { for (const r of records) { - if (r.oldValue?.toLowerCase() === "password") formerPasswords.add(r.target); + if (isPasswordTypeMutation(r)) formerPasswords.add(r.target); } }; // Passwords, codes, card details and file selections never enter field state. @@ -110,8 +121,7 @@ export function attachCooperationCapture() { } return ( (target instanceof HTMLInputElement && ["file", "hidden"].includes(target.type)) || - formerPasswords.has(target) || - sensitiveAutocomplete(target.getAttribute("autocomplete")) + isSensitiveFormField(target, formerPasswords) ); }; const label = (target: EventTarget | null) => { @@ -159,7 +169,7 @@ export function attachCooperationCapture() { const publish = () => { clearTimeout(timer); timer = undefined; - if (!enabled) return; + if (!enabled || !lease) return; updateInteraction(); state.fields = [...fields.values()]; void chrome.runtime @@ -242,13 +252,9 @@ export function attachCooperationCapture() { /** An attribute write the observation cannot see: unchanged, or page-private data no style reads. */ const invisibleAttribute = (r: MutationRecord) => { const name = r.attributeName?.toLowerCase() ?? ""; + if (isInternalBskAttribute(name)) return true; if (r.target instanceof Element && r.target.getAttribute(name) === r.oldValue) return true; - if ( - !name.startsWith("data-") || - name.startsWith("data-bsk") || - OBSERVED_DATA_ATTRIBUTES.has(name) - ) - return false; + if (!name.startsWith("data-") || OBSERVED_DATA_ATTRIBUTES.has(name)) return false; const facts = styleFacts(); return facts.readable && !facts.text.includes(name); }; @@ -358,6 +364,7 @@ export function attachCooperationCapture() { if (++burst > 10) return false; for (const root of roots) for (const parent of [root, ...root.querySelectorAll("*")]) { + if (isExtensionOverlayNode(parent)) continue; for (const child of parent.childNodes) if (isText(child) && texts.has(cleanText(child.data))) return false; if (parent instanceof Element) @@ -368,12 +375,32 @@ export function attachCooperationCapture() { } return true; }; + const notifySubframesIfAdded = (records: MutationRecord[]) => { + if (!enabled || subframeNotifyScheduled || !hasAddedSubframeElement(records)) return; + subframeNotifyScheduled = true; + queueMicrotask(() => { + subframeNotifyScheduled = false; + if (!enabled) return; + void chrome.runtime.sendMessage({ kind: COOPERATION_SUBFRAME_READY }).catch(() => {}); + }); + }; + // Page JS cannot see closed roots, but the extension can. Observation also + // traverses these roots, so source and focus guards must cover the same DOM. + const shadowRoot = (element: Element): ShadowRoot | null => openOrClosedShadowRoot(element); const mutations = (records: MutationRecord[]) => { markFormerPasswords(records); - const seen = records.filter( - (r) => !overlay(r.target instanceof Element ? r.target : r.target.parentElement), - ); - if (!enabled || !seen.length) return; + if (enabled) { + for (const r of records) { + if (r.type !== "childList" || isExtensionOverlayNode(r.target)) continue; + for (const node of r.addedNodes) { + if (node instanceof Element) observeShadowSubtree(node); + } + } + } + notifySubframesIfAdded(records); + if (!enabled || !lease) return; + const seen = records.filter((r) => !isInternalBskMutation(r)); + if (!seen.length) return; state.revision++; state.structure++; const texts = textChanges(seen); @@ -388,34 +415,37 @@ export function attachCooperationCapture() { schedule(); }; const observer = new MutationObserver(mutations); - // Page JS cannot see closed roots, but the extension can. Observation also - // traverses these roots, so source and focus guards must cover the same DOM. - const shadowRoot = (element: Element): ShadowRoot | null => - element.shadowRoot ?? - (element instanceof HTMLElement ? (chrome.dom?.openOrClosedShadowRoot(element) ?? null) : null); - const activeElement = () => { - let active = document.activeElement; - while (active) { - const nested = shadowRoot(active)?.activeElement; - if (!nested) break; - active = nested; + const observeRoot = (root: Document | ShadowRoot) => { + if (roots.has(root)) return; + observer.observe(root, { + subtree: true, + childList: true, + attributes: true, + attributeOldValue: true, + characterData: true, + characterDataOldValue: true, + }); + roots.add(root); + }; + const observeShadowSubtree = (target: Document | Element | ShadowRoot) => { + if (target.nodeType === Node.DOCUMENT_NODE || target instanceof ShadowRoot) { + observeRoot(target as Document | ShadowRoot); + } else if (target instanceof Element) { + if (isExtensionOverlayNode(target)) return; + const nested = shadowRoot(target); + if (nested) observeShadowSubtree(nested); + } + for (const element of target.querySelectorAll("*")) { + if (isExtensionOverlayNode(element)) continue; + const nested = shadowRoot(element); + if (nested) observeShadowSubtree(nested); } - return active; }; + const activeElement = () => resolveDeepActiveElement(document, shadowRoot); const controls = function* (root: Document | ShadowRoot): Generator { - if (!roots.has(root)) { - observer.observe(root, { - subtree: true, - childList: true, - attributes: true, - attributeOldValue: true, - characterData: true, - characterDataOldValue: true, - }); - roots.add(root); - } + observeRoot(root); for (const element of root.querySelectorAll("*")) { - if (overlay(element)) continue; + if (isExtensionOverlayNode(element)) continue; if (element.matches("input,textarea,select,[contenteditable=true]")) yield element; const nested = shadowRoot(element); if (nested) yield* controls(nested); @@ -451,7 +481,7 @@ export function attachCooperationCapture() { const mediaByRoot = new Map(); state.fieldsComplete = true; for (const target of controls(document)) { - if (overlay(target)) continue; + if (isExtensionOverlayNode(target)) continue; const fullValue = target instanceof HTMLInputElement && ["checkbox", "radio"].includes(target.type) ? String(target.checked) @@ -511,30 +541,32 @@ export function attachCooperationCapture() { } for (const element of fields.keys()) if (!element.isConnected) fields.delete(element); const media = [...roots].flatMap((root) => - [...root.querySelectorAll("video,audio")].map((element) => { - const stream = element.srcObject; - return { - id: id(element), - src: element.getAttribute("src"), - currentSrc: element.currentSrc, - srcObject: stream ? id(stream) : null, - tracks: - typeof MediaStream !== "undefined" && stream instanceof MediaStream - ? stream - .getTracks() - .map((track) => track.id) - .sort() - : [], - sources: [...element.children] - .filter((child): child is HTMLSourceElement => child instanceof HTMLSourceElement) - .map((source) => ({ - id: id(source), - src: source.src, - type: source.type, - media: source.media, - })), - }; - }), + [...root.querySelectorAll("video,audio")] + .filter((element) => !isExtensionOverlayNode(element)) + .map((element) => { + const stream = element.srcObject; + return { + id: id(element), + src: element.getAttribute("src"), + currentSrc: element.currentSrc, + srcObject: stream ? id(stream) : null, + tracks: + typeof MediaStream !== "undefined" && stream instanceof MediaStream + ? stream + .getTracks() + .map((track) => track.id) + .sort() + : [], + sources: [...element.children] + .filter((child): child is HTMLSourceElement => child instanceof HTMLSourceElement) + .map((source) => ({ + id: id(source), + src: source.src, + type: source.type, + media: source.media, + })), + }; + }), ); // currentSrc/srcObject may change without a DOM mutation. Keep the source // evidence fresh even when the page's mutation counter did not advance. @@ -612,8 +644,8 @@ export function attachCooperationCapture() { (dragging && ["dragend", "drop"].includes(event.type)) || (pointer && !pointer.released && ["pointerup", "pointercancel"].includes(event.type)) || (event.type === "keyup" && keys.has((event as KeyboardEvent).key)); - if (!release && event.composedPath().some(overlay)) return; - let target = event.composedPath()[0] ?? event.target; + if (!release && isExtensionOverlayEvent(event)) return; + let target = primaryEventTarget(event); // Closed shadow roots retarget editing events to their host. Resolve the // same deep focus used by collection and native text attribution. if ( @@ -841,17 +873,10 @@ export function attachCooperationCapture() { "input", "change", ]; - for (const event of events) - document.addEventListener(event, onEvent, { capture: true, passive: true }); // Chromium skips a node's legacy `mousewheel` listeners whenever that node has a - // `wheel` listener, even a passive one. Listen only while collecting so pages - // without a session keep their mousewheel handlers. Window capture runs before - // any document listener, so a page that stops propagation there cannot hide an - // Agent wheel from attribution even though this listener is registered late. - const listenWheel = (on: boolean) => - on - ? window.addEventListener("wheel", onEvent, { capture: true, passive: true }) - : window.removeEventListener("wheel", onEvent, true); + // `wheel` listener, even a passive one. Tier B stays completely inert (zero DOM, + // scroll, blur or wheel listeners and zero startup IPC) until Background arms or + // reads a controlled frame. const onScroll = () => { if (!enabled) return; const started = !state.scrolling; @@ -863,7 +888,6 @@ export function attachCooperationCapture() { }, 120); if (started) publish(); }; - document.addEventListener("scroll", onScroll, true); const onWindowBlur = () => { if ( !enabled || @@ -892,7 +916,25 @@ export function attachCooperationCapture() { state.revision++; publish(); }; - window.addEventListener("blur", onWindowBlur); + const setListenersAttached = (active: boolean) => { + if (listenersAttached === active) return; + listenersAttached = active; + if (active) { + for (const event of events) { + document.addEventListener(event, onEvent, { capture: true, passive: true }); + } + document.addEventListener("scroll", onScroll, true); + window.addEventListener("blur", onWindowBlur); + window.addEventListener("wheel", onEvent, { capture: true, passive: true }); + } else { + for (const event of events) { + document.removeEventListener(event, onEvent, true); + } + document.removeEventListener("scroll", onScroll, true); + window.removeEventListener("blur", onWindowBlur); + window.removeEventListener("wheel", onEvent, true); + } + }; const onMessage = ( message: { kind?: string; input?: InputExpectation; lease?: string }, _sender: chrome.runtime.MessageSender, @@ -904,6 +946,13 @@ export function attachCooperationCapture() { reply(describeFrameOwners(document, shadowRoot)); return false; } + if (message.kind === COOPERATION_ARM_FRAME) { + enabled = true; + setListenersAttached(true); + observeShadowSubtree(document); + reply({ ok: true }); + return false; + } if (typeof message.lease !== "string") return false; if (message.kind === COOPERATION_READ) { if (retired.has(message.lease)) { @@ -928,7 +977,7 @@ export function attachCooperationCapture() { } if (message.kind === COOPERATION_DISABLE) { enabled = false; - listenWheel(false); + setListenersAttached(false); clearTimeout(focusTimer); clearTimeout(pointerTimer); clearTimeout(eventTimer); @@ -954,7 +1003,7 @@ export function attachCooperationCapture() { reply({ ...state, busy: false, fields: [] }); } else if (message.kind === COOPERATION_READ) { enabled = true; - listenWheel(true); + setListenersAttached(true); reply(read()); } else if (message.kind === COOPERATION_ARM) { if (enabled && message.input?.type === "clear") { @@ -1011,8 +1060,8 @@ export function attachCooperationCapture() { return false; }; chrome.runtime.onMessage.addListener(onMessage); - void chrome.runtime.sendMessage({ kind: COOPERATION_READY }).catch(() => {}); return () => { + enabled = false; clearTimeout(timer); clearTimeout(focusTimer); clearTimeout(pointerTimer); @@ -1020,10 +1069,8 @@ export function attachCooperationCapture() { clearTimeout(inputTimer); clearTimeout(scrollTimer); observer.disconnect(); - chrome.runtime.onMessage.removeListener(onMessage); - window.removeEventListener("blur", onWindowBlur); - document.removeEventListener("scroll", onScroll, true); - for (const event of events) document.removeEventListener(event, onEvent, true); - listenWheel(false); + roots.clear(); + chrome.runtime?.onMessage?.removeListener(onMessage); + setListenersAttached(false); }; } diff --git a/apps/extension/src/content/dom-primitives.ts b/apps/extension/src/content/dom-primitives.ts new file mode 100644 index 000000000..c100842c8 --- /dev/null +++ b/apps/extension/src/content/dom-primitives.ts @@ -0,0 +1,205 @@ +import { + isOverlayHostElementName, + OVERLAY_HOST_MARKER_ATTR, + OVERLAY_HOST_SELECTOR, +} from "@/lib/overlay-bridge"; +import { RECORD_PASSWORD_ATTRIBUTE, sensitiveAutocomplete } from "@/lib/sensitive-autocomplete"; + +export const INTERNAL_BSK_ATTR_PREFIX = "data-bsk-"; + +export function primaryEventTarget(event: Event): EventTarget | null { + return event.composedPath()[0] ?? event.target; +} + +export function isExtensionOverlayElement(element: Element): boolean { + return ( + isOverlayHostElementName(element.localName) || element.hasAttribute(OVERLAY_HOST_MARKER_ATTR) + ); +} + +/** + * True when `target` is the extension's `` host, any + * `[data-bsk-overlay]` subtree, or any node inside its open/closed ShadowRoot. + */ +export function isExtensionOverlayNode(target: EventTarget | null): boolean { + if (!(target instanceof Node)) return false; + let current: Node | null = target; + while (current) { + if (current instanceof ShadowRoot) { + current = current.host; + continue; + } + if (current instanceof Element) { + if (isExtensionOverlayElement(current) || current.closest(OVERLAY_HOST_SELECTOR)) { + return true; + } + } + const root: Node = current.getRootNode(); + if (root instanceof ShadowRoot) { + current = root.host; + } else { + current = current instanceof Element ? null : current.parentElement; + } + } + return false; +} + +export function isExtensionOverlayEvent(event: Event): boolean { + for (const entry of event.composedPath()) { + if (isExtensionOverlayNode(entry)) return true; + } + return isExtensionOverlayNode(event.target); +} + +export function openOrClosedShadowRoot(element: Element): ShadowRoot | null { + if (isExtensionOverlayNode(element)) return null; + return ( + element.shadowRoot ?? + (element instanceof HTMLElement + ? (globalThis.chrome?.dom?.openOrClosedShadowRoot?.(element) ?? null) + : null) + ); +} + +export function resolveDeepActiveElement( + doc: Document = document, + getShadowRoot: (element: Element) => ShadowRoot | null = openOrClosedShadowRoot, +): Element | null { + let active = doc.activeElement; + if (active && isExtensionOverlayNode(active)) { + return doc.body ?? doc.documentElement; + } + while (active) { + const nested = getShadowRoot(active)?.activeElement; + if (!nested || isExtensionOverlayNode(nested)) break; + active = nested; + } + return active; +} + +export function isInternalBskAttribute(attributeName: string | null | undefined): boolean { + return ( + typeof attributeName === "string" && + attributeName.toLowerCase().startsWith(INTERNAL_BSK_ATTR_PREFIX) + ); +} + +/** + * True when a DOM mutation was produced solely by BrowserSkill's own overlay + * host or internal `data-bsk-*` bookkeeping attributes. + */ +export function isInternalBskMutation(record: MutationRecord): boolean { + if (isExtensionOverlayNode(record.target)) return true; + if (record.type === "attributes" && isInternalBskAttribute(record.attributeName)) { + return true; + } + if (record.type === "childList") { + const total = record.addedNodes.length + record.removedNodes.length; + if (total > 0) { + let allOverlay = true; + for (const node of record.addedNodes) { + if (!isExtensionOverlayNode(node)) { + allOverlay = false; + break; + } + } + if (allOverlay) { + for (const node of record.removedNodes) { + if (!isExtensionOverlayNode(node)) { + allOverlay = false; + break; + } + } + } + if (allOverlay) return true; + } + } + return false; +} + +export function isPasswordTypeMutation( + record: MutationRecord, +): record is MutationRecord & { target: Element } { + return record.oldValue?.toLowerCase() === "password" && record.target instanceof Element; +} + +export function isSensitiveFormField( + target: Element, + formerPasswords?: { has(element: Element): boolean }, +): boolean { + return ( + (target instanceof HTMLInputElement && target.type.toLowerCase() === "password") || + target.hasAttribute(RECORD_PASSWORD_ATTRIBUTE) || + Boolean(formerPasswords?.has(target)) || + sensitiveAutocomplete(target.getAttribute("autocomplete")) + ); +} + +const SUBFRAME_OWNER_SELECTOR = "iframe,frame,object,embed,fencedframe"; +const SUBFRAME_OWNER_TAGS = new Set(["iframe", "frame", "object", "embed", "fencedframe"]); + +function subtreeHasSubframeElement(root: Element | ShadowRoot): boolean { + if (root instanceof Element) { + if (isExtensionOverlayNode(root)) return false; + if (SUBFRAME_OWNER_TAGS.has(root.localName)) return true; + const nestedShadow = openOrClosedShadowRoot(root); + if (nestedShadow && subtreeHasSubframeElement(nestedShadow)) return true; + } + for (const match of root.querySelectorAll(SUBFRAME_OWNER_SELECTOR)) { + if (!isExtensionOverlayNode(match)) return true; + } + for (const element of root.querySelectorAll("*")) { + if (isExtensionOverlayNode(element)) continue; + const nestedShadow = openOrClosedShadowRoot(element); + if (nestedShadow && subtreeHasSubframeElement(nestedShadow)) return true; + } + return false; +} + +export function hasAddedSubframeElement(records: MutationRecord[]): boolean { + for (const record of records) { + if (record.type !== "childList" || isExtensionOverlayNode(record.target)) continue; + for (const node of record.addedNodes) { + if (!(node instanceof Element)) continue; + if (subtreeHasSubframeElement(node)) return true; + } + } + return false; +} + +/** + * Ensures repeated `chrome.scripting.executeScript` calls of the same WXT + * runtime content script on the same `Document` are strictly idempotent: + * WXT's `new ContentScriptContext(name)` dispatches + * `${runtimeId}:${name}:wxt:content-script-started` on `document` before + * entering `main(ctx)`. Intercepting that event in the capture phase while the + * first context is still valid preserves the live ShadowRoot, ports, and + * listeners, while still allowing cleanup after an extension reload + * (`ctx.isInvalid === true`). + */ +export function claimIdempotentContentScript( + ctx: { readonly isInvalid?: boolean; abort?: (reason?: unknown) => void }, + contentScriptName: string, + mountedSymbol: symbol, +): (() => void) | null { + const win = window as unknown as Record; + const runtimeId = chrome.runtime?.id ?? "mounted"; + if (win[mountedSymbol] === runtimeId) { + ctx.abort?.("Duplicate runtime injection skipped"); + return null; + } + win[mountedSymbol] = runtimeId; + const startedEventName = `${runtimeId}:${contentScriptName}:wxt:content-script-started`; + const preserveWhileValid = (event: Event) => { + if (!ctx.isInvalid) { + event.stopImmediatePropagation(); + } + }; + document.addEventListener(startedEventName, preserveWhileValid, true); + return () => { + document.removeEventListener(startedEventName, preserveWhileValid, true); + if (win[mountedSymbol] === runtimeId) { + delete win[mountedSymbol]; + } + }; +} diff --git a/apps/extension/src/content/record-capture.ts b/apps/extension/src/content/record-capture.ts index 32f41a1f9..1947ff8ef 100644 --- a/apps/extension/src/content/record-capture.ts +++ b/apps/extension/src/content/record-capture.ts @@ -15,7 +15,14 @@ import { } from "@/lib/hover-trigger-policy"; import type { RecordStepPayload } from "@/lib/record-bridge"; import { shouldRecordPress } from "@/lib/recording/draft-policy"; -import { RECORD_PASSWORD_ATTRIBUTE, sensitiveAutocomplete } from "@/lib/sensitive-autocomplete"; +import { RECORD_PASSWORD_ATTRIBUTE } from "@/lib/sensitive-autocomplete"; +import { + isExtensionOverlayEvent, + isInternalBskMutation, + isPasswordTypeMutation, + isSensitiveFormField, + primaryEventTarget, +} from "./dom-primitives"; import { closestHoverSurfaceCandidate, collectHoverSurfaceStates, @@ -60,32 +67,6 @@ const HOVER_REPLACE_SCORE_MARGIN = 50; const HOVER_CANDIDATE_LIMIT = 24; const HOVER_TRIGGER_LABEL_MAX = 48; -function eventTarget(event: Event): EventTarget | null { - return event.composedPath()[0] ?? event.target; -} - -function isOverlayTarget(target: EventTarget | null): boolean { - if (!(target instanceof Node)) return false; - const root = document.documentElement; - let node: Node | null = target; - while (node && node !== root) { - if (node instanceof Element && node.hasAttribute("data-bsk-overlay")) { - return true; - } - const rootNode: Node | Document | ShadowRoot = node.getRootNode(); - if (rootNode instanceof ShadowRoot) { - const host: Element = rootNode.host; - if (host.hasAttribute("data-bsk-overlay")) { - return true; - } - node = host; - } else { - node = node.parentNode; - } - } - return false; -} - function isTextFillable(el: Element): el is FillableElement { if (el instanceof HTMLElement && (el.isContentEditable || el.contentEditable === "true")) { return true; @@ -401,14 +382,12 @@ export function startRecordCapture( let hoverSurfaceStates = hoverSurfaceStateMap(collectHoverSurfaceStates()); const hoverSurfaceNodes = new Map(); let generatedControlClick: Element | null = null; - let navigationActionPending = false; - let navigationActionVersion = 0; const committedValues = new WeakMap(); // A "show password" toggle switches type=password to text; keep masking those fields. const formerPasswords = new Set(); const markFormerPasswords = (records: MutationRecord[]) => { for (const record of records) { - if (record.oldValue?.toLowerCase() === "password" && record.target instanceof Element) { + if (isPasswordTypeMutation(record)) { formerPasswords.add(record.target); record.target.setAttribute(RECORD_PASSWORD_ATTRIBUTE, ""); } @@ -421,23 +400,10 @@ export function startRecordCapture( attributeOldValue: true, }); - const markNavigationAction = () => { - navigationActionPending = true; - const version = ++navigationActionVersion; - setTimeout(() => { - if (navigationActionVersion === version) { - navigationActionPending = false; - } - }, 0); - }; - const emitFill = (session: FillSession) => { if (session.lastValue === session.baselineValue) return; markFormerPasswords(typeObserver.takeRecords()); - const isPassword = - (session.element instanceof HTMLInputElement && session.element.type === "password") || - formerPasswords.has(session.element) || - sensitiveAutocomplete(session.element.getAttribute("autocomplete")); + const isPassword = isSensitiveFormField(session.element, formerPasswords); const value = isPassword ? "***" : session.lastValue; emitStep({ op: "fill", @@ -476,14 +442,13 @@ export function startRecordCapture( }; }; - const emitNavigateIfChanged = (causedByAction?: boolean) => { + const emitNavigateIfChanged = () => { if (location.href === lastUrl) return; commitFillSession(); lastUrl = location.href; emitStep({ op: "navigate", url: location.href, - ...(causedByAction !== undefined ? { navigation_caused_by_action: causedByAction } : {}), }); }; @@ -707,13 +672,13 @@ export function startRecordCapture( const emitClick = (event: MouseEvent) => { // Only record clicks an LLM can re-identify (named interactive controls). - const target = describeEventTarget(eventTarget(event)); + const rawTarget = primaryEventTarget(event); + const target = describeEventTarget(rawTarget); if (!target) return; - markNavigationAction(); emitStep({ op: "click", target, - geometry: geometryForEventTarget(eventTarget(event)), + geometry: geometryForEventTarget(rawTarget), expects_navigation: true, }); }; @@ -738,8 +703,8 @@ export function startRecordCapture( }; const onMouseOver = (event: MouseEvent) => { - const target = eventTarget(event); - if (isOverlayTarget(target)) return; + if (isExtensionOverlayEvent(event)) return; + const target = primaryEventTarget(event); const now = Date.now(); if (pendingHover && target instanceof Element && pendingHover.element.contains(target)) { if (now - pendingHover.recordedAt <= HOVER_BEFORE_CLICK_MAX_MS) return; @@ -767,9 +732,8 @@ export function startRecordCapture( }; const onClick = (event: MouseEvent) => { - if (event.button !== 0) return; - const target = eventTarget(event); - if (isOverlayTarget(target)) return; + if (event.button !== 0 || isExtensionOverlayEvent(event)) return; + const target = primaryEventTarget(event); if (event.detail === 0 && generatedControlClick !== null && target === generatedControlClick) { generatedControlClick = null; return; @@ -810,7 +774,6 @@ export function startRecordCapture( } if (isInputCompletionClick(target, fillSession)) { - markNavigationAction(); const sessionElement = fillSession!.element; scheduleInputCompletionCommit( sessionElement, @@ -828,12 +791,14 @@ export function startRecordCapture( }; const onFocusIn = (event: FocusEvent) => { - const target = fillableFromTarget(eventTarget(event)); + if (isExtensionOverlayEvent(event)) return; + const target = fillableFromTarget(primaryEventTarget(event)); if (target) ensureFillSession(target); }; const onFocusOut = (event: FocusEvent) => { - const target = fillableFromTarget(eventTarget(event)); + if (isExtensionOverlayEvent(event)) return; + const target = fillableFromTarget(primaryEventTarget(event)); if (!target) return; if (!fillSession || fillSession.element !== target) return; syncFillSessionValue(target); @@ -841,7 +806,8 @@ export function startRecordCapture( }; const onInput = (event: Event) => { - const rawTarget = eventTarget(event); + if (isExtensionOverlayEvent(event)) return; + const rawTarget = primaryEventTarget(event); // A native ', + ); + return; + } + if (req.url === "/prerender-start") { + res.end( + 'Prerender HostActivate', + ); + return; + } + if (req.url === "/prerendered") { + res.end( + 'Prerendered Target', + ); + return; + } + if (req.url === "/record-page") { + res.end(`Record Target + + `); + return; + } + res.end(`User Page ${req.url} + + `); + }); + pageServer.listen(0, "127.0.0.1"); + await once(pageServer as unknown as EventEmitter, "listening"); + const port = (pageServer.address() as { port: number }).port; + const baseUrl = `http://127.0.0.1:${port}`; + + const cli = async (...args: string[]) => { + const result = await runExecFile(executable, ["--json", ...args], { + env, + timeout: 45_000, + }).catch((error) => { + error.message += `\n${error.stdout}\n${error.stderr}`; + throw error; + }); + return JSON.parse(result.stdout); + }; + + try { + let wsPort = 0; + await expect + .poll( + async () => { + try { + wsPort = JSON.parse( + await readFile(path.join(env.BSK_HOME!, "daemon.json"), "utf8"), + ).ws_port; + return wsPort; + } catch { + return 0; + } + }, + { timeout: 15_000 }, + ) + .toBeGreaterThan(0); + + const { copyPairingExtension, pairLocalBrowser } = await import( + new URL("../../../../../evals/browser/local-pairing.mjs", import.meta.url).href + ); + const extension = await copyPairingExtension( + distDir, + path.join(directory, "extension"), + wsPort, + ); + const manifestPath = path.join(extension, "manifest.json"); + const manifest = JSON.parse(await readFile(manifestPath, "utf8")); + manifest.name = "BrowserSkill Wave5A integration"; + expect(manifest.content_scripts).toBeUndefined(); + await writeFile(manifestPath, JSON.stringify(manifest)); + + const { withChrome } = await import( + new URL( + "../../../../../evals/browser/cases/regression/snapshot-coordinates/chrome.mjs", + import.meta.url, + ).href + ); + + const chromePath = process.env.BSK_TEST_CHROME ?? ""; + const useCdpExtensionLoad = + process.env.BSK_TEST_LOAD_CDP === "1" || + (chromePath.includes("Google Chrome") && !chromePath.includes("Chrome for Testing")); + + const attachedTabPages: { parent: string; session: string; prerender: boolean }[] = []; + await withChrome( + { + executable: process.env.BSK_TEST_CHROME, + extensionPath: extension, + extensionViaCdp: useCdpExtensionLoad, + deviceScale: 1, + zoom: 1, + extraArgs: ["--force-effective-connection-type=4G"], + onEvent: (event: { + method: string; + sessionId?: string; + params?: { sessionId: string; targetInfo: { type: string; subtype?: string } }; + }) => { + if ( + event.method === "Target.attachedToTarget" && + event.sessionId && + event.params?.targetInfo.type === "page" + ) { + attachedTabPages.push({ + parent: event.sessionId, + session: event.params.sessionId, + prerender: event.params.targetInfo.subtype === "prerender", + }); + } + }, + }, + async (send: Send) => { + let worker = await attachBackgroundByName(send, manifest.name); + const swEval = async (expression: string): Promise => { + const reply = await send<{ result: { value: T }; exceptionDetails?: unknown }>( + "Runtime.evaluate", + { expression, returnByValue: true, awaitPromise: true }, + worker.sessionId, + ); + if (reply.exceptionDetails) { + throw new Error(JSON.stringify(reply.exceptionDetails)); + } + return reply.result.value; + }; + + const tabEval = async (tabId: number, funcBody: string): Promise => { + return swEval(`(async () => { + const [res] = await chrome.scripting.executeScript({ + target: { tabId: ${tabId} }, + func: () => { ${funcBody} }, + }); + return res?.result; + })()`); + }; + + await pairLocalBrowser(send, { executable, env, worker }); + await expect + .poll(async () => (await cli("browsers")).length, { timeout: 15_000 }) + .toBe(1); + + // ========================================================================= + // Counterexample A: Zero-session zero-tax + bystander isolation + borrow + // ========================================================================= + await expect + .poll(() => swEval("chrome.scripting.getRegisteredContentScripts()")) + .toEqual([]); + + const userWindowId = await swEval( + "(async () => (await chrome.windows.getAll({ windowTypes: ['normal'] }))[0].id)()", + ); + const bystanderTabId = await swEval(`(async () => { + const tab = await chrome.tabs.create({ windowId: ${userWindowId}, url: ${JSON.stringify(`${baseUrl}/bystander`)}, active: false }); + return tab.id; + })()`); + const userTabId = await swEval(`(async () => { + const tab = await chrome.tabs.create({ windowId: ${userWindowId}, url: ${JSON.stringify(`${baseUrl}/borrow-candidate`)}, active: true }); + await chrome.windows.update(${userWindowId}, { focused: true }); + return tab.id; + })()`); + + await expect + .poll(() => + swEval( + `(async () => (await chrome.tabs.get(${userTabId})).status === 'complete' && (await chrome.tabs.get(${bystanderTabId})).status === 'complete')()`, + ), + ) + .toBe(true); + + // Both ordinary tabs have 0 Overlay DOM and 0 cooperation receivers when idle. + for (const id of [userTabId, bystanderTabId]) { + expect( + await tabEval( + id, + "return document.querySelector('browser-skill-overlay') !== null;", + ), + ).toBe(false); + expect( + await swEval( + `chrome.tabs.sendMessage(${id}, { kind: 'cooperation.read', lease: 'probe' }).then(() => 'ok', (e) => String(e?.message ?? e))`, + ), + ).toContain("Receiving end does not exist"); + } + + // Require explicit in-page borrow confirmation before borrowing `userTabId`. + await swEval( + "chrome.storage.local.set({ bsk_interaction_preferences: { confirmTabBorrow: true, requestHelpEnabled: true } })", + ); + + const sessionStart = (await cli("session", "start")) as { session_id: string }; + const sessionId = sessionStart.session_id; + + // Tier B cooperation is dynamically registered with persistAcrossSessions: false. + await expect + .poll(() => + swEval( + "chrome.scripting.getRegisteredContentScripts()", + ), + ) + .toEqual([ + expect.objectContaining({ + id: "bsk-tier-b-cooperation", + js: ["content-scripts/cooperation.js"], + allFrames: true, + matchOriginAsFallback: true, + runAt: "document_start", + persistAcrossSessions: false, + }), + ]); + + // Even after session start, opening another normal user tab loads 0 bytes of Tier A React UI. + const postSessionBystanderTabId = await swEval(`(async () => { + const tab = await chrome.tabs.create({ windowId: ${userWindowId}, url: ${JSON.stringify(`${baseUrl}/bystander-2`)}, active: false }); + return tab.id; + })()`); + await expect + .poll(() => + swEval( + `(async () => (await chrome.tabs.get(${postSessionBystanderTabId})).status === 'complete')()`, + ), + ) + .toBe(true); + expect( + await tabEval<{ hasOverlay: boolean; messages: unknown[] }>( + postSessionBystanderTabId, + "return { hasOverlay: document.querySelector('browser-skill-overlay') !== null, messages: window.__bskMessages ?? [] };", + ), + ).toEqual({ hasOverlay: false, messages: [] }); + + // Re-focus userWindowId + userTabId and trigger `tab borrow`, which must inject Tier A on demand + // into the candidate user tab BEFORE it becomes controlled so the Allow modal renders. + await swEval( + `chrome.tabs.update(${userTabId}, { active: true }).then(() => chrome.windows.update(${userWindowId}, { focused: true }))`, + ); + const borrowPromise = cli("tab", "borrow", "--session", sessionId, String(userTabId)); + + await expect + .poll( + () => + tabEval( + userTabId, + `const host = document.querySelector('browser-skill-overlay'); + const btn = host?.shadowRoot?.querySelector('[data-slot="borrow-confirmation-allow-button"]'); + if (!btn) return false; + btn.click(); + return true;`, + ), + { timeout: 10_000 }, + ) + .toBe(true); + + const borrowed = await borrowPromise; + expect(borrowed.tab_id).toBe(userTabId); + + // Borrowed tab now has both Tier A and Tier B (main + static child iframe), while bystanders remain at 0 Tier A! + const synced = await cli("cooperate", "sync", "--session", sessionId); + expect(synced.observation_id).toEqual(expect.any(String)); + for (const bystander of [bystanderTabId, postSessionBystanderTabId]) { + expect( + await tabEval( + bystander, + "return document.querySelector('browser-skill-overlay') !== null;", + ), + ).toBe(false); + } + + // ========================================================================= + // Counterexample B: Repeated injection idempotency, dynamic about:blank/srcdoc + // 50ms coalesced arming, and Speculation Rules prerender + // ========================================================================= + await tabEval( + userTabId, + "document.querySelector('browser-skill-overlay').dataset.bskIdentity = 'singleton-host';", + ); + for (let i = 0; i < 3; i++) { + await swEval( + `chrome.scripting.executeScript({ target: { tabId: ${userTabId} }, files: ['content-scripts/content.js'] })`, + ); + } + expect( + await tabEval<{ count: number; identity: string | undefined }>( + userTabId, + `const hosts = document.querySelectorAll('browser-skill-overlay'); + return { count: hosts.length, identity: hosts[0]?.dataset.bskIdentity };`, + ), + ).toEqual({ count: 1, identity: "singleton-host" }); + + // Dynamically insert BOTH a same-origin empty `about:blank` iframe AND an `