diff --git a/CHANGELOG.md b/CHANGELOG.md index 15f6f261..01773c9e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,9 +24,10 @@ Plugin cards nobody can see are paused with their state kept, hidden browser tab ### Changes +- **Browser engines for agents' background tabs.** A plugin can contribute a lighter browser engine (`browser:engine`, see [plugins](docs/plugins.md#browser-engines-browserengine)); `browser_new_tab` takes `engine` (`auto` by default, `chromium`, or the engine's id). With an engine installed and running, an agent's new tab opens in it in the background and the person's active tab stays; the person's own tabs always use Chromium. Engines without real layout get clicks by DOM. The tab moves to Chromium under the same tab id, with a `notice` for the agent, for screenshots, drags, downloads, bot walls, text too thin for the page, a missing CDP method, a crashed engine, or when it is shown, and the site is remembered for the session. Commands without a tab id go to the tab the agent opened last. The engine gets no cookies or profile. Example engine: the separate `canvastty-plugin-lightpanda`. Screenshots of background tabs work while the Browser card is off-screen. - **Launch modes.** Auto is now the default. Manual, Accept edits, Plan and Bypass (YOLO) are offered only where the CLI has them: Accept edits and Plan for Claude Code, Codex, Grok and OpenCode, Plan also for Cursor; Auto for a CLI without an auto mode of its own is its approval bypass and exists only inside agent isolation. Bypass is acknowledged once per CLI, checked in the main process and never given to a subagent. - **Delegation rules.** A subagent never gets more than its orchestrator (plan < manual < accept edits < auto, never Bypass), works only inside its orchestrator's project folder, and stays within the depth (2) and live-subagent (8) limits the person sets in Settings → Agents. A decision plugin's "ask" is a deny with the reason for CLIs that cannot ask. -- **Agent isolation.** An OS layer (macOS `sandbox-exec`, Linux bubblewrap) around subagents, plugin-started agents and every agent not in Manual: writes only in the project, the launch's temporary folder and the CLI's own folders; keys, other CLIs' credentials and CanvasTTY's tokens unreadable; fails closed. Windows has no layer yet, so subagents run in Manual there. Turning it off is the person's opt-in. +- **Agent isolation.** An OS layer (macOS `sandbox-exec`, Linux bubblewrap) around subagents, plugin-started agents and every agent not in Manual: writes only in the project, the launch's temporary folder and the CLI's own folders; keys, other CLIs' credentials and CanvasTTY's tokens unreadable; fails closed. Windows has no layer yet, so subagents run in Manual there; so they do on Linux where bubblewrap cannot create a user namespace (Ubuntu 24.04's AppArmor restriction), with the reason on the card and [how to allow it](docs/installing-and-security.md#linux-when-bubblewrap-cannot-start). Turning it off is the person's opt-in. - **Git audit.** After an isolated session ends, CanvasTTY checks the repositories it touched for git settings and files that would run programs outside isolation and offers **Neutralize** or **Keep as is**. - **Native agent helper.** `canvastty-helper` (Go) runs the MCP servers, the permission gate and the lifecycle hook on macOS and Linux; Windows keeps the JavaScript helpers by default and `CANVASTTY_HELPERS=node` forces them. Building from source needs Go 1.21 or newer for it (`npm run build:helpers`); without Go the JavaScript helpers are used. - **Performance.** Only what the main process and preload load is packaged, built-in skins ship as AVIF; the canvas camera lives outside React; off-screen DOM terminals are not rebuilt on scroll; summary and HOME-hidden terminal screens stop painting and defer selection redraws, while their parsers still receive complete output in bounded pieces so terminal state and history stay exact; the window loads while services start and Settings loads on demand; hidden native browser tabs, plugin frames nobody can see and closed Settings stop polling. diff --git a/CHANGELOG.ru.md b/CHANGELOG.ru.md index 32e33952..6512f4c8 100644 --- a/CHANGELOG.ru.md +++ b/CHANGELOG.ru.md @@ -24,9 +24,10 @@ A/B против 1.7.0 в первоначальном замере всей с ### Изменения +- **Браузерные движки для фоновых вкладок агентов.** Плагин может добавить более лёгкий браузерный движок (`browser:engine`, см. [плагины](docs/plugins.ru.md#браузерные-движки-browserengine)); `browser_new_tab` принимает `engine` (`auto` по умолчанию, `chromium` или id движка). Если движок установлен и запущен, новая вкладка агента открывается в нём в фоне, а активная вкладка человека остаётся; вкладки самого человека всегда в Chromium. Движки без настоящей раскладки получают клики через DOM. Вкладка переходит в Chromium с тем же id и `notice` для агента при скриншоте, перетаскивании, загрузке, защите от ботов, слишком малом тексте для страницы, отсутствующем методе CDP, падении движка или когда её показывают; сайт запоминается до конца сессии. Команды без id вкладки идут во вкладку, которую агент открыл последней. Движок не получает cookies и профиль. Пример движка — отдельный `canvastty-plugin-lightpanda`. Скриншоты фоновых вкладок работают, даже когда карточка браузера за пределами экрана. - **Режимы запуска.** «Авто» теперь по умолчанию. «Вручную», «Правки», «План» и «Обход» (YOLO) предлагаются только там, где их поддерживает CLI: «Правки» и «План» — Claude Code, Codex, Grok и OpenCode, «План» ещё и Cursor; «Авто» для CLI без собственного авторежима — это обход подтверждений, и он есть только внутри изоляции агентов. «Обход» человек подтверждает один раз для каждого CLI, он проверяется в main-процессе и никогда не передаётся субагенту. - **Правила делегирования.** Субагент получает не больше оркестратора (план < вручную < правки < авто, никогда «Обход»), работает только в папке проекта своего оркестратора и в пределах глубины (2) и числа живых субагентов (8), которые человек задаёт в Настройки → Агенты. Ответ «спросить» от плагина решений для CLI, которые не умеют спрашивать, становится запретом с причиной. -- **Изоляция агентов.** Слой ОС (macOS — `sandbox-exec`, Linux — bubblewrap) вокруг субагентов, агентов, запущенных плагинами, и любого агента не во «Вручную»: запись только в проект, временную папку запуска и папки своего CLI; ключи, учётные данные других CLI и токены CanvasTTY недоступны; при сбое запуск отклоняется. В Windows слоя пока нет, поэтому субагенты там работают во «Вручную». Выключить изоляцию — явный выбор человека. +- **Изоляция агентов.** Слой ОС (macOS — `sandbox-exec`, Linux — bubblewrap) вокруг субагентов, агентов, запущенных плагинами, и любого агента не во «Вручную»: запись только в проект, временную папку запуска и папки своего CLI; ключи, учётные данные других CLI и токены CanvasTTY недоступны; при сбое запуск отклоняется. В Windows слоя пока нет, поэтому субагенты там работают во «Вручную»; так же и в Linux, где bubblewrap не может создать пространство имён пользователя (ограничение AppArmor в Ubuntu 24.04): причина видна на карточке, [как разрешить](docs/installing-and-security.ru.md#linux-если-bubblewrap-не-запускается). Выключить изоляцию — явный выбор человека. - **Аудит git.** После окончания изолированной сессии CanvasTTY проверяет затронутые репозитории на настройки и файлы git, которые запустят программы вне изоляции, и предлагает **Обезвредить** или **Оставить как есть**. - **Нативный хелпер агентов.** `canvastty-helper` (Go) обслуживает MCP-серверы, проверку разрешений и хук жизненного цикла на macOS и Linux; Windows по умолчанию остаётся на JavaScript-хелперах, `CANVASTTY_HELPERS=node` включает их везде. Для сборки из исходников нужен Go 1.21 или новее (`npm run build:helpers`); без Go используются JavaScript-хелперы. - **Производительность.** В пакет попадает только то, что загружают main-процесс и preload, встроенные скины — в AVIF; камера холста живёт вне React; DOM-терминалы вне экрана не перестраиваются при прокрутке; терминалы в режиме миниатюр и при редактировании HOME прекращают отрисовку и откладывают перерисовку выделения, а их парсеры получают полный вывод ограниченными порциями, сохраняя состояние и историю; окно загружается, пока стартуют сервисы, а Настройки грузятся по требованию; скрытые вкладки браузера, невидимые фреймы плагинов и закрытые Настройки перестают опрашивать. diff --git a/CHANGELOG.zh-CN.md b/CHANGELOG.zh-CN.md index 89326aef..4f95e900 100644 --- a/CHANGELOG.zh-CN.md +++ b/CHANGELOG.zh-CN.md @@ -24,9 +24,10 @@ ### 变更 +- **用于 agent 后台标签页的浏览器引擎。** 插件可以提供更轻量的浏览器引擎（`browser:engine`，见[插件](docs/plugins.zh-CN.md)）；`browser_new_tab` 接受 `engine`（默认 `auto`、`chromium` 或引擎 id）。安装并运行引擎后，agent 的新标签页在后台用它打开，用户的活动标签页保持不变；用户自己的标签页始终使用 Chromium。没有真实布局的引擎通过 DOM 点击。在截图、拖拽、下载、机器人验证墙、文字相对页面过少、缺少 CDP 方法、引擎崩溃或标签页被显示时，标签页以相同 id 转到 Chromium，并给 agent 一个 `notice`；该网站在本次会话中被记住。没有标签页 id 的命令发往 agent 最后打开的标签页。引擎不会得到 cookie 或配置文件。示例引擎：独立的 `canvastty-plugin-lightpanda`。 浏览器卡片在屏幕外时，后台标签页的截图也能正常工作。 - **启动模式。** 现在默认是 Auto。Manual、Accept edits、Plan 与 Bypass（YOLO）只在 CLI 支持时提供：Accept edits 与 Plan 适用于 Claude Code、Codex、Grok 和 OpenCode，Plan 还适用于 Cursor；没有自带自动模式的 CLI，其 Auto 就是跳过审批，且只在智能体隔离内存在。Bypass 需要用户为每个 CLI 确认一次，由主进程检查，且绝不交给子智能体。 - **委派规则。** 子智能体的权限不超过其编排者（plan < manual < accept edits < auto，绝不为 Bypass），只在编排者的项目文件夹内工作，并受用户在 Settings → Agents 中设定的深度（2）与存活子智能体数（8）限制。对于无法询问的 CLI，决策插件的「ask」会变成附带原因的拒绝。 -- **智能体隔离。** 操作系统层（macOS 用 `sandbox-exec`，Linux 用 bubblewrap）包裹子智能体、插件启动的智能体以及所有非 Manual 模式的智能体：只能写入项目、本次启动的临时目录和其 CLI 自己的目录；密钥、其他 CLI 的凭据和 CanvasTTY 的 token 不可读；无法建立时拒绝启动。Windows 暂无隔离层，子智能体在那里以 Manual 运行。关闭隔离需由用户主动选择。 +- **智能体隔离。** 操作系统层（macOS 用 `sandbox-exec`，Linux 用 bubblewrap）包裹子智能体、插件启动的智能体以及所有非 Manual 模式的智能体：只能写入项目、本次启动的临时目录和其 CLI 自己的目录；密钥、其他 CLI 的凭据和 CanvasTTY 的 token 不可读；无法建立时拒绝启动。Windows 暂无隔离层，子智能体在那里以 Manual 运行；在 bubblewrap 无法创建用户命名空间的 Linux 上（Ubuntu 24.04 的 AppArmor 限制）也是如此，卡片会显示原因，并说明[如何允许](docs/installing-and-security.zh-CN.md#linuxbubblewrap-无法启动时)。关闭隔离需由用户主动选择。 - **Git 审计。** 隔离会话结束后，CanvasTTY 会检查它触及的仓库中是否有会在隔离外运行程序的 git 设置和文件，并提供 **Neutralize** 或 **Keep as is**。 - **原生智能体 helper。** `canvastty-helper`（Go）在 macOS 和 Linux 上运行 MCP 服务器、权限检查和生命周期 hook；Windows 默认仍使用 JavaScript helper，`CANVASTTY_HELPERS=node` 可在任何系统上强制使用它们。从源码构建需要 Go 1.21 或更高版本（`npm run build:helpers`）；没有 Go 时使用 JavaScript helper。 - **性能。** 只打包主进程和 preload 实际加载的内容，内置皮肤改用 AVIF；画布镜头放在 React 之外；屏幕外的 DOM 终端不再在滚动时重建；摘要模式和 HOME 编辑中隐藏的终端停止绘制并推迟选区重绘，解析器仍按有界批次接收完整输出以保留终端状态和历史；窗口在服务启动的同时加载，Settings 按需加载；隐藏的浏览器标签页、无人可见的插件帧以及关闭的 Settings 停止轮询。 diff --git a/agent/browser/SKILL.md b/agent/browser/SKILL.md index d941c21e..e93a5dae 100644 --- a/agent/browser/SKILL.md +++ b/agent/browser/SKILL.md @@ -24,6 +24,8 @@ Use this skill when the task needs the visible CanvasTTY browser. The browser is 4. Re-observe after navigation, dialogs, meaningful DOM changes, or any action whose result matters. 5. If the result contains `STALE_REF`, never retry the old ref. Call `browser_observe`, choose the replacement ref from the new revision, and retry once. +`browser_new_tab` may open your tab in the background, in a lighter engine the user installed (`engine: "auto"`, the default). Pass the returned `tabId` to every later call: commands without one go to the tab you opened last. Such a tab reads, observes, types and clicks by element, but has no screenshots; when a screenshot, a bot check or an unreadable page needs Chromium, the tab moves there by itself with the same tab ID, the result carries a `notice`, and old refs are stale. Use `engine: "chromium"` when the task is visual from the start. + Element refs belong to one tab, frame, and document revision. Do not copy a ref between tabs or reuse it after reload/navigation. The user or another agent may change the shared page between your calls; if the document revision changes, re-observe and continue from the new revision instead of guessing what changed. ## Reading and artifacts diff --git a/docs/browser.md b/docs/browser.md index 37faf904..1520c724 100644 --- a/docs/browser.md +++ b/docs/browser.md @@ -37,6 +37,8 @@ Agent mutations are ordered FIFO per tab, deduplicated by request ID, revision-c If the browser view has zero width or height, `browser_observe` returns `VIEWPORT_UNAVAILABLE` instead of a misleading empty list of controls. `browser_screenshot` returns the same retryable error for an empty capture. Bring the Browser card into view, then observe or capture again; reopening the tab is unnecessary. `browser_read_page` can still read document text while no drawable view is available. +An agent's `browser_new_tab` may open its tab in the background in a browser engine a plugin contributes (`engine: "auto"`, the default, when one is installed and running; `"chromium"` forces a normal tab). Such a tab is listed with its `engine`, is never shown, gets no cookies or profile, and moves to Chromium under the same tab id when a screenshot, a bot wall, thin text, a missing capability, an engine crash or showing the tab needs it; the agent's result says so. Tabs the person opens always use Chromium. See [Browser engines](plugins.md#browser-engines-browserengine). A background tab (an agent's own, or one that moved from an engine) can be captured while the Browser card is off-screen; only the tab shown in the card needs the card in view. + ## Website and file boundaries - Remote pages run sandboxed with context isolation and no Node.js or CanvasTTY preload. diff --git a/docs/browser.ru.md b/docs/browser.ru.md index 9218bd33..05bf619c 100644 --- a/docs/browser.ru.md +++ b/docs/browser.ru.md @@ -35,6 +35,8 @@ Agent mutations выполняются FIFO внутри вкладки, дедуплицируются по request ID, проверяют document revision до side effect, ограничены rate limit и timeout и блокируются, если обязательную запись audit attempt нельзя сохранить. Reads могут выполняться параллельно; у разных вкладок независимые mutation lanes. +`browser_new_tab` агента может открыть вкладку в фоне в браузерном движке, который добавил плагин (`engine: "auto"` по умолчанию, если движок установлен и запущен; `"chromium"` даёт обычную вкладку). Такая вкладка видна в списке со своим `engine`, никогда не показывается, не получает cookies и профиль и переходит в Chromium с тем же id, когда нужен скриншот, встретилась защита от ботов, текста слишком мало, движку чего-то не хватает, он упал или вкладку показывают; результат агента об этом сообщает. Вкладки, которые открывает человек, всегда в Chromium. См. [Браузерные движки](plugins.ru.md#браузерные-движки-browserengine). Фоновую вкладку (собственную вкладку агента или переехавшую из движка) можно снять скриншотом, даже когда карточка браузера за пределами экрана; только вкладке, показанной в карточке, нужна видимая карточка. + ## Границы сайтов и файлов - Удалённые страницы работают в sandbox с context isolation, без Node.js и preload CanvasTTY. diff --git a/docs/browser.zh-CN.md b/docs/browser.zh-CN.md index f7a641c6..294a0f35 100644 --- a/docs/browser.zh-CN.md +++ b/docs/browser.zh-CN.md @@ -35,6 +35,8 @@ CanvasTTY `1.0.2` 已从 HOME 提供内置浏览器，它是可信的画布应 智能体 mutation 在每个标签页内按 FIFO 执行，按 request ID 去重，在产生副作用前检查 document revision，并受 rate limit 与 timeout 限制；若必需的审计 attempt 无法写入，该 mutation 会被阻止。read 可以并行执行，不同标签页使用独立 mutation lane。 +agent 的 `browser_new_tab` 可以在插件提供的浏览器引擎中于后台打开标签页（引擎已安装并运行时的默认 `engine: "auto"`；`"chromium"` 强制普通标签页）。这种标签页在列表中带有 `engine`，从不显示，不会得到 cookie 或配置文件；在需要截图、遇到机器人验证墙、文字过少、引擎缺少能力、引擎崩溃或标签页被显示时，它以相同 id 转到 Chromium，并在 agent 的结果中说明。用户打开的标签页始终使用 Chromium。见[插件文档](plugins.zh-CN.md)。 浏览器卡片在屏幕外时，后台标签页（agent 自己的，或从引擎迁移过来的）也可以截图；只有卡片中显示的标签页需要卡片在视野内。 + ## 网站与文件边界 - 远程页面运行在 sandbox 中，启用 context isolation，不含 Node.js 或 CanvasTTY preload。 diff --git a/docs/canvastty-plugin.schema.json b/docs/canvastty-plugin.schema.json index af23562f..72f2e886 100644 --- a/docs/canvastty-plugin.schema.json +++ b/docs/canvastty-plugin.schema.json @@ -37,7 +37,7 @@ "type": "array", "maxItems": 12, "uniqueItems": true, - "items": { "enum": ["storage", "secrets", "sessions:read", "limits:read", "launcher:open", "external:open", "browser:open", "media:library", "playlists:read", "playlists:write", "hermes:hud", "network", "launch:contribute", "environment:provide", "decision:provide", "tools:agents", "sessions:events", "sessions:read-screen", "sessions:launch", "sessions:control", "cards:decorate"] } + "items": { "enum": ["storage", "secrets", "sessions:read", "limits:read", "launcher:open", "external:open", "browser:open", "media:library", "playlists:read", "playlists:write", "hermes:hud", "network", "launch:contribute", "environment:provide", "decision:provide", "tools:agents", "sessions:events", "sessions:read-screen", "sessions:launch", "sessions:control", "cards:decorate", "browser:engine"] } }, "contributions": { "type": "array", @@ -83,7 +83,7 @@ "type": "array", "maxItems": 12, "uniqueItems": true, - "items": { "enum": ["storage", "secrets", "sessions:read", "limits:read", "launcher:open", "external:open", "browser:open", "media:library", "playlists:read", "playlists:write", "hermes:hud", "network", "launch:contribute", "environment:provide", "decision:provide", "tools:agents", "sessions:events", "sessions:read-screen", "sessions:launch", "sessions:control", "cards:decorate"] } + "items": { "enum": ["storage", "secrets", "sessions:read", "limits:read", "launcher:open", "external:open", "browser:open", "media:library", "playlists:read", "playlists:write", "hermes:hud", "network", "launch:contribute", "environment:provide", "decision:provide", "tools:agents", "sessions:events", "sessions:read-screen", "sessions:launch", "sessions:control", "cards:decorate", "browser:engine"] } }, "files": { "type": "array", "minItems": 1, "maxItems": 500, "items": { "$ref": "#/$defs/moduleAsset" } } } @@ -151,7 +151,20 @@ "description": "Card actions (needs cards:decorate): items in the options menu of matching cards; the host calls canvastty.cards.invoke and shows the answer's message as a toast.", "type": "array", "minItems": 1, "maxItems": 8, "items": { "$ref": "#/$defs/cardAction" } - } + }, + "browserEngine": { "$ref": "#/$defs/browserEngine" } + } + }, + "browserEngine": { + "description": "A browser engine for agents' background tabs (needs browser:engine; ids unique within the plugin). For each such tab the host calls canvastty.browserEngine.openTab { engineId, tabId } and expects { webSocketUrl } (a ws:// CDP endpoint on 127.0.0.1, [::1] or localhost with a port); canvastty.browserEngine.closeTab { engineId, tabId } is a notification. The core decides which tabs use it (only agents' new tabs, never shown, no cookies or profile) and moves a tab to Chromium on screenshots, bot walls, thin text, missing CDP methods, a disconnect, or when the tab is shown.", + "type": "object", + "additionalProperties": false, + "required": ["id", "title"], + "properties": { + "id": { "description": "What agents pass as engine to browser_new_tab; never auto or chromium.", "type": "string", "minLength": 1, "maxLength": 64, "pattern": "^[a-z0-9](?:[a-z0-9._-]{0,62}[a-z0-9])?$", "not": { "enum": ["auto", "chromium"] } }, + "title": { "type": "string", "minLength": 1, "maxLength": 80 }, + "description": { "type": "string", "minLength": 1, "maxLength": 240 }, + "layout": { "description": "The engine lays pages out for real (boxes and viewport mean something). false or omitted: observation skips geometry and clicks and hovers go through the DOM (element.click()).", "type": "boolean" } } }, "agentTool": { diff --git a/docs/installing-and-security.md b/docs/installing-and-security.md index 135735eb..d9385511 100644 --- a/docs/installing-and-security.md +++ b/docs/installing-and-security.md @@ -67,6 +67,25 @@ OpenCode Auto inspects only regular local config files of at most 1 MiB. An exis **Git audit.** When an isolated agent session ends, is closed or is restored after a quit, CanvasTTY checks the repositories under its folder whose git folder changed. If git would now run something outside isolation (a `core.hooksPath`, a filter or diff driver, `fsmonitor`, a hook file, `info/attributes`), a notice lists exactly what changed; **Neutralize** removes those keys and disables those files, **Keep as is** leaves them. It does not undo other file changes inside the project. +### Linux: when bubblewrap cannot start + +Ubuntu 24.04 and later (and other distributions with `kernel.apparmor_restrict_unprivileged_userns=1`) let only programs with an AppArmor profile create unprivileged user namespaces, which bubblewrap needs. CanvasTTY checks this once (again a minute after a failure) and, when `bwrap` is installed but cannot start, treats it like a computer without an isolation layer: subagents and plugin-started agents run in Manual, and the card says why. To allow it, either give bubblewrap its own profile (recommended, it affects only `bwrap`): + +```sh +sudo tee /etc/apparmor.d/bwrap >/dev/null <<'EOF' +abi , +include + +profile bwrap /usr/bin/bwrap flags=(unconfined) { + userns, + include if exists +} +EOF +sudo apparmor_parser -r /etc/apparmor.d/bwrap +``` + +or lift the restriction for every program with `sudo sysctl kernel.apparmor_restrict_unprivileged_userns=0` (add the line to `/etc/sysctl.d/60-userns.conf` to keep it after a reboot). New launches use the layer within a minute; no restart is needed. + ## Repository guards ```bash diff --git a/docs/installing-and-security.ru.md b/docs/installing-and-security.ru.md index eedadfba..3ff3e588 100644 --- a/docs/installing-and-security.ru.md +++ b/docs/installing-and-security.ru.md @@ -59,6 +59,25 @@ **Аудит git.** Когда изолированная сессия агента заканчивается, закрывается или восстанавливается после выхода, CanvasTTY проверяет репозитории в её папке, у которых изменилась папка git. Если git теперь запустит что-то вне изоляции (`core.hooksPath`, фильтр или diff-драйвер, `fsmonitor`, файл хука, `info/attributes`), появляется уведомление со списком изменений; **Обезвредить** удаляет эти ключи и отключает файлы, **Оставить как есть** ничего не трогает. Другие изменения файлов в проекте аудит не откатывает. +### Linux: если bubblewrap не запускается + +Ubuntu 24.04 и новее (и другие дистрибутивы с `kernel.apparmor_restrict_unprivileged_userns=1`) разрешают создавать непривилегированные пространства имён пользователя только программам с профилем AppArmor, а bubblewrap без них не работает. CanvasTTY проверяет это один раз (после неудачи — снова через минуту) и, если `bwrap` установлен, но не запускается, ведёт себя как на компьютере без слоя изоляции: субагенты и агенты от плагинов работают вручную, а карточка объясняет почему. Чтобы разрешить, дайте bubblewrap собственный профиль (рекомендуется, касается только `bwrap`): + +```sh +sudo tee /etc/apparmor.d/bwrap >/dev/null <<'EOF' +abi , +include + +profile bwrap /usr/bin/bwrap flags=(unconfined) { + userns, + include if exists +} +EOF +sudo apparmor_parser -r /etc/apparmor.d/bwrap +``` + +или снимите ограничение для всех программ: `sudo sysctl kernel.apparmor_restrict_unprivileged_userns=0` (строка в `/etc/sysctl.d/60-userns.conf` сохранит это после перезагрузки). Новые запуски получат слой в течение минуты, перезапуск не нужен. + ## Защита репозитория ```bash diff --git a/docs/installing-and-security.zh-CN.md b/docs/installing-and-security.zh-CN.md index 1f45f704..8fb06f85 100644 --- a/docs/installing-and-security.zh-CN.md +++ b/docs/installing-and-security.zh-CN.md @@ -59,6 +59,25 @@ **Git 审计。** 隔离的智能体会话结束、被关闭或在退出后恢复时，CanvasTTY 会检查其文件夹下 git 目录发生变化的仓库。如果 git 现在会在隔离外运行某些东西（`core.hooksPath`、filter 或 diff 驱动、`fsmonitor`、hook 文件、`info/attributes`），会出现一条列出具体变化的通知；**Neutralize** 删除这些键并停用这些文件，**Keep as is** 保持不变。它不会撤销项目内的其他文件改动。 +### Linux：bubblewrap 无法启动时 + +Ubuntu 24.04 及更新版本（以及其他设置了 `kernel.apparmor_restrict_unprivileged_userns=1` 的发行版）只允许带 AppArmor 配置文件的程序创建非特权用户命名空间，而 bubblewrap 需要它。CanvasTTY 会检查一次（失败后一分钟再查），如果 `bwrap` 已安装却无法启动，就按没有隔离层的电脑处理：子智能体和插件启动的智能体以手动模式运行，卡片会说明原因。要允许它，可以为 bubblewrap 单独添加配置文件（推荐，只影响 `bwrap`）： + +```sh +sudo tee /etc/apparmor.d/bwrap >/dev/null <<'EOF' +abi , +include + +profile bwrap /usr/bin/bwrap flags=(unconfined) { + userns, + include if exists +} +EOF +sudo apparmor_parser -r /etc/apparmor.d/bwrap +``` + +或者对所有程序解除限制：`sudo sysctl kernel.apparmor_restrict_unprivileged_userns=0`（写入 `/etc/sysctl.d/60-userns.conf` 可在重启后保留）。新的启动会在一分钟内用上隔离层，无需重启。 + ## 仓库防护 ```bash diff --git a/docs/plugin-api.d.ts b/docs/plugin-api.d.ts index 6a5bffd1..8ab1b19a 100644 --- a/docs/plugin-api.d.ts +++ b/docs/plugin-api.d.ts @@ -79,7 +79,7 @@ export interface CanvasTTYPluginContext { id: string; name: string; version: string; - permissions: Array<"storage" | "secrets" | "sessions:read" | "limits:read" | "launcher:open" | "external:open" | "browser:open" | "media:library" | "playlists:read" | "playlists:write" | "hermes:hud" | "network" | "launch:contribute" | "environment:provide" | "decision:provide" | "tools:agents" | "sessions:events" | "sessions:read-screen" | "sessions:launch" | "sessions:control" | "cards:decorate">; + permissions: Array<"storage" | "secrets" | "sessions:read" | "limits:read" | "launcher:open" | "external:open" | "browser:open" | "media:library" | "playlists:read" | "playlists:write" | "hermes:hud" | "network" | "launch:contribute" | "environment:provide" | "decision:provide" | "tools:agents" | "sessions:events" | "sessions:read-screen" | "sessions:launch" | "sessions:control" | "cards:decorate" | "browser:engine">; modules: string[]; }; contribution: { @@ -175,6 +175,22 @@ export interface CanvasTTYPluginServiceManifestEntry { tools?: CanvasTTYAgentTool[]; /** Card actions; needs `cards:decorate`. Up to 8; ids unique within the plugin. */ cardActions?: CanvasTTYCardAction[]; + /** A browser engine for agents' background tabs; needs `browser:engine`. Ids unique within the plugin. */ + browserEngine?: CanvasTTYBrowserEngine; +} + +/** + * The host calls `canvastty.browserEngine.openTab` `{ engineId, tabId }` for each agent background tab and expects + * `{ webSocketUrl }`: a `ws://` CDP endpoint on 127.0.0.1, [::1] or localhost with a port, one page per connection. + * `canvastty.browserEngine.closeTab` `{ engineId, tabId }` is a notification. + */ +export interface CanvasTTYBrowserEngine { + /** What agents pass as `engine` to browser_new_tab: `^[a-z0-9](?:[a-z0-9._-]{0,62}[a-z0-9])?$`, never `auto` or `chromium`. */ + id: string; + title: string; + description?: string; + /** Real layout (boxes, viewport). Omitted or false: observation skips geometry, clicks and hovers go through the DOM. */ + layout?: boolean; } export type CanvasTTYSessionRole = "agent" | "orchestrator" | "subagent"; diff --git a/docs/plugins.md b/docs/plugins.md index 10a9896e..0047ad17 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -385,6 +385,23 @@ A service with `cards:decorate` can put a badge on any card and declare up to 8 The full example is [`examples/plugins/collect-demo`](../examples/plugins/collect-demo): the card action **Show changes** on cards in the `worktree` environment (from `env-worktree`) shows `git diff --stat` of the worktree and sets a "N changed" badge, and the tool `collect-demo__diffstat` gives orchestrators the same for their own folder or a subagent's, which it learns about from session events. +### Browser engines (`browser:engine`) + +A service may run another browser engine for agents' background tabs, for example a headless engine that reads pages with far less CPU and memory than Chromium. It declares one `browserEngine`: + +```json +"permissions": ["browser:engine"], +"services": [{ + "id": "engine", "title": "Engine", "entry": "services/engine.mjs", + "browserEngine": { "id": "lightpanda", "title": "Lightpanda", "layout": false } +}] +``` + +- `id` is what agents pass as `engine` to `browser_new_tab` (`[a-z0-9][a-z0-9._-]*`, at most 64 characters, never `auto` or `chromium`, unique within the plugin). `layout` says the engine lays pages out for real; without it (the default) observation skips element geometry and clicks and hovers go through the DOM (`element.click()`), so an engine with synthetic boxes still works. +- For each tab the host calls `canvastty.browserEngine.openTab` `{ engineId, tabId }` (host-only, 20 s) and expects `{ webSocketUrl }`: a `ws://` CDP endpoint on `127.0.0.1`, `[::1]` or `localhost` with a port, anything else is refused. The core opens its own connection there, creates the page with `Target.createTarget` and drives it over a flat session, one connection per tab. `canvastty.browserEngine.closeTab` `{ engineId, tabId }` (a notification) says the tab is gone, so the service can stop its process when idle. The service starts, supervises and stops the engine process itself. +- The core owns the policy. Only an agent's new tab may use an engine: with `engine: "auto"` (the default) the first running engine, or the one it names. A tab the person opens, and `engine: "chromium"`, always get Chromium. An engine tab is never shown: the person or an agent showing it moves it to Chromium first. The engine gets no cookies, no browser profile and no credentials, only the URL; the tab's page starts empty. +- Fallback: the tab moves to Chromium under the same tab id (next document revision, refs stale) when a screenshot, a drag or a download is asked for, the page looks like a bot wall (a challenge title or text, a `/cdn-cgi/challenge-platform/` request, a 403, 429 or 503 document), the text is far too thin for the page's size, the engine lacks a CDP method (`-32601`), or the engine disconnects. The agent's result carries a `notice`; an action on an element ref answers `STALE_REF` with `details.movedToChromium`. A site that failed this way goes straight to Chromium for the rest of the session. If `openTab` fails, the tab opens in Chromium and `auto` skips that engine for a minute. + ### Base protection and redaction (core) Two safety parts are built in and need no plugin: @@ -410,6 +427,7 @@ host.onStorageChange(listener) notifies every live contribution of the same plug | `sessions:launch` | `sessions.create` | Starts visible agent cards through the normal launch | | `sessions:control` | `sessions.send`, `sessions.stop` | Types into and closes only the cards the plugin started | | `cards:decorate` | `cards.setBadge`, a service's `cardActions`, `canvastty.cards.invoke` | Plain-text badges on cards and actions in their menu | +| `browser:engine` | A service's `browserEngine` and `canvastty.browserEngine.*` | Receives the URLs of agents' background tabs and serves them from its own engine; no cookies, profile or credentials | | `limits:read` | `limits.get` | The same sanitized `LimitsSnapshot` used by HOME | | `launcher:open` | `launcher.open` | Opens the built-in provider Focus Card or terminal action; it does not bypass user launch choices | | `external:open` | `external.open` | Opens only an explicit HTTP(S) URL through the OS | diff --git a/docs/plugins.ru.md b/docs/plugins.ru.md index 53f5c453..5d2cfe9d 100644 --- a/docs/plugins.ru.md +++ b/docs/plugins.ru.md @@ -363,6 +363,23 @@ interface PluginSessionEvent { Полный пример — [`examples/plugins/collect-demo`](../examples/plugins/collect-demo): действие **Show changes** на окнах в среде `worktree` (из `env-worktree`) показывает `git diff --stat` worktree и ставит метку «N changed», а инструмент `collect-demo__diffstat` даёт оркестраторам то же для своей папки или папки субагента, о котором плагин узнаёт из событий сессий. +### Браузерные движки (`browser:engine`) + +Сервис может запускать другой браузерный движок для фоновых вкладок агентов, например headless-движок, который читает страницы с гораздо меньшими затратами CPU и памяти, чем Chromium. Он объявляет один `browserEngine`: + +```json +"permissions": ["browser:engine"], +"services": [{ + "id": "engine", "title": "Engine", "entry": "services/engine.mjs", + "browserEngine": { "id": "lightpanda", "title": "Lightpanda", "layout": false } +}] +``` + +- `id` агенты передают как `engine` в `browser_new_tab` (`[a-z0-9][a-z0-9._-]*`, не длиннее 64 символов, не `auto` и не `chromium`, уникален в плагине). `layout` означает, что движок по-настоящему раскладывает страницу; без него (по умолчанию) наблюдение не использует геометрию элементов, а клики и наведение идут через DOM (`element.click()`), так что движок с фиктивными координатами тоже работает. +- Для каждой вкладки хост вызывает `canvastty.browserEngine.openTab` `{ engineId, tabId }` (только хост, 20 с) и ждёт `{ webSocketUrl }`: CDP-адрес `ws://` на `127.0.0.1`, `[::1]` или `localhost` с портом, иначе отказ. Ядро само подключается туда, создаёт страницу через `Target.createTarget` и управляет ею через плоскую сессию, одно соединение на вкладку. `canvastty.browserEngine.closeTab` `{ engineId, tabId }` (уведомление) сообщает, что вкладки больше нет, и сервис может остановить процесс при простое. Процесс движка сервис запускает, перезапускает и останавливает сам. +- Правила задаёт ядро. Движок может получить только новую вкладку агента: при `engine: "auto"` (по умолчанию) первый запущенный движок или тот, что назван. Вкладка, которую открывает человек, и `engine: "chromium"` всегда идут в Chromium. Вкладка движка никогда не показывается: если её показывает человек или агент, она сначала переходит в Chromium. Движок не получает cookies, профиль браузера и учётные данные — только URL; страница вкладки начинается пустой. +- Переход в Chromium: вкладка переходит туда с тем же id (следующая ревизия документа, прежние ref устарели), когда просят скриншот, перетаскивание или загрузку, когда страница похожа на защиту от ботов (заголовок или текст проверки, запрос `/cdn-cgi/challenge-platform/`, документ с кодом 403, 429 или 503), когда текста слишком мало для размера страницы, когда у движка нет метода CDP (`-32601`) или он отключился. В результате агента есть `notice`; действие по ref элемента отвечает `STALE_REF` с `details.movedToChromium`. Сайт, на котором так случилось, до конца сессии сразу открывается в Chromium. Если `openTab` не удался, вкладка открывается в Chromium, а `auto` минуту не выбирает этот движок. + ### Базовая защита и скрытие секретов (ядро) Две части безопасности встроены и не требуют плагина: @@ -388,6 +405,7 @@ host.onStorageChange(listener) сообщает всем открытым пов | `sessions:launch` | `sessions.create` | Запускает видимые окна агентов через обычный запуск | | `sessions:control` | `sessions.send`, `sessions.stop` | Вводит текст и закрывает только окна, запущенные плагином | | `cards:decorate` | `cards.setBadge`, `cardActions` сервиса, `canvastty.cards.invoke` | Текстовые метки на окнах и действия в их меню | +| `browser:engine` | `browserEngine` сервиса и `canvastty.browserEngine.*` | Получает URL фоновых вкладок агентов и открывает их в своём движке; без cookies, профиля и учётных данных | | `limits:read` | `limits.get` | Тот же очищенный `LimitsSnapshot`, который использует HOME | | `launcher:open` | `launcher.open` | Открывает штатную Focus Card или запуск терминала; не обходит пользовательский выбор | | `external:open` | `external.open` | Передаёт ОС только явную HTTP(S)-ссылку | diff --git a/docs/plugins.zh-CN.md b/docs/plugins.zh-CN.md index 86b34d03..c91ff2c6 100644 --- a/docs/plugins.zh-CN.md +++ b/docs/plugins.zh-CN.md @@ -361,6 +361,23 @@ interface PluginSessionEvent { 完整示例见 [`examples/plugins/collect-demo`](../examples/plugins/collect-demo)：在 `worktree` 环境（来自 `env-worktree`）的卡片上，动作 **Show changes** 显示 worktree 的 `git diff --stat` 并设置“N changed”标记；工具 `collect-demo__diffstat` 为编排器提供同样的信息，针对它自己的文件夹或某个子 agent 的文件夹（插件通过会话事件得知子 agent）。 +### 浏览器引擎（`browser:engine`） + +服务可以为 agent 的后台标签页运行另一种浏览器引擎，例如以远低于 Chromium 的 CPU 和内存读取页面的无头引擎。它声明一个 `browserEngine`： + +```json +"permissions": ["browser:engine"], +"services": [{ + "id": "engine", "title": "Engine", "entry": "services/engine.mjs", + "browserEngine": { "id": "lightpanda", "title": "Lightpanda", "layout": false } +}] +``` + +- `id` 是 agent 传给 `browser_new_tab` 的 `engine`（`[a-z0-9][a-z0-9._-]*`，最多 64 个字符，不能是 `auto` 或 `chromium`，插件内唯一）。`layout` 表示引擎真正进行页面布局；省略时（默认）观察不使用元素几何信息，点击和悬停通过 DOM（`element.click()`）完成，因此坐标不真实的引擎也能工作。 +- 对每个标签页，宿主调用 `canvastty.browserEngine.openTab` `{ engineId, tabId }`（仅宿主，20 s），并期望得到 `{ webSocketUrl }`：`127.0.0.1`、`[::1]` 或 `localhost` 上带端口的 `ws://` CDP 地址，其他地址会被拒绝。核心自己连接该地址，用 `Target.createTarget` 创建页面并通过扁平会话驱动它，每个标签页一个连接。`canvastty.browserEngine.closeTab` `{ engineId, tabId }`（通知）表示标签页已关闭，服务可在空闲时停止进程。引擎进程由服务自己启动、监管和停止。 +- 策略由核心决定。只有 agent 新建的标签页可以使用引擎：`engine: "auto"`（默认）使用第一个运行中的引擎，或使用指定的引擎。用户打开的标签页和 `engine: "chromium"` 始终使用 Chromium。引擎标签页从不显示：用户或 agent 显示它时，它会先转到 Chromium。引擎不会得到 cookie、浏览器配置文件或凭据，只得到 URL；标签页从空白页开始。 +- 回退：在请求截图、拖拽或下载，页面像机器人验证墙（验证标题或文字、`/cdn-cgi/challenge-platform/` 请求、403/429/503 文档），文字相对页面大小过少，引擎缺少 CDP 方法（`-32601`）或引擎断开时，标签页以相同 id 转到 Chromium（文档版本递增，旧 ref 失效）。agent 的结果带有 `notice`；针对元素 ref 的操作返回带 `details.movedToChromium` 的 `STALE_REF`。以这种方式失败的网站在本次会话剩余时间内直接使用 Chromium。如果 `openTab` 失败，标签页在 Chromium 中打开，`auto` 在一分钟内跳过该引擎。 + ### 基础保护与密钥遮蔽（核心） 两项安全功能内置，无需插件： @@ -386,6 +403,7 @@ host.onStorageChange(listener) 会把 host.storage.set 的写入通知给同一 | `sessions:launch` | `sessions.create` | 通过常规启动流程启动可见的 agent 卡片 | | `sessions:control` | `sessions.send`、`sessions.stop` | 只能向本插件启动的卡片输入文本并关闭它们 | | `cards:decorate` | `cards.setBadge`、服务的 `cardActions`、`canvastty.cards.invoke` | 卡片上的纯文本标记及其菜单中的动作 | +| `browser:engine` | 服务的 `browserEngine` 和 `canvastty.browserEngine.*` | 接收 agent 后台标签页的 URL 并用自己的引擎打开；没有 cookie、配置文件或凭据 | | `limits:read` | `limits.get` | 与 HOME 使用的同一个脱敏 `LimitsSnapshot` | | `launcher:open` | `launcher.open` | 打开内置服务商的 Focus Card 或终端动作；不会绕过用户的启动选择 | | `external:open` | `external.open` | 仅通过操作系统打开明确的 HTTP(S) URL | diff --git a/native/canvastty-helper/catalog.json b/native/canvastty-helper/catalog.json index 1668abfa..d3ec39a2 100644 --- a/native/canvastty-helper/catalog.json +++ b/native/canvastty-helper/catalog.json @@ -15,7 +15,7 @@ }, { "name": "browser_new_tab", - "description": "Open a new visible tab. Re-observe after navigation before using element refs.", + "description": "Open a new tab; use the returned tabId for later commands. engine: auto (default) opens it in the background in a lighter engine when the person installed one (reading, observing, clicking by element), otherwise as a visible Chromium tab; chromium always opens a visible Chromium tab. A background tab moves to Chromium by itself when needed (screenshots, bot checks, unreadable pages, or when shown): same tabId, stale refs, and the result says so.", "inputSchema": { "type": "object", "properties": { @@ -23,6 +23,11 @@ "type": "string", "minLength": 1, "maxLength": 2048 + }, + "engine": { + "type": "string", + "minLength": 1, + "maxLength": 64 } }, "required": [], diff --git a/native/windows-agent-pipe-host/src/main.cc b/native/windows-agent-pipe-host/src/main.cc index 2693ae9f..a1b3a803 100644 --- a/native/windows-agent-pipe-host/src/main.cc +++ b/native/windows-agent-pipe-host/src/main.cc @@ -713,12 +713,33 @@ bool SelfTest() { return server_ok && client_ok.load(); } -bool ParseParentPid(int argc, wchar_t** argv, DWORD* parent_pid) { - if (argc != 3 || std::wcscmp(argv[1], L"--parent-pid") != 0) return false; +// A restarted host may reuse the name its failed predecessor published, so clients that already hold it +// (and their reconnect tokens) can come back. Only names this host itself would generate are accepted. +bool IsGeneratedPipeName(const wchar_t* value) { + constexpr wchar_t kPrefix[] = L"\\\\.\\pipe\\canvastty-agent-"; + const std::size_t prefix_length = std::wcslen(kPrefix); + if (std::wcsncmp(value, kPrefix, prefix_length) != 0) return false; + const wchar_t* suffix = value + prefix_length; + if (std::wcslen(suffix) != 32) return false; + for (const wchar_t* cursor = suffix; *cursor != L'\0'; ++cursor) { + const bool digit = *cursor >= L'0' && *cursor <= L'9'; + const bool lower_hex = *cursor >= L'a' && *cursor <= L'f'; + if (!digit && !lower_hex) return false; + } + return true; +} + +bool ParseArguments(int argc, wchar_t** argv, DWORD* parent_pid, std::wstring* pipe_name) { + if ((argc != 3 && argc != 5) || std::wcscmp(argv[1], L"--parent-pid") != 0) return false; wchar_t* end = nullptr; const unsigned long value = std::wcstoul(argv[2], &end, 10); if (end == argv[2] || *end != L'\0' || value == 0 || value > MAXDWORD) return false; *parent_pid = static_cast(value); + pipe_name->clear(); + if (argc == 5) { + if (std::wcscmp(argv[3], L"--pipe-name") != 0 || !IsGeneratedPipeName(argv[4])) return false; + pipe_name->assign(argv[4]); + } return true; } @@ -743,8 +764,9 @@ int wmain(int argc, wchar_t** argv) { } DWORD parent_pid = 0; - if (!ParseParentPid(argc, argv, &parent_pid)) { - SendTextFrame(FrameType::kFatal, "Expected --parent-pid ."); + std::wstring requested_pipe_name; + if (!ParseArguments(argc, argv, &parent_pid, &requested_pipe_name)) { + SendTextFrame(FrameType::kFatal, "Expected --parent-pid [--pipe-name ]."); return 13; } HANDLE parent = OpenProcess(SYNCHRONIZE, FALSE, parent_pid); @@ -758,7 +780,7 @@ int wmain(int argc, wchar_t** argv) { CloseHandle(parent); return 15; } - const std::wstring pipe_name = RandomPipeName(); + const std::wstring pipe_name = requested_pipe_name.empty() ? RandomPipeName() : requested_pipe_name; if (pipe_name.empty()) { SendTextFrame(FrameType::kFatal, "Secure pipe-name generation failed."); CloseHandle(parent); diff --git a/src/agent-browser/tool-catalog.mjs b/src/agent-browser/tool-catalog.mjs index 4fd855c4..9ca0bf55 100644 --- a/src/agent-browser/tool-catalog.mjs +++ b/src/agent-browser/tool-catalog.mjs @@ -41,7 +41,10 @@ function tool(name, description, properties = {}, required = []) { export const TOOL_DEFINITIONS = Object.freeze([ tool("browser_list_tabs", "List visible browser tabs and their stable tab IDs."), - tool("browser_new_tab", "Open a new visible tab. Re-observe after navigation before using element refs.", { url }), + tool("browser_new_tab", "Open a new tab; use the returned tabId for later commands. engine: auto (default) opens it in the background in a lighter engine when the person installed one (reading, observing, clicking by element), otherwise as a visible Chromium tab; chromium always opens a visible Chromium tab. A background tab moves to Chromium by itself when needed (screenshots, bot checks, unreadable pages, or when shown): same tabId, stale refs, and the result says so.", { + url, + engine: string({ minLength: 1, maxLength: 64 }) + }), tool("browser_close_tab", "Close a visible tab by stable tab ID.", { tabId }, ["tabId"]), tool("browser_activate_tab", "Make a tab active and visible.", { tabId }, ["tabId"]), tool("browser_navigate", "Navigate a tab, or the active tab when tabId is omitted, to an HTTP(S) URL.", { diff --git a/src/main/index.ts b/src/main/index.ts index 3bd22a55..b698874f 100644 --- a/src/main/index.ts +++ b/src/main/index.ts @@ -51,7 +51,7 @@ import { AgentControlService } from "./services/AgentControlService"; import { AgentIsolation } from "./services/isolation/AgentIsolation"; import type { AgentProviderId, LaunchProfileId } from "../shared/contracts"; import { HermesHudService } from "./services/HermesHudService"; -import { BrowserService } from "./services/BrowserService"; +import { BrowserService, type BrowserServiceOptions } from "./services/BrowserService"; import { CanvasNavigationInputController } from "./services/CanvasNavigationOverride"; import { activeCanvasWheelBinding } from "../shared/canvasNavigation"; import type { ProviderSmokeTarget } from "./services/browser/ProviderElectronSmoke"; @@ -118,6 +118,8 @@ protocol.registerSchemesAsPrivileged([ } ]); +/** A contributed browser engine may have to start its process before it can open a tab. */ +const BROWSER_ENGINE_OPEN_TIMEOUT_MS = 20_000; let mainWindow: BrowserWindow | null = null; let evenG2: EvenG2Controller | null = null; const browserRequests = new Map }>(); @@ -402,8 +404,20 @@ async function initializeServices(ipc: IpcRegistrar): Promise { browserService = new BrowserService(() => mainWindow, { userDataPath, restoreTabs: settings.get().browserRestoreTabs, + pauseHiddenTabs: settings.get().browserPauseHiddenTabs, + ...browserLifecycleTimingOverride(process.env.CANVASTTY_BROWSER_LIFECYCLE_MS), canvasWheelCaptureMode: settings.get().canvasWheelCaptureMode, canvasNavigationInput, + // Agents' background tabs may run in a plugin-contributed engine (browser:engine); policy stays in the core. + engines: { + providers: () => pluginManager!.browserEngineProviders() + .filter((provider) => pluginServices!.running(provider.pluginId, provider.serviceId)), + openTab: (provider, tabId) => pluginServices!.hostCall(provider.pluginId, provider.serviceId, + "canvastty.browserEngine.openTab", { engineId: provider.engineId, tabId }, BROWSER_ENGINE_OPEN_TIMEOUT_MS), + closeTab: (provider, tabId) => { + pluginServices?.notify(provider.pluginId, provider.serviceId, "canvastty.browserEngine.closeTab", { engineId: provider.engineId, tabId }); + } + }, ...(process.env.CANVASTTY_BROWSER_SMOKE_URL ? { downloadRoot: join(userDataPath, "browser-smoke-downloads") } : {}) @@ -758,6 +772,7 @@ async function initializeServices(ipc: IpcRegistrar): Promise { browserService?.setRestoreTabs(next.browserRestoreTabs).catch((error: unknown) => { console.warn("CanvasTTY browser tab restore setting could not be applied.", error); }); + browserService?.setPauseHiddenTabs(next.browserPauseHiddenTabs); browserService?.cancelCanvasNavigationGesture(); browserService?.setCanvasWheelCaptureMode(next.canvasWheelCaptureMode); canvasNavigationInput?.setBindings({ @@ -1267,3 +1282,13 @@ function securePluginStorageAvailable(): boolean { if (!safeStorage.isEncryptionAvailable()) return false; return process.platform !== "linux" || safeStorage.getSelectedStorageBackend() !== "basic_text"; } + +/** + * Shorter pause/sleep delays for measurements and smoke runs: "freezeMs,discardMs" (for example "3000,8000"). + * Anything else is ignored and the defaults (30 s, 10 min) apply. + */ +function browserLifecycleTimingOverride(value: string | undefined): Pick { + const match = /^(\d{3,9}),(\d{3,9})$/u.exec(value ?? ""); + if (!match) return {}; + return { tabLifecycle: { freezeAfterMs: Number(match[1]), discardAfterMs: Number(match[2]) } }; +} diff --git a/src/main/services/BrowserService.ts b/src/main/services/BrowserService.ts index 8f893c17..5148151e 100644 --- a/src/main/services/BrowserService.ts +++ b/src/main/services/BrowserService.ts @@ -10,7 +10,7 @@ import { View, WebContentsView } from "electron"; -import type { DownloadItem, Session, WebContents, WebPreferences } from "electron"; +import type { DownloadItem, NavigationEntry, Session, WebContents, WebPreferences } from "electron"; import type { AgentPresenceSnapshot, BrowserActivityEvent, @@ -41,9 +41,17 @@ import { BrowserCanvasSinkViewportController } from "./browser/BrowserCanvasSink import { BrowserAuditStore } from "./browser/BrowserAuditStore.ts"; import { browserPageWheelReply, type BrowserPageWheelReply } from "./browser/BrowserCanvasWheel.ts"; import { clipBrowserViewportBounds, normalizeBrowserViewportBounds, sameBrowserViewport } from "./browser/BrowserViewport.ts"; -import { BrowserCore, type BrowserCoreHost, type BrowserCoreTab } from "./browser/BrowserCore.ts"; +import { BrowserCore, type BrowserCoreHost, type BrowserCoreTab, type BrowserOpenedTab } from "./browser/BrowserCore.ts"; +import { BrowserEngineTabs, type BrowserEngineProvider } from "./browser/BrowserEngineTabs.ts"; +import type { EngineFallbackReason } from "./browser/BrowserErrors.ts"; +import { CHROMIUM_ENGINE } from "./browser/TabDriver.ts"; import { BrowserKernelError } from "./browser/BrowserErrors.ts"; import { NativeViewSync } from "./browser/NativeViewSync.ts"; +import { + BrowserTabLifecycle, + type TabLifecycleOptions, + type TabLifecycleState +} from "./browser/BrowserTabLifecycle.ts"; import { BrowserPolicyService, DEFAULT_BROWSER_URL, @@ -61,6 +69,16 @@ const BROWSER_PARTITION = "persist:canvastty-browser"; const MAX_DOWNLOAD_HISTORY = 100; const MAX_FAVICON_BYTES = 256 * 1024; const HUMAN_ACTOR: BrowserActor = { kind: "human", connectionId: "canvastty-renderer" }; +/** A sleeping tab's card picture: small enough to ride along in every browser state update. */ +const SLEEP_PREVIEW_MAX_WIDTH = 480; +const SLEEP_PREVIEW_JPEG_QUALITY = 60; +/** Waits bounded so a stuck page cannot hold a tab's lifecycle queue (and the commands behind it). */ +const SLEEP_CAPTURE_TIMEOUT_MS = 2_000; +const WAKE_LOAD_TIMEOUT_MS = 10_000; +/** How long a tab that moved from a contributed engine to Chromium may load before the command runs anyway. */ +const ENGINE_FALLBACK_LOAD_TIMEOUT_MS = 15_000; +/** A background tab has never been shown and has no size; screenshots and layout need one. */ +const BACKGROUND_TAB_SIZE = { width: 1280, height: 800 }; // Mirrors the `.browser-card { border-radius: 17px }` declaration in src/renderer/src/styles/app.css: the // stylesheet owns this visual property, so the two have to stay in sync by hand. @@ -87,6 +105,22 @@ interface BrowserTab { canvasSinkViewport: BrowserCanvasSinkViewportController; /** Mirrors `view.setVisible` so background throttling can be recomputed without an Electron getter. */ visible: boolean; + /** Paused by the hidden-tab lifecycle (CDP Page.setWebLifecycleState frozen). */ + frozen: boolean; + /** Between media-started-playing and media-paused: a playing tab is never paused. */ + mediaPlaying: boolean; + /** Set while the tab sleeps: its WebContents is closed and this is what brings it back. */ + sleeping: SleepingTab | null; +} + +interface SleepingTab { + title: string; + canGoBack: boolean; + canGoForward: boolean; + /** Back/forward history with each entry's page state (scroll position, form state), for navigationHistory.restore. */ + entries: NavigationEntry[]; + index: number; + preview: string | null; } interface DownloadWaiter { @@ -107,6 +141,19 @@ export interface BrowserServiceOptions { canvasWheelCaptureMode?: CanvasWheelCaptureMode; now?: () => number; canvasNavigationInput?: CanvasNavigationInputController; + /** "Pause hidden browser tabs": freeze hidden tabs after a while and put long-hidden ones to sleep. */ + pauseHiddenTabs?: boolean; + /** Timing overrides for the hidden-tab lifecycle (measurements and smoke runs). */ + tabLifecycle?: Pick; + /** Plugin-contributed browser engines for agents' background tabs (`browser:engine`). */ + engines?: BrowserEngineHost; +} + +/** How BrowserService reaches the plugin services that contribute browser engines. */ +export interface BrowserEngineHost { + providers(): BrowserEngineProvider[]; + openTab(provider: BrowserEngineProvider, tabId: string): Promise; + closeTab(provider: BrowserEngineProvider, tabId: string): void; } export class BrowserService { @@ -119,8 +166,19 @@ export class BrowserService { private readonly policy: BrowserPolicyService; private readonly audit: BrowserAuditStore; private readonly busyAutomationTabs = new Set(); - private readonly automation = new BrowserAutomationService((tabId, busy) => this.setTabAutomationBusy(tabId, busy)); + private readonly lifecycle: BrowserTabLifecycle; + /** Sleeping tabs a show has already asked to wake, so a burst of view syncs asks once. */ + private readonly wakingTabs = new Set(); + private readonly automation = new BrowserAutomationService( + (tabId, busy) => this.setTabAutomationBusy(tabId, busy), + { + beforeCommand: async (tabId) => { await this.lifecycle.ensureLive(tabId); }, + captureSurface: async (tabId) => this.lendCaptureSurface(tabId) + } + ); private readonly agents: AgentRegistry; + /** Agents' background tabs in plugin-contributed engines: no view, never shown, never saved. */ + private readonly engineTabs: BrowserEngineTabs; private readonly canvasGestures: BrowserCanvasGestureController; private readonly canvasPointers: BrowserCanvasPointerRouter; private readonly clipView = new View(); @@ -148,6 +206,8 @@ export class BrowserService { private restoreTabsEnabled: boolean; private clipOwnerId: number | null = null; private clipTabId: string | null = null; + /** Background tabs whose view is in the window only for a screenshot (lendCaptureSurface). */ + private readonly lentTabs = new Set(); private pointerTabId: string | null = null; private presenceWindow: BrowserWindow | null = null; private presenceWindowReady: Promise | null = null; @@ -159,6 +219,18 @@ export class BrowserService { const userDataPath = options.userDataPath ?? app.getPath("userData"); const downloadRoot = join(options.downloadRoot ?? join(app.getPath("downloads"), "CanvasTTY"), randomUUID()); this.restoreTabsEnabled = options.restoreTabs ?? true; + this.lifecycle = new BrowserTabLifecycle({ + freezeBlocker: (tabId) => this.freezeBlocker(tabId), + discardBlocker: (tabId) => this.discardBlocker(tabId), + freeze: (tabId) => this.freezeTab(tabId), + resume: (tabId) => this.resumeTab(tabId), + discard: (tabId) => this.sleepTab(tabId), + restore: (tabId) => this.wakeTab(tabId), + stateChanged: (tabId, state) => this.lifecycleChanged(tabId, state) + }, { + enabled: options.pauseHiddenTabs ?? true, + ...options.tabLifecycle + }); this.store = new BrowserStore(userDataPath); this.policy = new BrowserPolicyService({ downloadRoot, @@ -168,6 +240,20 @@ export class BrowserService { this.audit = new BrowserAuditStore(userDataPath, { now: this.now }); // Presence that stopped heartbeating expires on its own; the timer runs only while an agent is present. this.agents = new AgentRegistry(this.now, { onExpired: () => this.presenceChanged() }); + const engines = options.engines; + this.engineTabs = new BrowserEngineTabs({ + automation: this.automation, + providers: () => engines?.providers() ?? [], + openEngineTab: (provider, tabId) => engines + ? engines.openTab(provider, tabId) + : Promise.reject(new Error("No browser engine is installed.")), + closeEngineTab: (provider, tabId) => engines?.closeTab(provider, tabId), + openChromiumTab: (tabId, url, revision, openOptions) => this.openBackgroundChromiumTab(tabId, url, revision, openOptions), + changed: () => { + if (!this.disposed) this.emit(); + }, + now: this.now + }); this.canvasGestures = new BrowserCanvasGestureController({ getOwner: () => this.getOwner(), getViewport: () => this.viewport, @@ -220,6 +306,9 @@ export class BrowserService { getTab: (tabId) => this.coreTab(tabId), ensureRuntime: () => this.ensureRuntime(), newTab: (url) => this.hostNewTab(url), + openTab: (url, openOptions) => this.hostOpenTab(url, openOptions), + tabEngine: (tabId) => this.engineTabs.engineOf(tabId) ?? (this.tabs.has(tabId) ? CHROMIUM_ENGINE : null), + moveTabToChromium: (tabId, reason) => this.moveTabToChromium(tabId, reason), closeTab: (tabId) => this.hostCloseTab(tabId), activateTab: (tabId) => this.hostActivateTab(tabId), navigateTab: (tabId, url) => this.hostNavigate(tabId, url), @@ -230,7 +319,8 @@ export class BrowserService { waitForDownload: (tabId, timeoutMs, signal) => this.waitForDownload(tabId, timeoutMs, signal), touchActor: (actor, tabId, cursor) => this.touchActor(actor, tabId, cursor), heartbeatActor: (actor, timestamp) => this.heartbeatActor(actor, timestamp), - disconnectActor: (actor) => this.disconnectActor(actor) + disconnectActor: (actor) => this.disconnectActor(actor), + prepareTab: (tabId) => this.lifecycle.ensureLive(tabId) }; this.core = new BrowserCore({ host, @@ -249,9 +339,12 @@ export class BrowserService { getState(): BrowserSnapshot { const agentValues = this.agents.snapshot(); const runtimeTabs = [...this.tabs.values()]; - const tabs = runtimeTabs.length > 0 - ? runtimeTabs.map((tab) => this.tabSnapshot(tab, agentValues)) - : this.persisted.tabs.map((tab) => this.persistedTabSnapshot(tab, agentValues)); + const tabs = [ + ...runtimeTabs.length > 0 + ? runtimeTabs.map((tab) => this.tabSnapshot(tab, agentValues)) + : this.persisted.tabs.map((tab) => this.persistedTabSnapshot(tab, agentValues)), + ...this.engineTabs.snapshots(agentValues) + ]; return { tabs, activeTabId: runtimeTabs.length > 0 ? this.activeTabId : this.persisted.activeTabId, @@ -323,10 +416,15 @@ export class BrowserService { } } + setPauseHiddenTabs(enabled: boolean): void { + this.lifecycle.setEnabled(enabled); + } + async dispose(): Promise { if (this.disposed) return; await this.readyPromise.catch(() => undefined); this.disposed = true; + this.lifecycle.dispose(); const draining = this.core.shutdown(); await this.persistRuntime().catch(() => undefined); this.visible = false; @@ -334,6 +432,7 @@ export class BrowserService { this.canvasGestures.endSequence(false); this.canvasGestures.clear(); this.agents.dispose(); + this.engineTabs.dispose(); this.destroyRuntimeTabs(); if (this.browserSession && this.browserPagePreloadId) { this.browserSession.unregisterPreloadScript(this.browserPagePreloadId); @@ -485,7 +584,7 @@ export class BrowserService { this.requireOwner(); this.visible = true; for (const [id, tab] of this.tabs) { - if (!tab.view.webContents.isDestroyed()) continue; + if (tab.sleeping || !tab.view.webContents.isDestroyed()) continue; this.automation.unregister(id); this.tabs.delete(id); } @@ -504,6 +603,8 @@ export class BrowserService { } private coreTab(tabId: string): BrowserCoreTab | null { + const engineTab = this.engineTabs.coreTab(tabId); + if (engineTab) return engineTab; const tab = this.tabs.get(tabId); if (tab) { return { id: tab.id, url: this.tabUrl(tab), documentRevision: tab.documentRevision, status: tab.status }; @@ -512,9 +613,120 @@ export class BrowserService { return saved ? { id: saved.id, url: saved.url, documentRevision: 0, status: "ready" } : null; } + /** + * An agent's tab goes to a contributed engine when `choose` picks one: it opens in the background and the active tab + * stays. If the engine cannot open it, or none applies, it is a normal Chromium tab (made active, as always). + */ + private async hostOpenTab(url: string, options: { engine?: string; actor: BrowserActor }): Promise { + const choice = this.engineTabs.choose({ engine: options.engine, actor: options.actor, url }); + if (choice.provider) { + await this.ensureRuntime(); + if (this.tabs.size + this.engineTabs.size >= MAX_BROWSER_TABS) { + throw new BrowserKernelError("RATE_LIMITED", `Browser tab limit is ${MAX_BROWSER_TABS}.`); + } + const normalized = this.policy.assertNavigationUrl(url); + try { + const tabId = await this.engineTabs.open(choice.provider, normalized); + return { snapshot: this.getState(), tabId, engine: choice.provider.engineId }; + } catch (error) { + console.warn("CanvasTTY browser engine could not open a tab; using Chromium.", error); + const snapshot = await this.hostNewTab(url); + return { + snapshot, + tabId: snapshot.activeTabId, + engine: CHROMIUM_ENGINE, + ...(options.engine && options.engine !== "auto" + ? { notice: `Browser engine "${choice.provider.engineId}" could not open the tab; it opened in Chromium.` } + : {}) + }; + } + } + const snapshot = await this.hostNewTab(url); + return { snapshot, tabId: snapshot.activeTabId, engine: CHROMIUM_ENGINE, ...(choice.notice ? { notice: choice.notice } : {}) }; + } + + private async moveTabToChromium(tabId: string, reason: EngineFallbackReason): Promise { + await this.ensureRuntime(); + await this.engineTabs.moveToChromium(tabId, reason, { waitForLoad: reason !== "revealed" }); + } + + /** + * The Chromium tab a contributed engine's tab continues in: same id, the given revision, not activated. It gets the + * size of a normal page (it was never on screen) so layout and screenshots work while it stays hidden. + */ + private async openBackgroundChromiumTab( + tabId: string, + url: string, + revision: number, + options: { waitForLoad: boolean } + ): Promise { + if (this.disposed) throw new BrowserKernelError("BRIDGE_UNAVAILABLE", "Browser service is disposed."); + const normalized = this.policy.assertNavigationUrl(url); + const tab = this.createRuntimeTab(tabId, normalized, revision); + this.native.setBounds(tab.view, { + x: 0, + y: 0, + width: Math.max(BACKGROUND_TAB_SIZE.width, Math.round(this.viewport.width)), + height: Math.max(BACKGROUND_TAB_SIZE.height, Math.round(this.viewport.height)) + }); + await this.persistRuntime(); + this.syncViews(); + this.emit(); + const loading = this.loadTab(tab, normalized); + if (!options.waitForLoad) return; + let timer: NodeJS.Timeout | undefined; + await Promise.race([ + loading, + new Promise((resolve) => { + timer = setTimeout(resolve, ENGINE_FALLBACK_LOAD_TIMEOUT_MS); + }) + ]).finally(() => { + if (timer) clearTimeout(timer); + }); + } + + /** + * A screenshot of a background tab (an agent's tab, or one that moved from a contributed engine) while the person + * looks elsewhere. Chromium paints a view only while some of it lies inside a shown window; a view that is hidden, + * in no window, or entirely outside it gives an empty capture. So for the capture the tab's view, at its page size, + * is put in the window with only its top-left pixel inside the window's bottom-right corner, under a one-pixel + * cover in the window's background color, and taken out right after: the whole page is painted and captured, and + * the person sees nothing. The clip view's own tab (the active tab while its card is off-screen) is not lent: its + * page stays hidden while it is busy, and the answer asks for the card to be brought into view as before. + */ + private async lendCaptureSurface(tabId: string): Promise<(() => void) | null> { + const tab = this.tabs.get(tabId); + if (!tab || tab.visible || tab.sleeping || tab.view.webContents.isDestroyed() || this.clipTabId === tab.id) return null; + const owner = this.getOwner(); + if (!owner || owner.isDestroyed()) return null; + const [contentWidth, contentHeight] = owner.getContentSize(); + if (!contentWidth || !contentHeight) return null; + const current = tab.view.getBounds(); + const width = current.width > 0 ? current.width : Math.max(BACKGROUND_TAB_SIZE.width, Math.round(this.viewport.width)); + const height = current.height > 0 ? current.height : Math.max(BACKGROUND_TAB_SIZE.height, Math.round(this.viewport.height)); + this.lentTabs.add(tab.id); + owner.contentView.addChildView(tab.view); + this.native.setBounds(tab.view, { x: contentWidth - 1, y: contentHeight - 1, width, height }); + this.native.setVisible(tab.view, true); + const cover = new View(); + cover.setBackgroundColor(owner.getBackgroundColor() || "#000000"); + owner.contentView.addChildView(cover); + cover.setBounds({ x: contentWidth - 1, y: contentHeight - 1, width: 1, height: 1 }); + return () => { + this.lentTabs.delete(tab.id); + if (owner.isDestroyed()) return; + owner.contentView.removeChildView(cover); + if (owner.contentView.children.includes(tab.view)) owner.contentView.removeChildView(tab.view); + // Closed, put to sleep, or shown (and mounted in the clip view) meanwhile: syncViews owns it then. + if (this.disposed || this.tabs.get(tab.id) !== tab || tab.visible || tab.sleeping || tab.view.webContents.isDestroyed()) return; + this.native.setVisible(tab.view, false); + this.native.setBounds(tab.view, current.width > 0 ? current : { x: 0, y: 0, width, height }); + }; + } + private async hostNewTab(url: string): Promise { await this.ensureRuntime(); - if (this.tabs.size >= MAX_BROWSER_TABS) { + if (this.tabs.size + this.engineTabs.size >= MAX_BROWSER_TABS) { throw new BrowserKernelError("RATE_LIMITED", `Browser tab limit is ${MAX_BROWSER_TABS}.`); } const normalized = this.policy.assertNavigationUrl(url); @@ -530,6 +742,8 @@ export class BrowserService { private async hostActivateTab(tabId: string): Promise { await this.ensureRuntime(); + // A contributed engine's tab is never shown: it becomes a Chromium tab first. + if (this.engineTabs.has(tabId)) await this.engineTabs.moveToChromium(tabId, "revealed", { waitForLoad: false }); const tab = this.requireTab(tabId); this.canvasPointers.cancelNavigationGesture(); this.invalidateCanvasSequence(false); @@ -553,6 +767,11 @@ export class BrowserService { private async hostCloseTab(tabId: string): Promise { await this.ensureRuntime(); + if (this.engineTabs.has(tabId)) { + this.engineTabs.close(tabId); + this.pendingDialogs.delete(tabId); + return this.getState(); + } const tab = this.requireTab(tabId); if (this.activeTabId === tabId) { this.invalidateCanvasSequence(false); @@ -569,6 +788,10 @@ export class BrowserService { private async hostNavigate(tabId: string, url: string): Promise { await this.ensureRuntime(); + if (this.engineTabs.has(tabId)) { + await this.engineTabs.navigate(tabId, this.policy.assertNavigationUrl(url)); + return this.getState(); + } const tab = this.requireTab(tabId); const normalized = this.policy.assertNavigationUrl(url); tab.lastSafeUrl = normalized; @@ -587,6 +810,10 @@ export class BrowserService { private async hostBack(tabId: string): Promise { await this.ensureRuntime(); + if (this.engineTabs.has(tabId)) { + await this.engineTabs.history(tabId, -1); + return this.getState(); + } const tab = this.requireTab(tabId); if (tab.view.webContents.navigationHistory.canGoBack()) { tab.loading = true; @@ -601,6 +828,10 @@ export class BrowserService { private async hostForward(tabId: string): Promise { await this.ensureRuntime(); + if (this.engineTabs.has(tabId)) { + await this.engineTabs.history(tabId, 1); + return this.getState(); + } const tab = this.requireTab(tabId); if (tab.view.webContents.navigationHistory.canGoForward()) { tab.loading = true; @@ -615,6 +846,16 @@ export class BrowserService { private async hostReload(tabId: string): Promise { await this.ensureRuntime(); + if (this.engineTabs.has(tabId)) { + await this.engineTabs.reload(tabId); + return this.getState(); + } + this.requireTab(tabId); + // A sleeping tab's wake is a load of its page already; a paused one resumes before it reloads. + if ((await this.lifecycle.ensureLive(tabId)).reloaded) { + this.emit(); + return this.getState(); + } let tab = this.requireTab(tabId); if (tab.view.webContents.isDestroyed()) { const url = tab.lastSafeUrl; @@ -642,6 +883,9 @@ export class BrowserService { const view = existingContents ? new WebContentsView({ webContents: existingContents }) : new WebContentsView({ webPreferences: remoteBrowserWebPreferences() }); + // Electron's `view.webContents` getter returns undefined once the WebContents is closed (a sleeping or destroyed + // tab), and every `tab.view.webContents.isDestroyed()` check would throw. Pin the object: closed, it still answers. + Object.defineProperty(view, "webContents", { value: view.webContents, configurable: true, enumerable: true }); const tab: BrowserTab = { id, view, @@ -653,9 +897,13 @@ export class BrowserService { lastSafeUrl: url, canvasCursor: new BrowserCanvasCursorController(view.webContents), canvasSinkViewport: new BrowserCanvasSinkViewportController(view.webContents), - visible: false + visible: false, + frozen: false, + mediaPlaying: false, + sleeping: null }; this.tabs.set(id, tab); + this.lifecycle.track(id, false); // A reused popup WebContents may already be running unthrottled; recompute from our own state. this.applyBackgroundThrottling(tab); tab.canvasCursor.set(browserCanvasNavigationCursor(this.canvasNavigationInput?.active ?? false, false)); @@ -774,6 +1022,8 @@ export class BrowserService { this.emit(); }); contents.on("did-finish-load", () => tab.canvasCursor.refresh()); + contents.on("media-started-playing", () => { tab.mediaPlaying = true; }); + contents.on("media-paused", () => { tab.mediaPlaying = false; }); contents.on("did-fail-load", (_event, errorCode, _errorDescription, _url, isMainFrame) => { if (!isMainFrame || errorCode === -3) return; tab.loading = false; @@ -828,7 +1078,8 @@ export class BrowserService { if (this.activeTabId === tab.id) { this.invalidateCanvasSequence(false); } - if (!this.tabs.has(tab.id)) return; + // A closed tab, or one put to sleep (or already woken into a new WebContents), did not crash. + if (this.tabs.get(tab.id) !== tab || tab.sleeping) return; tab.loading = false; tab.status = "crashed"; tab.crashState = "destroyed"; @@ -868,7 +1119,9 @@ export class BrowserService { event.preventDefault(); return; } - const tabId = [...this.tabs.values()].find((tab) => tab.view.webContents.id === contents.id)?.id ?? null; + const tabId = [...this.tabs.values()].find((tab) => ( + !tab.view.webContents.isDestroyed() && tab.view.webContents.id === contents.id + ))?.id ?? null; const download: BrowserDownloadSnapshot = { id, tabId, @@ -1033,6 +1286,9 @@ export class BrowserService { } private destroyTab(tab: BrowserTab): void { + this.lifecycle.untrack(tab.id); + this.lentTabs.delete(tab.id); + this.wakingTabs.delete(tab.id); this.canvasPointers.cancelTab(tab.id); if (this.activeTabId === tab.id) { this.invalidateCanvasSequence(false); @@ -1062,6 +1318,9 @@ export class BrowserService { const right = left + (visibleRectangle?.width ?? 0); const bottom = top + (visibleRectangle?.height ?? 0); const active = this.activeTabId ? this.tabs.get(this.activeTabId) : undefined; + if (active?.sleeping && this.visible && this.viewport.surface === "native" && visibleRectangle !== null) { + this.wakeForShow(active.id); + } if (!active || active.view.webContents.isDestroyed()) { this.hideClipView(); this.syncPresenceOverlay(null); @@ -1150,9 +1409,17 @@ export class BrowserService { /** Mirrors `view.setVisible` into `tab.visible` and recomputes background throttling for it. */ private setTabVisible(tab: BrowserTab, visible: boolean): void { - this.native.setVisible(tab.view, visible); + // A sleeping tab has no page to show; syncViews wakes it first. + if (tab.sleeping) { + tab.visible = false; + return; + } + // A tab lent a surface for a screenshot stays painted until the capture gives it back. + if (visible || !this.lentTabs.has(tab.id)) this.native.setVisible(tab.view, visible); + const changed = tab.visible !== visible; tab.visible = visible; this.applyBackgroundThrottling(tab); + if (changed && this.tabs.get(tab.id) === tab) this.lifecycle.setVisible(tab.id, visible); } private setTabAutomationBusy(tabId: string, busy: boolean): void { @@ -1160,6 +1427,126 @@ export class BrowserService { else this.busyAutomationTabs.delete(tabId); const tab = this.tabs.get(tabId); if (tab) this.applyBackgroundThrottling(tab); + this.lifecycle.setBusy(tabId, busy); + } + + private wakeForShow(tabId: string): void { + if (this.wakingTabs.has(tabId)) return; + this.wakingTabs.add(tabId); + void this.lifecycle.ensureLive(tabId).catch((error: unknown) => { + console.warn("CanvasTTY could not wake a sleeping browser tab.", error); + }).finally(() => { + this.wakingTabs.delete(tabId); + if (this.disposed) return; + this.syncViews(); + this.emit(); + }); + } + + /** Why a hidden tab must keep running now: anything the person or a page could lose by a pause. */ + private freezeBlocker(tabId: string): string | null { + const tab = this.tabs.get(tabId); + if (!tab || tab.sleeping || tab.view.webContents.isDestroyed()) return "closed"; + const contents = tab.view.webContents; + if (tab.status === "crashed") return "crashed"; + if (tab.loading) return "loading"; + if (tab.mediaPlaying || contents.isCurrentlyAudible()) return "media"; + if (this.pendingDialogs.has(tabId)) return "dialog"; + if (this.downloads.some((download) => download.tabId === tabId && download.completedAt === null)) return "download"; + if (contents.isBeingCaptured()) return "capture"; + if (contents.isDevToolsOpened()) return "devtools"; + return null; + } + + /** Sleeping also loses what the page holds in memory: never with an agent on the tab or a beforeunload handler. */ + private async discardBlocker(tabId: string): Promise { + const blocker = this.freezeBlocker(tabId); + if (blocker) return blocker; + if (this.agents.forTab(tabId).length > 0) return "agent"; + if (await this.automation.hasBeforeUnload(tabId)) return "beforeunload"; + return this.freezeBlocker(tabId); + } + + private async freezeTab(tabId: string): Promise { + const tab = this.tabs.get(tabId); + if (!tab || tab.sleeping || tab.view.webContents.isDestroyed()) return; + // Chromium keeps a page frozen only while it may be throttled; a hidden, undriven tab already is. + this.applyBackgroundThrottling(tab); + await this.automation.setLifecycleState(tabId, "frozen"); + } + + private async resumeTab(tabId: string): Promise { + const tab = this.tabs.get(tabId); + if (!tab || tab.sleeping || tab.view.webContents.isDestroyed()) return; + await this.automation.setLifecycleState(tabId, "active"); + } + + private lifecycleChanged(tabId: string, state: TabLifecycleState): void { + const tab = this.tabs.get(tabId); + if (tab) tab.frozen = state === "frozen"; + this.emit(); + } + + /** + * Puts a hidden tab to sleep: its WebContents is closed, and the tab keeps its address, title, favicon, history + * (with each entry's scroll and form state) and a small picture for the card. False when the tab woke meanwhile. + */ + private async sleepTab(tabId: string): Promise { + const tab = this.tabs.get(tabId); + if (!tab || tab.sleeping || tab.view.webContents.isDestroyed()) return false; + const contents = tab.view.webContents; + const preview = await boundedWait(capturePreview(contents), SLEEP_CAPTURE_TIMEOUT_MS).catch(() => null); + if (this.tabs.get(tabId) !== tab || tab.visible || this.busyAutomationTabs.has(tabId) || contents.isDestroyed()) { + return false; + } + const history = contents.navigationHistory; + tab.lastSafeUrl = this.tabUrl(tab); + tab.sleeping = { + title: contents.getTitle(), + canGoBack: history.canGoBack(), + canGoForward: history.canGoForward(), + entries: history.getAllEntries(), + index: history.getActiveIndex(), + preview + }; + this.canvasPointers.cancelTab(tab.id); + if (this.activeTabId === tab.id) this.invalidateCanvasSequence(false); + tab.canvasCursor.dispose(); + tab.canvasSinkViewport.dispose(); + this.automation.unregister(tab.id); + this.clipView.removeChildView(tab.view); + if (this.clipTabId === tab.id) this.clipTabId = null; + if (this.pointerTabId === tab.id) this.pointerTabId = null; + this.busyAutomationTabs.delete(tab.id); + this.pendingDialogs.delete(tab.id); + tab.frozen = false; + tab.loading = false; + tab.mediaPlaying = false; + contents.close({ waitForBeforeUnload: false }); + this.emit(); + return true; + } + + /** Brings a sleeping tab back in a new WebContents, on the history entry it slept on. */ + private async wakeTab(tabId: string): Promise { + const asleep = this.tabs.get(tabId); + const saved = asleep?.sleeping; + if (!asleep || !saved || this.disposed) return; + const tab = this.createRuntimeTab(tabId, asleep.lastSafeUrl, asleep.documentRevision + 1); + tab.favicon = asleep.favicon; + const current = saved.entries[saved.index]; + let loading: Promise; + if (current && isSafeBrowserUrl(current.url)) { + loading = tab.view.webContents.navigationHistory.restore({ entries: saved.entries, index: saved.index }) + .catch(() => this.loadTab(tab, asleep.lastSafeUrl)); + } else { + loading = this.loadTab(tab, asleep.lastSafeUrl); + } + this.syncViews(); + this.emit(); + // A command after the wake finds the page loaded (or as far as it got): a slow page cannot hold the queue. + await boundedWait(loading, WAKE_LOAD_TIMEOUT_MS).catch(() => undefined); + if (this.tabs.get(tabId) === tab) void this.persistRuntime(); } /** @@ -1315,6 +1702,26 @@ export class BrowserService { private tabSnapshot(tab: BrowserTab, agents: readonly AgentPresenceSnapshot[]): BrowserTabSnapshot { const contents = tab.view.webContents; const url = this.tabUrl(tab); + const tabAgents = agents + .filter((presence) => presence.currentTabId === tab.id) + .map((presence) => structuredClone(presence)); + if (tab.sleeping) { + return { + id: tab.id, + url, + title: tab.sleeping.title || displayUrl(url), + loading: false, + canGoBack: tab.sleeping.canGoBack, + canGoForward: tab.sleeping.canGoForward, + documentRevision: tab.documentRevision, + status: "ready", + favicon: tab.favicon, + agents: tabAgents, + crashState: null, + lifecycle: "sleeping", + ...(this.activeTabId === tab.id ? { preview: tab.sleeping.preview } : {}) + }; + } const title = contents.isDestroyed() ? "" : contents.getTitle(); return { id: tab.id, @@ -1326,8 +1733,9 @@ export class BrowserService { documentRevision: tab.documentRevision, status: tab.status, favicon: tab.favicon, - agents: agents.filter((presence) => presence.currentTabId === tab.id).map((presence) => structuredClone(presence)), - crashState: tab.crashState + agents: tabAgents, + crashState: tab.crashState, + ...(tab.frozen ? { lifecycle: "paused" as const } : {}) }; } @@ -1455,3 +1863,26 @@ async function readBoundedResponse(response: Response, maxBytes: number): Promis } return Buffer.concat(chunks, size); } + +/** A small JPEG of the page for a sleeping tab's card, or null when the page draws nothing. */ +async function capturePreview(contents: WebContents): Promise { + let image = await contents.capturePage(undefined, { stayHidden: true, stayAwake: true }); + if (image.isEmpty()) return null; + const size = image.getSize(); + if (size.width <= 0 || size.height <= 0) return null; + if (size.width > SLEEP_PREVIEW_MAX_WIDTH) { + image = image.resize({ width: SLEEP_PREVIEW_MAX_WIDTH, quality: "good" }); + } + return `data:image/jpeg;base64,${image.toJPEG(SLEEP_PREVIEW_JPEG_QUALITY).toString("base64")}`; +} + +function boundedWait(promise: Promise, ms: number): Promise { + let timer: NodeJS.Timeout | undefined; + return Promise.race([ + promise, + new Promise((_resolve, reject) => { + timer = setTimeout(() => reject(new Error("Timed out.")), ms); + timer.unref?.(); + }) + ]).finally(() => clearTimeout(timer)); +} diff --git a/src/main/services/PluginManager.ts b/src/main/services/PluginManager.ts index a68c5604..927b553a 100644 --- a/src/main/services/PluginManager.ts +++ b/src/main/services/PluginManager.ts @@ -35,6 +35,7 @@ import type { PluginServiceLaunch, PluginAgentTool, PluginCardAction, + PluginBrowserEngine, PluginCardActionFilter, PluginUpdateStatus, Size @@ -53,6 +54,7 @@ import type { DecisionService } from "./DecisionHooks.ts"; import { MAX_DECIDE_TIMEOUT_MS, MIN_DECIDE_TIMEOUT_MS } from "../../agent-runtime/runtime-protocol.mjs"; import type { AgentToolProvider } from "./PluginAgentTools.ts"; import type { CardActionProvider } from "./PluginCards.ts"; +import type { BrowserEngineProvider } from "./browser/BrowserEngineTabs.ts"; import { AGENT_PROVIDERS } from "../../shared/contracts.ts"; const MANIFEST_FILE = "canvastty.plugin.json"; @@ -144,7 +146,8 @@ const PLUGIN_PERMISSIONS = new Set([ "sessions:read-screen", "sessions:launch", "sessions:control", - "cards:decorate" + "cards:decorate", + "browser:engine" ]); interface StoredPluginRecord { @@ -643,6 +646,18 @@ export class PluginManager { })); } + /** Services whose browser engine may take agents' background tabs: enabled, native code trusted, `browser:engine`. */ + browserEngineProviders(): BrowserEngineProvider[] { + return this.trustedServicesWith("browser:engine", (service) => service.browserEngine).map(({ plugin, service, name }) => ({ + pluginId: plugin, + pluginName: name, + serviceId: service.id, + engineId: service.browserEngine!.id, + title: service.browserEngine!.title, + layout: service.browserEngine!.layout + })); + } + private trustedServicesWith( permission: PluginPermission, declares: (service: PluginService) => unknown @@ -1510,6 +1525,9 @@ export function validatePluginManifest(candidate: unknown): PluginManifest { if (service.cardActions && !permissions.includes("cards:decorate") && !granted?.includes("cards:decorate")) { throw new Error(`Plugin service ${service.id} adds card actions and needs the cards:decorate permission.`); } + if (service.browserEngine && !permissions.includes("browser:engine") && !granted?.includes("browser:engine")) { + throw new Error(`Plugin service ${service.id} contributes a browser engine and needs the browser:engine permission.`); + } } const coreFiles = candidate.coreFiles === undefined ? [] : validateModuleFiles(candidate.coreFiles, "coreFiles"); if (modules.length > 0 && coreFiles.length === 0) { @@ -1641,7 +1659,7 @@ function validateServices(value: unknown, moduleIds: ReadonlySet): Plugi const services = value.map((candidate): PluginService => { if (!isRecord(candidate)) throw new Error("Every plugin service must be an object."); assertOnlyKeys(candidate, [ - "id", "title", "description", "entry", "module", "launch", "environments", "decide", "tools", "cardActions" + "id", "title", "description", "entry", "module", "launch", "environments", "decide", "tools", "cardActions", "browserEngine" ], "Plugin service"); const id = requiredString(candidate.id, "service id", 64); if (!isContributionId(id) || ids.has(id)) throw new Error(`Plugin service id is invalid or duplicated: ${id}.`); @@ -1661,13 +1679,15 @@ function validateServices(value: unknown, moduleIds: ReadonlySet): Plugi const decide = candidate.decide === undefined ? undefined : validateServiceDecide(candidate.decide); const tools = candidate.tools === undefined ? undefined : validateServiceTools(candidate.tools); const cardActions = candidate.cardActions === undefined ? undefined : validateCardActions(candidate.cardActions); + const browserEngine = candidate.browserEngine === undefined ? undefined : validateBrowserEngine(candidate.browserEngine); return { id, title, ...(description ? { description } : {}), entry, ...(module ? { module } : {}), ...(launch ? { launch } : {}), ...(environments ? { environments } : {}), ...(decide ? { decide } : {}), ...(tools ? { tools } : {}), - ...(cardActions ? { cardActions } : {}) + ...(cardActions ? { cardActions } : {}), + ...(browserEngine ? { browserEngine } : {}) }; }); // Agents see `__` and cards `` + action id, so both are unique per plugin. @@ -1675,6 +1695,9 @@ function validateServices(value: unknown, moduleIds: ReadonlySet): Plugi if (new Set(toolNames).size !== toolNames.length) throw new Error("Plugin agent tool names must be unique."); const actionIds = services.flatMap((service) => service.cardActions ?? []).map((action) => action.id); if (new Set(actionIds).size !== actionIds.length) throw new Error("Plugin card action ids must be unique."); + // Agents name an engine by its id, so a plugin declares each id once. + const engineIds = services.flatMap((service) => service.browserEngine ? [service.browserEngine.id] : []); + if (new Set(engineIds).size !== engineIds.length) throw new Error("Plugin browser engine ids must be unique."); // "Allow decisions" is confirmed per plugin, so one service per plugin answers. if (services.filter((service) => service.decide).length > 1) { throw new Error("At most one plugin service may decide on tool calls."); @@ -1762,6 +1785,22 @@ function validateServiceTools(value: unknown): PluginAgentTool[] { }); } +const BROWSER_ENGINE_ID = /^[a-z0-9](?:[a-z0-9._-]{0,62}[a-z0-9])?$/; +const RESERVED_BROWSER_ENGINE_IDS = new Set(["auto", "chromium"]); + +function validateBrowserEngine(value: unknown): PluginBrowserEngine { + if (!isRecord(value)) throw new Error("Plugin service browserEngine must be an object."); + assertOnlyKeys(value, ["id", "title", "description", "layout"], "Plugin browser engine"); + const id = requiredString(value.id, "browser engine id", 64); + if (!BROWSER_ENGINE_ID.test(id) || RESERVED_BROWSER_ENGINE_IDS.has(id)) { + throw new Error(`Plugin browser engine id is invalid or reserved: ${id}.`); + } + const title = requiredString(value.title, "browser engine title", 80); + const description = optionalString(value.description, "browser engine description", 240); + if (value.layout !== undefined && typeof value.layout !== "boolean") throw new Error("Plugin browser engine layout must be true or false."); + return { id, title, ...(description ? { description } : {}), layout: value.layout === true }; +} + const MAX_CARD_ACTIONS = 8; function validateCardActions(value: unknown): PluginCardAction[] { diff --git a/src/main/services/SettingsStore.ts b/src/main/services/SettingsStore.ts index 5ba49188..90c21cf0 100644 --- a/src/main/services/SettingsStore.ts +++ b/src/main/services/SettingsStore.ts @@ -407,6 +407,7 @@ function createDefaults(systemLocale: string, platform: string): AppSettings { browserAgentAccess: true, browserShowAgentPresence: true, browserRestoreTabs: true, + browserPauseHiddenTabs: true, attentionNotifications: true, attentionQueueVisible: true, attentionQueuePlacement: "bottom-right", @@ -676,6 +677,9 @@ export function normalizeSettings( browserRestoreTabs: typeof source.browserRestoreTabs === "boolean" ? source.browserRestoreTabs : fallback.browserRestoreTabs, + browserPauseHiddenTabs: typeof source.browserPauseHiddenTabs === "boolean" + ? source.browserPauseHiddenTabs + : fallback.browserPauseHiddenTabs, attentionNotifications: typeof source.attentionNotifications === "boolean" ? source.attentionNotifications : fallback.attentionNotifications, diff --git a/src/main/services/agent-browser/AgentGateway.ts b/src/main/services/agent-browser/AgentGateway.ts index 9ecc0a5a..2f08d799 100644 --- a/src/main/services/agent-browser/AgentGateway.ts +++ b/src/main/services/agent-browser/AgentGateway.ts @@ -43,6 +43,8 @@ import { const DEFAULT_CAPABILITY_TTL_MS = 60_000; const MAX_TRANSPORT_RESTART_ATTEMPTS = 3; const TRANSPORT_RESTART_BASE_DELAY_MS = 500; +/** A host that dies within this long of starting counts toward MAX_TRANSPORT_RESTART_ATTEMPTS; a longer run resets it. */ +const TRANSPORT_FAST_FAILURE_WINDOW_MS = 60_000; export const WINDOWS_AGENT_GATEWAY_UNAVAILABLE = "Agent browser access on Windows requires the packaged current-user-only named-pipe host."; @@ -112,6 +114,9 @@ export class AgentGateway { /** Bumped by close(): a Unix bring-up still creating or opening its socket then knows it was closed. */ private closeGeneration = 0; private restartTimer: NodeJS.Timeout | undefined; + /** The pipe name the first host published; a replacement listens on it again so helpers can reconnect. */ + private windowsPipeName: string | null = null; + private transportStartedAt: number | null = null; private restartAttempts = 0; private restartToken = 0; private recovering = false; @@ -166,10 +171,12 @@ export class AgentGateway { const settle = () => { if (this.startPromise === starting) this.startPromise = null; }; + const recovering = this.recovering; void starting.then(() => { settle(); this.recovering = false; - this.restartAttempts = 0; + // A replacement host counts as recovered only once it outlives the fast-failure window (see the fatal handler). + if (!recovering) this.restartAttempts = 0; }, settle); return starting; } @@ -179,7 +186,8 @@ export class AgentGateway { const transport = this.windowsPipeHostFactory({ hostPath: this.windowsHostPath, platform: this.platform, - parentPid: process.pid + parentPid: process.pid, + ...(this.windowsPipeName ? { pipeName: this.windowsPipeName } : {}) }); this.windowsTransport = transport; transport.on("fatal", () => this.handleTransportFatal(transport)); @@ -192,6 +200,8 @@ export class AgentGateway { throw new Error("Windows agent pipe host was superseded during startup."); } this.endpoint = endpoint; + this.windowsPipeName = endpoint; + this.transportStartedAt = this.now(); this.expiryTimer = setInterval(() => this.expireConnections(), 1_000); this.expiryTimer.unref(); return endpoint; @@ -335,6 +345,8 @@ export class AgentGateway { this.server = null; this.windowsTransport = null; this.endpoint = null; + this.windowsPipeName = null; + this.transportStartedAt = null; this.ownedRuntimeDirectory = null; if (server) await closeServer(server); if (windowsTransport) await windowsTransport.close(); @@ -344,7 +356,10 @@ export class AgentGateway { private handleTransportFatal(transport: WindowsPipeHostTransport): void { // A transport that was already replaced must not disturb its successor. if (this.windowsTransport !== transport) return; + const ranFor = this.transportStartedAt === null ? 0 : this.now() - this.transportStartedAt; + if (ranFor >= TRANSPORT_FAST_FAILURE_WINDOW_MS) this.restartAttempts = 0; this.windowsTransport = null; + this.transportStartedAt = null; this.endpoint = null; clearInterval(this.expiryTimer); this.expiryTimer = undefined; @@ -387,7 +402,6 @@ export class AgentGateway { return; } this.recovering = false; - this.restartAttempts = 0; } catch { if (token === this.restartToken && this.enabled) this.scheduleTransportRestart(); } diff --git a/src/main/services/agent-browser/OrchestrationGateway.ts b/src/main/services/agent-browser/OrchestrationGateway.ts index 407cd1c3..5a8d6094 100644 --- a/src/main/services/agent-browser/OrchestrationGateway.ts +++ b/src/main/services/agent-browser/OrchestrationGateway.ts @@ -41,6 +41,8 @@ import { const CAPABILITY_TTL_MS = 60_000; const MAX_TRANSPORT_RESTART_ATTEMPTS = 3; const TRANSPORT_RESTART_BASE_DELAY_MS = 500; +/** A replacement host that dies within this long of starting still counts toward MAX_TRANSPORT_RESTART_ATTEMPTS. */ +const TRANSPORT_FAST_FAILURE_WINDOW_MS = 60_000; interface CapabilityLease { connectionId: string; @@ -100,6 +102,8 @@ export class OrchestrationGateway { private generation = 0; private restartTimer: NodeJS.Timeout | null = null; private restartAttempts = 0; + /** When the current Windows pipe host started listening. */ + private transportStartedAt: number | null = null; private restartToken = 0; private recovering = false; /** The start caller (or recovery attempt) that currently owns a shared in-flight start. */ @@ -204,6 +208,7 @@ export class OrchestrationGateway { throw new Error("The Windows orchestration pipe host failed during startup."); } this.socketEndpoint = endpoint; + this.transportStartedAt = this.now(); } else { // Unix domain sockets cap at ~104 path bytes (macOS); fall back to a short // current-user directory exactly like the browser gateway does. @@ -253,12 +258,18 @@ export class OrchestrationGateway { this.socketEndpoint = null; this.ownedRuntimeDirectory = null; this.running = false; + this.transportStartedAt = null; } private handleTransportFatal(transport: WindowsPipeHostTransport, generation: number): void { // A late event from an old host, or one superseded by explicit stop(), cannot affect a successor. if (this.windowsTransport !== transport || generation !== this.generation) return; const wasRunning = this.running; + // A host that served for a while earns a fresh retry budget; one that dies right after starting does not, so a + // host that keeps crashing on start-up is not restarted forever. + const ranFor = this.transportStartedAt === null ? 0 : this.now() - this.transportStartedAt; + if (ranFor >= TRANSPORT_FAST_FAILURE_WINDOW_MS) this.restartAttempts = 0; + this.transportStartedAt = null; this.windowsTransport = null; this.socketEndpoint = null; this.running = false; @@ -317,8 +328,8 @@ export class OrchestrationGateway { if (this.recovering && this.enabled) this.scheduleTransportRestart(); return; } + // The retry budget is kept until the replacement outlives TRANSPORT_FAST_FAILURE_WINDOW_MS (see the fatal handler). this.recovering = false; - this.restartAttempts = 0; } catch { if (intent === this.startIntent && token === this.restartToken && this.enabled) this.scheduleTransportRestart(); } diff --git a/src/main/services/agent-browser/WindowsPipeHostTransport.ts b/src/main/services/agent-browser/WindowsPipeHostTransport.ts index 50ba164b..e2ddec30 100644 --- a/src/main/services/agent-browser/WindowsPipeHostTransport.ts +++ b/src/main/services/agent-browser/WindowsPipeHostTransport.ts @@ -43,6 +43,11 @@ export interface WindowsPipeHostTransportOptions { startupTimeoutMs?: number; platform?: NodeJS.Platform; spawnHost?: typeof spawn; + /** + * The endpoint a previous host of the same gateway published. A restarted host listens on it again, so clients + * that already hold the address (and their reconnect tokens) can come back; omitted, the host picks a fresh name. + */ + pipeName?: string; } interface RelayFrame { @@ -60,7 +65,7 @@ interface RelayFrame { */ export class WindowsPipeHostTransport extends EventEmitter { private readonly options: Required> - & Pick; + & Pick; private readonly sockets = new Map(); private decoder = new RelayFrameDecoder(); private child: ChildProcessWithoutNullStreams | null = null; @@ -77,8 +82,12 @@ export class WindowsPipeHostTransport extends EventEmitter { parentPid: options.parentPid ?? process.pid, startupTimeoutMs: options.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS, platform: options.platform ?? process.platform, - spawnHost: options.spawnHost + spawnHost: options.spawnHost, + pipeName: options.pipeName }; + if (options.pipeName !== undefined && !isGeneratedPipeName(options.pipeName)) { + throw new Error("Windows agent pipe host name is not one the host generates."); + } } get address(): string { @@ -111,7 +120,10 @@ export class WindowsPipeHostTransport extends EventEmitter { const spawnHost = this.options.spawnHost ?? spawn; const child = spawnHost( this.options.hostPath, - ["--parent-pid", String(this.options.parentPid)], + [ + "--parent-pid", String(this.options.parentPid), + ...(this.options.pipeName ? ["--pipe-name", this.options.pipeName] : []) + ], { stdio: ["pipe", "pipe", "pipe"], windowsHide: true, @@ -150,6 +162,9 @@ export class WindowsPipeHostTransport extends EventEmitter { for (const frame of this.decoder.push(chunk)) { if (frame.type === HOST_TO_PARENT.ready && !settled) { const endpoint = parseReadyEndpoint(frame); + if (this.options.pipeName && endpoint !== this.options.pipeName) { + throw new Error("Windows pipe host listened on a different endpoint than requested."); + } this.endpoint = endpoint; this.started = true; settled = true; @@ -423,6 +438,11 @@ function parseReadyEndpoint(frame: RelayFrame): string { return endpoint; } +/** The names the native host generates: the fixed prefix and 16 random bytes in lowercase hex. */ +function isGeneratedPipeName(value: string): boolean { + return /^\\\\\.\\pipe\\canvastty-agent-[0-9a-f]{32}$/u.test(value); +} + function safeHostMessage(payload: Buffer): string { return payload.toString("utf8").replace(/[\u0000-\u001f\u007f]/g, " ").slice(0, 1_024) || "Windows agent pipe host reported a fatal error."; diff --git a/src/main/services/browser/BrowserAutomationService.ts b/src/main/services/browser/BrowserAutomationService.ts index d6b721e8..8608ee9a 100644 --- a/src/main/services/browser/BrowserAutomationService.ts +++ b/src/main/services/browser/BrowserAutomationService.ts @@ -9,7 +9,8 @@ import type { BrowserObservation, BrowserObservedElement } from "../../../shared/contracts.ts"; -import { BrowserKernelError, throwIfAborted } from "./BrowserErrors.ts"; +import { BrowserKernelError, EngineFallbackRequired, throwIfAborted } from "./BrowserErrors.ts"; +import { CHROMIUM_ENGINE, ElectronTabDriver, type TabDriver } from "./TabDriver.ts"; import { SENSITIVE_FIELD_SOURCE, isSensitiveFieldIdentity } from "../safety/sensitiveNames.ts"; const MAX_OBSERVE_ELEMENTS = 200; @@ -28,6 +29,8 @@ const WAIT_POLL_MAX_MS = 1_000; const AUTOMATION_BUSY_GRACE_MS = 500; export const BROWSER_SCREENSHOT_MAX_BINARY_BYTES = 340 * 1024; const CDP_VERSION = "1.3"; +/** How long a lifecycle probe (beforeunload listeners, scroll position) may take before the answer is "unknown". */ +const LIFECYCLE_PROBE_TIMEOUT_MS = 2_000; const PRESENCE_WORLD = "canvastty-agent-presence"; const INTERACTIVE_ROLES = new Set([ "button", "checkbox", "combobox", "link", "listbox", @@ -87,12 +90,11 @@ interface RefEntry { interface TabSession { tabId: string; - contents: WebContents; + driver: TabDriver; revision: number; refs: Map; attachPromise: Promise | null; - messageListener: (_event: unknown, method: string, params: unknown) => void; - detachListener: (_event: unknown, reason: string) => void; + unlisten: () => void; onDialog?: (dialog: BrowserDialogSnapshot | null) => void; presences: AgentPresenceSnapshot[]; presenceContextId: number | null; @@ -105,6 +107,9 @@ interface TabSession { dialogBlockedCommands: Set>; inflightRequests: Set; networkLastChangeAt: number; + /** Contributed engines only: the main document's HTTP status and whether a bot challenge was requested. */ + documentStatus: number | null; + challengeSeen: boolean; } interface ElectronDialogInfo { @@ -147,13 +152,77 @@ export interface BrowserPointerResult { y: number; } +export interface BrowserAutomationOptions { + /** + * Runs before every automation command reaches the page, after the tab is marked busy: BrowserService resumes a + * paused (frozen) tab here, so no CDP command waits on a page whose tasks are stopped. + */ + beforeCommand?(tabId: string): Promise; + /** + * Before a screenshot of a background tab nobody sees (an agent's tab, or one that moved from a contributed engine): + * lets the host give it a surface to paint on for the capture, outside what the person sees. Returns what undoes + * that, or null when the host did nothing. + */ + captureSurface?(tabId: string): Promise<(() => void) | null>; +} + +/** A surface lent for one capture paints after a moment: this many tries, this far apart. */ +const LENT_SURFACE_CAPTURE_TRIES = 8; +const LENT_SURFACE_CAPTURE_DELAY_MS = 120; + export class BrowserAutomationService { private readonly sessions = new Map(); private readonly busyTimers = new Map(); private readonly onBusyChange?: (tabId: string, busy: boolean) => void; + private readonly beforeCommand?: (tabId: string) => Promise; + private readonly captureSurface?: (tabId: string) => Promise<(() => void) | null>; - constructor(onBusyChange?: (tabId: string, busy: boolean) => void) { + constructor(onBusyChange?: (tabId: string, busy: boolean) => void, options: BrowserAutomationOptions = {}) { this.onBusyChange = onBusyChange; + this.beforeCommand = options.beforeCommand; + this.captureSurface = options.captureSurface; + } + + /** + * Freezes or resumes the page through the tab's own debugger attachment (CDP Page.setWebLifecycleState). This is + * not an automation command: it does not mark the tab busy. Chromium only keeps a page frozen while background + * throttling is allowed for it, which BrowserService guarantees for the hidden, undriven tabs it freezes. + */ + async setLifecycleState(tabId: string, state: "frozen" | "active"): Promise { + const session = this.sessions.get(tabId); + if (!session || session.driver.isDestroyed()) return; + await this.attach(session); + await session.driver.send("Page.setWebLifecycleState", { state }); + } + + /** + * Whether the top document has a beforeunload handler (listener or `onbeforeunload`): the page may hold input the + * person has not saved. Any failure or a slow answer counts as yes, so an unknown page is never discarded. + */ + async hasBeforeUnload(tabId: string): Promise { + const session = this.sessions.get(tabId); + if (!session || session.driver.isDestroyed()) return true; + const group = "canvastty-lifecycle"; + const probe = async (): Promise => { + await this.attach(session); + const driver = session.driver; + try { + const windowObject = await driver.send("Runtime.evaluate", { + expression: "window", + objectGroup: group, + silent: true + }) as { result?: { objectId?: string } }; + const objectId = windowObject.result?.objectId; + if (!objectId) return true; + const listeners = await driver.send("DOMDebugger.getEventListeners", { objectId }) as { + listeners?: Array<{ type?: string }>; + }; + return (listeners.listeners ?? []).some((listener) => listener.type === "beforeunload"); + } finally { + await driver.send("Runtime.releaseObjectGroup", { objectGroup: group }).catch(() => undefined); + } + }; + return await withTimeout(probe(), LIFECYCLE_PROBE_TIMEOUT_MS).catch(() => true); } /** @@ -180,6 +249,7 @@ export class BrowserAutomationService { this.onBusyChange?.(tabId, false); } + /** Registers a Browser card tab: its Electron WebContents is the driver. */ async register( tabId: string, contents: WebContents, @@ -187,32 +257,34 @@ export class BrowserAutomationService { onDialog?: (dialog: BrowserDialogSnapshot | null) => void ): Promise { const current = this.sessions.get(tabId); - if (current?.contents === contents) { + if (current?.driver instanceof ElectronTabDriver && current.driver.contents === contents) { current.revision = revision; current.onDialog = onDialog; await this.attach(current); return; } - if (current) this.unregister(tabId); + await this.registerDriver(tabId, new ElectronTabDriver(contents), revision, onDialog); + } + + /** + * Registers a tab behind any driver: a contributed engine's tab (CdpTabDriver) comes here directly. Such a tab has + * no JavaScript dialogs to answer, no lifecycle states and no presence overlay: nobody sees it. + */ + async registerDriver( + tabId: string, + driver: TabDriver, + revision: number, + onDialog?: (dialog: BrowserDialogSnapshot | null) => void + ): Promise { + if (this.sessions.has(tabId)) this.unregister(tabId); const session: TabSession = { tabId, - contents, + driver, revision, refs: new Map(), attachPromise: null, - messageListener: (_event, method, params) => this.onMessage(tabId, method, params), - detachListener: () => { - const live = this.sessions.get(tabId); - if (live) { - live.attachPromise = null; - live.presenceContextId = null; - live.presenceLastPayload = null; - live.refs.clear(); - live.inflightRequests.clear(); - live.networkLastChangeAt = Date.now(); - } - }, + unlisten: () => undefined, onDialog, presences: [], presenceContextId: null, @@ -224,36 +296,58 @@ export class BrowserAutomationService { dialogOpenedWaiters: new Set(), dialogBlockedCommands: new Set(), inflightRequests: new Set(), - networkLastChangeAt: Date.now() + networkLastChangeAt: Date.now(), + documentStatus: null, + challengeSeen: false }; this.sessions.set(tabId, session); - // Electron consumes JavaScript dialogs in its private WebContents handler before - // CDP can emit Page.javascriptDialogOpening. Replace that handler for this - // dedicated remote WebContents so alert/confirm/prompt remain pending until the - // trusted browser chrome or an authenticated agent answers them. - const dialogEvents = contents as unknown as EventEmitter; - dialogEvents.removeAllListeners(ELECTRON_RUN_DIALOG_EVENT); - dialogEvents.on(ELECTRON_RUN_DIALOG_EVENT, session.electronDialogListener); - dialogEvents.prependListener(ELECTRON_CANCEL_DIALOGS_EVENT, session.electronCancelDialogsListener); - contents.debugger.on("message", session.messageListener); - contents.debugger.on("detach", session.detachListener); + if (driver instanceof ElectronTabDriver) { + // Electron consumes JavaScript dialogs in its private WebContents handler before + // CDP can emit Page.javascriptDialogOpening. Replace that handler for this + // dedicated remote WebContents so alert/confirm/prompt remain pending until the + // trusted browser chrome or an authenticated agent answers them. + const dialogEvents = driver.contents as unknown as EventEmitter; + dialogEvents.removeAllListeners(ELECTRON_RUN_DIALOG_EVENT); + dialogEvents.on(ELECTRON_RUN_DIALOG_EVENT, session.electronDialogListener); + dialogEvents.prependListener(ELECTRON_CANCEL_DIALOGS_EVENT, session.electronCancelDialogsListener); + } + session.unlisten = driver.listen({ + message: (method, params) => this.onMessage(tabId, method, params), + detach: () => { + const live = this.sessions.get(tabId); + if (live) { + live.attachPromise = null; + live.presenceContextId = null; + live.presenceLastPayload = null; + live.refs.clear(); + live.inflightRequests.clear(); + live.networkLastChangeAt = Date.now(); + } + } + }); await this.attach(session); } + /** The engine driving a registered tab (`chromium` for Browser card tabs), or null. */ + engineOf(tabId: string): string | null { + return this.sessions.get(tabId)?.driver.engine ?? null; + } + unregister(tabId: string): void { const session = this.sessions.get(tabId); if (!session) return; this.sessions.delete(tabId); this.clearBusy(tabId); this.cancelElectronDialog(tabId, session); - const dialogEvents = session.contents as unknown as EventEmitter; - dialogEvents.removeListener(ELECTRON_RUN_DIALOG_EVENT, session.electronDialogListener); - dialogEvents.removeListener(ELECTRON_CANCEL_DIALOGS_EVENT, session.electronCancelDialogsListener); - session.contents.debugger.removeListener("message", session.messageListener); - session.contents.debugger.removeListener("detach", session.detachListener); - if (!session.contents.isDestroyed() && session.contents.debugger.isAttached()) { + if (session.driver instanceof ElectronTabDriver) { + const dialogEvents = session.driver.contents as unknown as EventEmitter; + dialogEvents.removeListener(ELECTRON_RUN_DIALOG_EVENT, session.electronDialogListener); + dialogEvents.removeListener(ELECTRON_CANCEL_DIALOGS_EVENT, session.electronCancelDialogsListener); + } + session.unlisten(); + if (!session.driver.isDestroyed() && session.driver.isAttached()) { try { - session.contents.debugger.detach(); + session.driver.detach(); } catch { // Closing a tab can race with Chromium detaching the debugger. } @@ -276,30 +370,41 @@ export class BrowserAutomationService { ): Promise { const session = await this.ready(tabId, revision); throwIfAborted(options.signal); - const nodes = (await this.fullAxTree(session, options.signal)).filter((node) => { + const tree = await this.fullAxTree(session, options.signal); + if (isContributed(session) && options.cursor === undefined) await this.assertEngineContent(session, tree, false); + const nodes = tree.filter((node) => { const role = axString(node.role); return !node.ignored && node.backendDOMNodeId && INTERACTIVE_ROLES.has(role); }).slice(0, 1_000); - const metrics = await this.command<{ cssLayoutViewport?: { clientWidth?: number; clientHeight?: number } }>( - session, - "Page.getLayoutMetrics" - ); - const viewportWidth = metrics.cssLayoutViewport?.clientWidth ?? Number.MAX_SAFE_INTEGER; - const viewportHeight = metrics.cssLayoutViewport?.clientHeight ?? Number.MAX_SAFE_INTEGER; - if (viewportWidth <= 0 || viewportHeight <= 0) throw viewportUnavailable(); const offset = decodeCursor(options.cursor, revision); const limit = clampInteger(options.limit, 1, MAX_OBSERVE_ELEMENTS, 80); // One box-model round trip per element: measured only up to this page and one more (is there a next page?). const wanted = offset + limit + 1; - const visible: Array<{ node: CdpAxNode; bounds: BrowserElementBounds }> = []; - for (const node of nodes) { - if (visible.length >= wanted) break; - throwIfAborted(options.signal); - const bounds = await this.box(session, node.backendDOMNodeId!); - if (!bounds || bounds.width <= 0 || bounds.height <= 0) continue; - if (bounds.x + bounds.width <= 0 || bounds.y + bounds.height <= 0 - || bounds.x >= viewportWidth || bounds.y >= viewportHeight) continue; - visible.push({ node, bounds }); + const visible: Array<{ node: CdpAxNode; bounds: BrowserElementBounds | null }> = []; + if (!session.driver.layout) { + // An engine without real layout has no meaningful boxes or viewport: every interactive node counts, in + // document order, and actions on them go through the DOM instead of coordinates. + for (const node of nodes) { + if (visible.length >= wanted) break; + visible.push({ node, bounds: null }); + } + } else { + const metrics = await this.command<{ cssLayoutViewport?: { clientWidth?: number; clientHeight?: number } }>( + session, + "Page.getLayoutMetrics" + ); + const viewportWidth = metrics.cssLayoutViewport?.clientWidth ?? Number.MAX_SAFE_INTEGER; + const viewportHeight = metrics.cssLayoutViewport?.clientHeight ?? Number.MAX_SAFE_INTEGER; + if (viewportWidth <= 0 || viewportHeight <= 0) throw viewportUnavailable(); + for (const node of nodes) { + if (visible.length >= wanted) break; + throwIfAborted(options.signal); + const bounds = await this.box(session, node.backendDOMNodeId!); + if (!bounds || bounds.width <= 0 || bounds.height <= 0) continue; + if (bounds.x + bounds.width <= 0 || bounds.y + bounds.height <= 0 + || bounds.x >= viewportWidth || bounds.y >= viewportHeight) continue; + visible.push({ node, bounds }); + } } const page = visible.slice(offset, offset + limit); const elements: BrowserObservedElement[] = []; @@ -336,8 +441,8 @@ export class BrowserAutomationService { return { untrustedWebContent: true, tabId, - url: session.contents.getURL(), - title: session.contents.getTitle(), + url: session.driver.url(), + title: session.driver.title(), documentRevision: revision, elements, nextCursor: nextOffset < visible.length ? encodeCursor(revision, nextOffset) : null @@ -351,7 +456,9 @@ export class BrowserAutomationService { ): Promise { const session = await this.ready(tabId, revision); throwIfAborted(options.signal); - const readable = (await this.fullAxTree(session, options.signal)).filter((node) => ( + const tree = await this.fullAxTree(session, options.signal); + if (isContributed(session) && options.cursor === undefined) await this.assertEngineContent(session, tree, true); + const readable = tree.filter((node) => ( !node.ignored && READABLE_ROLES.has(axString(node.role)) )); const offset = decodeCursor(options.cursor, revision); @@ -393,8 +500,8 @@ export class BrowserAutomationService { return { untrustedWebContent: true, tabId, - url: session.contents.getURL(), - title: session.contents.getTitle(), + url: session.driver.url(), + title: session.driver.title(), documentRevision: revision, text: parts.join("\n"), links, @@ -408,10 +515,14 @@ export class BrowserAutomationService { async screenshot(tabId: string, revision: number, signal?: AbortSignal): Promise { const session = await this.ready(tabId, revision); throwIfAborted(signal); + // A contributed engine's picture is not what the site looks like (Lightpanda draws no CSS): Chromium takes it. + if (!(session.driver instanceof ElectronTabDriver)) throw new EngineFallbackRequired("screenshot"); + const contents = session.driver.contents; const sensitiveBefore = await this.sensitiveBoundsForScreenshot(session); const masks = await this.maskSensitiveInputs(session); + const lent = this.captureSurface ? await this.captureSurface(tabId).catch(() => null) : null; try { - let image = await session.contents.capturePage(); + let image = lent ? await captureLentSurface(contents, signal) : await contents.capturePage(); throwIfAborted(signal); const capturedSize = image.getSize(); if (image.isEmpty() || capturedSize.width <= 0 || capturedSize.height <= 0) throw viewportUnavailable(); @@ -449,6 +560,7 @@ export class BrowserAutomationService { height: size.height }; } finally { + lent?.(); await this.restoreSensitiveInputs(session, masks).catch(() => undefined); } } @@ -488,7 +600,7 @@ export class BrowserAutomationService { backendNodeId: number ): Promise { try { - const response = await session.contents.debugger.sendCommand("DOM.getBoxModel", { + const response = await session.driver.send("DOM.getBoxModel", { backendNodeId }) as { model?: { border?: number[]; content?: number[] } }; const quad = response.model?.border ?? response.model?.content; @@ -566,8 +678,12 @@ export class BrowserAutomationService { ref: BrowserElementRef | string | undefined, signal?: AbortSignal ): Promise { - const { session, point } = await this.refPoint(tabId, revision, ref); + const { session, entry, point } = await this.refPoint(tabId, revision, ref); throwIfAborted(signal); + if (!session.driver.layout) { + await this.domCall(session, entry, DOM_CLICK_FUNCTION); + return point; + } const pressed = await this.commandAllowDialog(session, "Input.dispatchMouseEvent", { type: "mousePressed", ...point, button: "left", clickCount: 1 }); @@ -584,8 +700,12 @@ export class BrowserAutomationService { ref: BrowserElementRef | string | undefined, signal?: AbortSignal ): Promise { - const { session, point } = await this.refPoint(tabId, revision, ref); + const { session, entry, point } = await this.refPoint(tabId, revision, ref); throwIfAborted(signal); + if (!session.driver.layout) { + await this.domCall(session, entry, DOM_HOVER_FUNCTION); + return point; + } await this.commandAllowDialog(session, "Input.dispatchMouseEvent", { type: "mouseMoved", ...point }); return point; } @@ -694,13 +814,27 @@ export class BrowserAutomationService { ): Promise { const session = await this.ready(tabId, revision); throwIfAborted(signal); - const point = ref === undefined - ? await this.viewportCenter(session) - : (await this.refPoint(tabId, revision, ref)).point; const fallback = direction === "up" ? { x: 0, y: -600 } : direction === "left" ? { x: -600, y: 0 } : direction === "right" ? { x: 600, y: 0 } : { x: 0, y: 600 }; + if (!session.driver.layout) { + const x = clampNumber(deltaX, -5_000, 5_000, fallback.x); + const y = clampNumber(deltaY, -5_000, 5_000, fallback.y); + if (ref === undefined) { + await this.commandAllowDialog(session, "Runtime.evaluate", { + expression: `window.scrollBy(${x},${y})`, + silent: true + }); + } else { + const { entry } = await this.refPoint(tabId, revision, ref); + await this.domCall(session, entry, DOM_SCROLL_FUNCTION, [{ value: x }, { value: y }]); + } + return { x: 0, y: 0 }; + } + const point = ref === undefined + ? await this.viewportCenter(session) + : (await this.refPoint(tabId, revision, ref)).point; await this.commandAllowDialog(session, "Input.dispatchMouseEvent", { type: "mouseWheel", ...point, @@ -755,6 +889,8 @@ export class BrowserAutomationService { const source = await this.refPoint(tabId, revision, ref); const target = await this.refPoint(tabId, revision, targetRef); throwIfAborted(signal); + // Dragging is pointer geometry through and through: an engine without layout cannot do it. + if (!source.session.driver.layout) throw new EngineFallbackRequired("unsupported-action"); const positioned = await this.commandAllowDialog(source.session, "Input.dispatchMouseEvent", { type: "mouseMoved", ...source.point }); @@ -836,7 +972,7 @@ export class BrowserAutomationService { while (Date.now() - startedAt < timeoutMs) { throwIfAborted(signal); const session = await this.ready(tabId, revision); - if (condition === "load" && !session.contents.isLoading()) return { matched: true }; + if (condition === "load" && !session.driver.isLoading()) return { matched: true }; if (condition === "network-idle") { const networkIdleSince = session.inflightRequests.size === 0 ? session.networkLastChangeAt @@ -844,14 +980,14 @@ export class BrowserAutomationService { // Keep isLoading as an extra document-readiness guard, but network // idleness is defined by CDP request lifecycle events rather than this // coarse WebContents flag. - if (networkIdleSince !== null && !session.contents.isLoading()) { + if (networkIdleSince !== null && !session.driver.isLoading()) { idleSince = Math.max(idleSince ?? 0, networkIdleSince); } else { idleSince = null; } if (idleSince !== null && Date.now() - idleSince >= 500) return { matched: true }; } - if (condition === "url" && value && session.contents.getURL().includes(value)) return { matched: true }; + if (condition === "url" && value && session.driver.url().includes(value)) return { matched: true }; if (condition === "text" && value) { let cursor: string | undefined; for (let pageIndex = 0; pageIndex < 20; pageIndex += 1) { @@ -896,11 +1032,25 @@ export class BrowserAutomationService { private async ready(tabId: string, revision: number): Promise { const session = this.requireSession(tabId); - if (session.contents.isDestroyed()) { + if (session.driver.isDestroyed()) { + if (isContributed(session)) throw new EngineFallbackRequired("engine-disconnected"); throw new BrowserKernelError("TAB_CLOSED", "Browser tab is closed."); } if (session.revision !== revision) throw staleRef(session.revision); this.markBusy(tabId); + if (this.beforeCommand) { + try { + await this.beforeCommand(tabId); + } catch (error) { + throw new BrowserKernelError("BRIDGE_UNAVAILABLE", "Browser tab could not be resumed.", { + retryable: true, + cause: error + }); + } + if (this.sessions.get(tabId) !== session || session.driver.isDestroyed()) { + throw new BrowserKernelError("TAB_CLOSED", "Browser tab is closed."); + } + } await this.attach(session); return session; } @@ -915,14 +1065,14 @@ export class BrowserAutomationService { if (session.attachPromise) return session.attachPromise; session.attachPromise = (async () => { try { - if (!session.contents.debugger.isAttached()) session.contents.debugger.attach(CDP_VERSION); - await session.contents.debugger.sendCommand("Page.enable"); - await session.contents.debugger.sendCommand("DOM.enable"); - await session.contents.debugger.sendCommand("Runtime.enable"); - await session.contents.debugger.sendCommand("Accessibility.enable"); + if (!session.driver.isAttached()) session.driver.attach(CDP_VERSION); + await session.driver.send("Page.enable"); + await session.driver.send("DOM.enable"); + await session.driver.send("Runtime.enable"); + await session.driver.send("Accessibility.enable"); session.inflightRequests.clear(); session.networkLastChangeAt = Date.now(); - await session.contents.debugger.sendCommand("Network.enable"); + await session.driver.send("Network.enable"); if (session.presences.length > 0) await this.renderPresence(session); } catch (error) { session.attachPromise = null; @@ -942,8 +1092,13 @@ export class BrowserAutomationService { ): Promise { this.markBusy(session.tabId); try { - return await session.contents.debugger.sendCommand(method, params) as T; + return await session.driver.send(method, params) as T; } catch (error) { + if (isContributed(session)) { + // A contributed engine that lost its connection, or lacks the method, hands the tab to Chromium. + if (session.driver.isDestroyed()) throw new EngineFallbackRequired("engine-disconnected"); + if (isUnsupportedMethodError(error)) throw new EngineFallbackRequired("unsupported-method", `The browser engine does not support ${method}.`); + } if (/node|document|object/i.test(error instanceof Error ? error.message : String(error))) throw staleRef(); throw new BrowserKernelError("BRIDGE_UNAVAILABLE", "Chromium automation command failed.", { retryable: true, @@ -1082,6 +1237,8 @@ export class BrowserAutomationService { ): Promise<{ session: TabSession; entry: RefEntry; point: BrowserPointerResult }> { const session = await this.ready(tabId, revision); const entry = this.resolveRef(session, tabId, revision, ref); + // Without layout there is no point to aim at: callers act on the element itself. + if (!session.driver.layout) return { session, entry, point: { x: 0, y: 0 } }; const bounds = await this.box(session, entry.value.backendNodeId) ?? entry.bounds; if (!bounds || bounds.width <= 0 || bounds.height <= 0) throw staleRef(); return { @@ -1091,9 +1248,70 @@ export class BrowserAutomationService { }; } + /** Resolves the element and calls a function on it through the DOM: how engines without layout click and hover. */ + private async domCall( + session: TabSession, + entry: RefEntry, + functionDeclaration: string, + args: Array<{ value: unknown }> = [] + ): Promise { + const resolved = await this.command<{ object?: { objectId?: string } }>(session, "DOM.resolveNode", { + backendNodeId: entry.value.backendNodeId + }); + const objectId = resolved.object?.objectId; + if (!objectId) throw staleRef(); + const call = await this.commandAllowDialog<{ exceptionDetails?: unknown }>(session, "Runtime.callFunctionOn", { + objectId, + functionDeclaration, + arguments: args, + silent: true + }); + if (call.completed && call.value.exceptionDetails) throw staleRef(); + } + + /** + * A contributed engine's page, on the first page of a read or an observation: a bot wall (a challenge title or + * text, a challenge request, a 403, 429 or 503 document) or, for a read, text far too thin for the page's size + * hands the tab to Chromium. The thin-text check waits until the page has loaded and its requests have settled. + */ + private async assertEngineContent(session: TabSession, tree: readonly CdpAxNode[], checkThin: boolean): Promise { + let text = ""; + for (const node of tree) { + if (node.ignored || !READABLE_ROLES.has(axString(node.role))) continue; + const name = axString(node.name).trim(); + if (!name) continue; + text += `${name}\n`; + if (text.length > ENGINE_TEXT_SAMPLE_CHARS) break; + } + const textChars = text.trim().length; + if (session.challengeSeen + || (session.documentStatus !== null && BOT_WALL_STATUSES.has(session.documentStatus)) + || BOT_WALL_TITLE.test(session.driver.title()) + || (textChars < BOT_WALL_MAX_TEXT_CHARS && BOT_WALL_TEXT.test(text))) { + throw new EngineFallbackRequired("bot-wall"); + } + if (!checkThin || session.driver.isLoading() || session.inflightRequests.size > 0) return; + // More text than the thin rules could ever call thin: no need to measure the page. + if (textChars >= ENGINE_TEXT_SAMPLE_CHARS) return; + let htmlChars = 0; + try { + const response = await this.command<{ result?: { value?: unknown } }>(session, "Runtime.evaluate", { + expression: "document.documentElement ? document.documentElement.outerHTML.length : 0", + returnByValue: true, + silent: true + }); + htmlChars = typeof response.result?.value === "number" ? response.result.value : 0; + } catch (error) { + if (error instanceof EngineFallbackRequired) throw error; + return; + } + if (isThinText(textChars, htmlChars)) throw new EngineFallbackRequired("thin-text"); + } + private onMessage(tabId: string, method: string, params: unknown): void { const session = this.sessions.get(tabId); if (!session || !params || typeof params !== "object") return; + if (isContributed(session)) trackEngineDocument(session, method, params); if (method === "Network.requestWillBeSent") { const requestId = (params as { requestId?: unknown }).requestId; if (typeof requestId === "string" && requestId) { @@ -1197,7 +1415,7 @@ export class BrowserAutomationService { } private async renderPresence(session: TabSession): Promise { - if (!session.contents.debugger.isAttached()) return; + if (!session.driver.isAttached()) return; let worldCreated = false; if (session.presenceContextId === null) { const tree = await this.command<{ frameTree?: { frame?: { id?: string } } }>(session, "Page.getFrameTree"); @@ -1234,6 +1452,74 @@ export class BrowserAutomationService { } } +const DOM_CLICK_FUNCTION = "function(){if(typeof this.scrollIntoView==='function')this.scrollIntoView({block:'center'});if(typeof this.focus==='function')this.focus();if(typeof this.click==='function')this.click();else this.dispatchEvent(new MouseEvent('click',{bubbles:true,cancelable:true}));}"; +const DOM_HOVER_FUNCTION = "function(){for(const type of ['mouseover','mouseenter','mousemove'])this.dispatchEvent(new MouseEvent(type,{bubbles:type!=='mouseenter',cancelable:true}));}"; +const DOM_SCROLL_FUNCTION = "function(x,y){if(typeof this.scrollBy==='function')this.scrollBy(x,y);}"; + +/** Characters of page text sampled for the bot-wall and thin-text checks. */ +const ENGINE_TEXT_SAMPLE_CHARS = 10_000; +/** Under this much text a page counts as thin when its HTML is at least THIN_TEXT_MIN_HTML_CHARS. */ +const THIN_TEXT_CHARS = 200; +const THIN_TEXT_MIN_HTML_CHARS = 3_000; +/** Or when its text is under this share of the HTML (for pages up to THIN_RATIO_MAX_HTML_CHARS). */ +const THIN_TEXT_RATIO = 0.002; +const THIN_RATIO_MIN_HTML_CHARS = 100_000; +const THIN_RATIO_MAX_HTML_CHARS = 4_000_000; +/** Challenge phrases count only on short pages: an article may quote them. */ +const BOT_WALL_MAX_TEXT_CHARS = 3_000; +const BOT_WALL_STATUSES = new Set([403, 429, 503]); +const BOT_WALL_TITLE = /^\s*(?:just a moment|attention required|access denied|are you a robot|verifying you are human|security check)\b|captcha/i; +const BOT_WALL_TEXT = /performing security verification|checking (?:if the site connection is secure|your browser)|verify(?:ing)? (?:that )?you are (?:a )?human|bots use duckduckgo|enable javascript and cookies to continue|unusual traffic from your computer|press (?:and|&) hold/i; +const CHALLENGE_REQUEST = /\/cdn-cgi\/challenge-platform\//; + +export function isThinText(textChars: number, htmlChars: number): boolean { + if (htmlChars >= THIN_TEXT_MIN_HTML_CHARS && textChars < THIN_TEXT_CHARS) return true; + return htmlChars >= THIN_RATIO_MIN_HTML_CHARS && htmlChars <= THIN_RATIO_MAX_HTML_CHARS + && textChars < htmlChars * THIN_TEXT_RATIO; +} + +/** The main document's status and bot-challenge requests, for the bot-wall check (contributed engines). */ +function trackEngineDocument(session: TabSession, method: string, params: object): void { + if (method === "Network.responseReceived") { + const source = params as { type?: unknown; response?: { status?: unknown } }; + if (source.type !== "Document") return; + session.documentStatus = typeof source.response?.status === "number" ? source.response.status : null; + session.challengeSeen = false; + } else if (method === "Network.requestWillBeSent") { + const url = (params as { request?: { url?: unknown } }).request?.url; + if (typeof url === "string" && CHALLENGE_REQUEST.test(url)) session.challengeSeen = true; + } +} + +function isUnsupportedMethodError(error: unknown): boolean { + if (error && typeof error === "object" && (error as { code?: unknown }).code === -32601) return true; + const message = error instanceof Error ? error.message : String(error); + return /wasn't found|method not found|unknown method|not[ _]implemented|unknown command/i.test(message); +} + +/** A surface lent for the capture needs a frame or two before it has pixels. */ +async function captureLentSurface(contents: WebContents, signal?: AbortSignal): Promise { + let last: Electron.NativeImage | null = null; + for (let attempt = 0; attempt < LENT_SURFACE_CAPTURE_TRIES; attempt += 1) { + throwIfAborted(signal); + try { + // stayHidden/stayAwake: the page keeps its hidden state and is painted for the capture even if the window is occluded. + last = await contents.capturePage(undefined, { stayHidden: true, stayAwake: true }); + if (!last.isEmpty() && last.getSize().width > 0) return last; + } catch { + // The compositor has no frame for the view yet. + } + await new Promise((resolve) => setTimeout(resolve, LENT_SURFACE_CAPTURE_DELAY_MS)); + } + if (last) return last; + throw viewportUnavailable(); +} + +/** A tab a plugin-contributed engine drives (not a Browser card tab). */ +function isContributed(session: TabSession): boolean { + return session.driver.engine !== CHROMIUM_ENGINE; +} + function presenceExpression(payload: string): string { return `(()=>{const values=${payload};let host=globalThis.__canvasttyPresenceHost;if(!host||!host.isConnected){host=document.createElement('div');host.setAttribute('data-canvastty-presence','');host.style.cssText='all:initial!important;position:fixed!important;inset:0!important;z-index:2147483647!important;pointer-events:none!important;overflow:visible!important;';document.documentElement.appendChild(host);globalThis.__canvasttyPresenceHost=host;}host.replaceChildren(...values.map(v=>{const marker=document.createElement('div');marker.style.cssText='all:initial!important;position:absolute!important;left:'+v.x+'px!important;top:'+v.y+'px!important;transform:translate(-3px,-3px)!important;pointer-events:none!important;opacity:'+(v.stale?'.45':'1')+'!important;';const dot=document.createElement('span');dot.style.cssText='display:block!important;width:10px!important;height:10px!important;border-radius:999px!important;background:'+v.color+'!important;border:2px solid white!important;box-shadow:0 1px 5px rgba(0,0,0,.45)!important;';marker.append(dot);return marker;}));return true;})()`; } @@ -1489,3 +1775,14 @@ function abortableDelay(ms: number, signal?: AbortSignal): Promise { else signal.addEventListener("abort", abort, { once: true }); }); } + +function withTimeout(promise: Promise, ms: number): Promise { + let timer: NodeJS.Timeout | undefined; + return Promise.race([ + promise, + new Promise((_resolve, reject) => { + timer = setTimeout(() => reject(new Error("Timed out.")), ms); + timer.unref?.(); + }) + ]).finally(() => clearTimeout(timer)); +} diff --git a/src/main/services/browser/BrowserCommandDispatcher.ts b/src/main/services/browser/BrowserCommandDispatcher.ts index a69718bb..b3d743d3 100644 --- a/src/main/services/browser/BrowserCommandDispatcher.ts +++ b/src/main/services/browser/BrowserCommandDispatcher.ts @@ -37,6 +37,7 @@ export interface BrowserAuditWriter { export interface BrowserDispatchExecution { data?: unknown; tabId?: string | null; + notice?: string; } export interface BrowserCommandDispatcherOptions { @@ -277,7 +278,8 @@ export class BrowserCommandDispatcher { commandSequence, revisionBefore, revisionAfter, - ...(execution.data === undefined ? {} : { data: execution.data }) + ...(execution.data === undefined ? {} : { data: execution.data }), + ...(execution.notice ? { notice: execution.notice } : {}) }; } catch (error) { const normalized = timed.timedOut diff --git a/src/main/services/browser/BrowserCore.ts b/src/main/services/browser/BrowserCore.ts index 2397aea8..fd2f0ee5 100644 --- a/src/main/services/browser/BrowserCore.ts +++ b/src/main/services/browser/BrowserCore.ts @@ -11,7 +11,9 @@ import type { import { BrowserAutomationService, type BrowserPointerResult } from "./BrowserAutomationService.ts"; import { BrowserAuditStore } from "./BrowserAuditStore.ts"; import { BrowserCommandDispatcher } from "./BrowserCommandDispatcher.ts"; -import { BrowserKernelError, throwIfAborted } from "./BrowserErrors.ts"; +import { BrowserKernelError, EngineFallbackRequired, throwIfAborted, type EngineFallbackReason } from "./BrowserErrors.ts"; +import { isEngineRequest } from "./BrowserEngineTabs.ts"; +import { CHROMIUM_ENGINE } from "./TabDriver.ts"; import { BrowserPolicyService, DEFAULT_BROWSER_URL } from "./BrowserPolicyService.ts"; import { isSensitiveName } from "../safety/sensitiveNames.ts"; import type { AgentDisconnectReason } from "../agent-browser/protocol.ts"; @@ -39,8 +41,71 @@ export interface BrowserCoreHost { touchActor(actor: BrowserActor, tabId: string | null, cursor?: BrowserPointerResult): void; heartbeatActor(actor: BrowserActor, timestamp: number): void; disconnectActor(actor: BrowserActor): void; + /** + * Wakes a tab CanvasTTY paused or put to sleep while it was hidden, before a command for it runs. `reloaded` means + * the tab's page was loaded again: its earlier element refs are stale. + */ + prepareTab?(tabId: string): Promise<{ reloaded: boolean }>; + /** + * Opens a tab for this actor. An agent's tab may go to a plugin-contributed engine (then it stays in the background: + * `tabId` is not the active tab); a person's tab always opens in Chromium. Without it, `newTab` opens every tab. + */ + openTab?(url: string, options: { engine?: string; actor: BrowserActor }): Promise; + /** The engine driving a tab: `chromium` or null for Browser card tabs, the engine id for a contributed engine's tab. */ + tabEngine?(tabId: string): string | null; + /** + * Moves a contributed engine's tab to Chromium under the same id at the next document revision. Resolves once the + * page loaded (bounded), or at once when the person revealed the tab. + */ + moveTabToChromium?(tabId: string, reason: EngineFallbackReason): Promise; } +export interface BrowserOpenedTab { + snapshot: BrowserSnapshot; + tabId: string | null; + /** `chromium` or the contributed engine's id. */ + engine: string; + notice?: string; +} + +const ENGINE_FALLBACK_TEXT: Record = { + "bot-wall": "the site showed a bot check", + "thin-text": "the engine got too little text from the page", + "unsupported-method": "the engine does not support this command", + "unsupported-action": "the engine cannot do this action", + screenshot: "screenshots need Chromium", + "engine-disconnected": "the engine stopped", + revealed: "the tab was shown", + "remembered-site": "this site needed Chromium earlier" +}; + +export function engineFallbackNotice(reason: EngineFallbackReason): string { + return `Browser tab moved to Chromium (${ENGINE_FALLBACK_TEXT[reason]}). It keeps its tab ID; element refs from before are stale, observe it again.`; +} + +/** Commands a contributed engine's tab never runs: the tab moves to Chromium first. */ +function upfrontFallback(command: BrowserCommand): EngineFallbackReason | null { + switch (command.type) { + case "browser_screenshot": return "screenshot"; + case "browser_activate_tab": return "revealed"; + case "browser_drag": + case "browser_download_wait": return "unsupported-action"; + case "browser_wait_for": return command.condition === "download" ? "unsupported-action" : null; + default: return null; + } +} + +/** Commands that act on an element ref: after a move to Chromium their refs are stale, so they are not run again. */ +function actsOnRef(command: BrowserCommand): boolean { + return command.ref !== undefined || command.targetRef !== undefined; +} + +/** Agents' background tabs, per agent: commands without a tab id go to the tab the agent opened last. */ +const MAX_BACKGROUND_TAB_ACTORS = 256; + +export const BROWSER_TAB_RELOADED_NOTICE = + "Browser tab was reloaded: CanvasTTY had put it to sleep while it was hidden. Element refs from before are stale; observe it again."; + export interface BrowserCoreOptions { host: BrowserCoreHost; automation: BrowserAutomationService; @@ -54,6 +119,7 @@ export class BrowserCore { private readonly automation: BrowserAutomationService; private readonly policy: BrowserPolicyService; private readonly dispatcher: BrowserCommandDispatcher; + private readonly backgroundTabs = new Map(); constructor(options: BrowserCoreOptions) { this.host = options.host; @@ -69,7 +135,7 @@ export class BrowserCore { } execute(actor: BrowserActor, command: BrowserCommand, signal?: AbortSignal): Promise { - const normalized = normalizeTabCommand(command, this.host.getSnapshot().activeTabId); + const normalized = normalizeTabCommand(command, this.defaultTabId(actor)); return this.dispatcher.execute(actor, normalized, signal).then((result) => ( actor.kind === "agent" ? sanitizeAgentResult(result) : result )); @@ -101,6 +167,7 @@ export class BrowserCore { agentDisconnected(actor: BrowserActor, reason?: AgentDisconnectReason): void { // A dropped socket reconnects with the same connection; a revoked or expired one does not. this.dispatcher.clearActor(actor, { reconnecting: reason === "closed" }); + if (reason !== "closed" && actor.kind === "agent") this.backgroundTabs.delete(backgroundKey(actor)); this.host.disconnectActor(actor); } @@ -116,7 +183,7 @@ export class BrowserCore { actor: BrowserActor, command: BrowserCommand, signal: AbortSignal - ): Promise<{ data?: unknown; tabId?: string | null }> { + ): Promise<{ data?: unknown; tabId?: string | null; notice?: string }> { throwIfAborted(signal); assertCommandArguments(command); const tabId = this.resolveTabId(command); @@ -142,6 +209,16 @@ export class BrowserCore { } if (command.type === "browser_new_tab") { const url = command.url === undefined ? DEFAULT_BROWSER_URL : this.policy.assertNavigationUrl(command.url); + if (this.host.openTab) { + const opened = await this.host.openTab(url, { engine: command.engine, actor }); + this.host.touchActor(actor, opened.tabId); + if (actor.kind === "agent") this.rememberBackgroundTab(actor, opened); + return { + data: dataForActor(actor, opened.snapshot), + tabId: opened.tabId, + ...(opened.notice ? { notice: opened.notice } : {}) + }; + } const snapshot = await this.host.newTab(url); this.host.touchActor(actor, snapshot.activeTabId); return { data: dataForActor(actor, snapshot), tabId: snapshot.activeTabId }; @@ -152,6 +229,105 @@ export class BrowserCore { const requiredTabId = tabId ?? (() => { throw new BrowserKernelError("TAB_NOT_FOUND", "Browser command requires a tab."); })(); + // Closing needs no live page, and reload wakes a sleeping tab itself (that load is the reload). + const wake = command.type === "browser_close_tab" || command.type === "browser_reload" || !this.host.prepareTab + ? { reloaded: false } + : await this.host.prepareTab(requiredTabId); + throwIfAborted(signal); + if (!wake.reloaded) return await this.executeOnEngine(actor, command, requiredTabId, signal); + try { + return { ...await this.executeOnEngine(actor, command, requiredTabId, signal), notice: BROWSER_TAB_RELOADED_NOTICE }; + } catch (error) { + if (!(error instanceof BrowserKernelError)) throw error; + throw new BrowserKernelError(error.code, error.message, { + retryable: error.retryable, + details: { ...(error.details ?? {}), tabReloaded: true }, + cause: error + }); + } + } + + /** + * Runs a tab command, moving a contributed engine's tab to Chromium when it has to: before the command for what the + * engine never does (screenshots, showing the tab, drags, downloads), after it when automation asks for it (a bot + * wall, thin text, a missing CDP method, the engine gone). The command then runs again on Chromium, except one that + * acts on an element ref: those refs are stale, and the agent is told to observe again. + */ + private async executeOnEngine( + actor: BrowserActor, + command: BrowserCommand, + tabId: string, + signal: AbortSignal + ): Promise<{ data?: unknown; tabId?: string | null; notice?: string }> { + const move = this.host.moveTabToChromium?.bind(this.host); + const engine = this.host.tabEngine?.(tabId) ?? null; + if (move && engine !== null && engine !== CHROMIUM_ENGINE) { + const reason = upfrontFallback(command); + if (reason) { + await move(tabId, reason); + throwIfAborted(signal); + try { + const result = await this.executeTabCommand(actor, command, tabId, signal); + return reason === "revealed" ? result : { ...result, notice: engineFallbackNotice(reason) }; + } catch (error) { + // The move happened even when the command then failed on Chromium: the agent must know its refs are stale. + if (!(error instanceof BrowserKernelError) || reason === "revealed") throw error; + throw new BrowserKernelError(error.code, error.message, { + retryable: error.retryable, + details: { ...(error.details ?? {}), movedToChromium: true }, + cause: error + }); + } + } + } + try { + return await this.executeTabCommand(actor, command, tabId, signal); + } catch (error) { + if (!(error instanceof EngineFallbackRequired)) throw error; + if (!move) { + throw new BrowserKernelError("BRIDGE_UNAVAILABLE", "Browser engine cannot continue this tab.", { retryable: true, cause: error }); + } + await move(tabId, error.reason); + throwIfAborted(signal); + if (actsOnRef(command)) { + throw new BrowserKernelError("STALE_REF", engineFallbackNotice(error.reason), { + retryable: true, + details: { movedToChromium: true }, + cause: error + }); + } + return { ...await this.executeTabCommand(actor, command, tabId, signal), notice: engineFallbackNotice(error.reason) }; + } + } + + /** The tab a command without a tab id goes to: the agent's own background tab while it exists, else the active one. */ + private defaultTabId(actor: BrowserActor): string | null { + const active = this.host.getSnapshot().activeTabId; + if (actor.kind !== "agent") return active; + const key = backgroundKey(actor); + const background = this.backgroundTabs.get(key); + if (!background) return active; + if (this.host.getTab(background)) return background; + this.backgroundTabs.delete(key); + return active; + } + + private rememberBackgroundTab(actor: Extract, opened: BrowserOpenedTab): void { + const key = backgroundKey(actor); + this.backgroundTabs.delete(key); + if (!opened.tabId || opened.tabId === opened.snapshot.activeTabId) return; + this.backgroundTabs.set(key, opened.tabId); + while (this.backgroundTabs.size > MAX_BACKGROUND_TAB_ACTORS) { + this.backgroundTabs.delete(this.backgroundTabs.keys().next().value!); + } + } + + private async executeTabCommand( + actor: BrowserActor, + command: BrowserCommand, + requiredTabId: string, + signal: AbortSignal + ): Promise<{ data?: unknown; tabId?: string | null }> { const tab = this.host.getTab(requiredTabId); if (!tab) throw new BrowserKernelError("TAB_NOT_FOUND", "Browser tab is unavailable."); if (tab.status === "crashed" && command.type !== "browser_reload" && command.type !== "browser_close_tab") { @@ -175,6 +351,8 @@ export class BrowserCore { return { data: dataForActor(actor, snapshot), tabId: requiredTabId }; } case "browser_activate_tab": + // The agent chose a tab to work on in the open: its commands without a tab id go to the active tab again. + if (actor.kind === "agent") this.backgroundTabs.delete(backgroundKey(actor)); return { data: dataForActor(actor, await this.host.activateTab(requiredTabId)), tabId: requiredTabId }; case "browser_navigate": { const url = this.policy.assertNavigationUrl(command.url); @@ -307,6 +485,10 @@ export class BrowserCore { } } +function backgroundKey(actor: Extract): string { + return `${actor.agentId}\u0000${actor.terminalSessionId}`; +} + function boundedTimeout(value: number | undefined): number { return Number.isFinite(value) ? Math.min(120_000, Math.max(50, value!)) : 120_000; } @@ -330,6 +512,9 @@ function assertCommandArguments(command: BrowserCommand): void { throw new BrowserKernelError("STALE_REF", "Browser element reference is invalid.", { retryable: true }); } } + if (command.engine !== undefined && (command.type !== "browser_new_tab" || !isEngineRequest(command.engine))) { + throw new BrowserKernelError("PERMISSION_DENIED", "Browser engine must be auto, chromium or an installed engine id."); + } if (command.type === "browser_navigate" && typeof command.url !== "string") { throw new BrowserKernelError("INVALID_URL", "Browser navigate requires a URL."); } @@ -445,7 +630,8 @@ function sanitizeAgentValue(value: unknown, key = "", depth = 0): unknown { if (depth > 12) return "[REDACTED]"; if (value === null || typeof value === "number" || typeof value === "boolean") return value; const normalizedKey = key.toLowerCase(); - if (normalizedKey === "favicon") return null; + // The favicon and a sleeping tab's preview are for the person's card: raw page pixels, never redacted for agents. + if (normalizedKey === "favicon" || normalizedKey === "preview") return null; if (isSensitiveName(normalizedKey)) { return "[REDACTED]"; } diff --git a/src/main/services/browser/BrowserEngineTabs.ts b/src/main/services/browser/BrowserEngineTabs.ts new file mode 100644 index 00000000..39334837 --- /dev/null +++ b/src/main/services/browser/BrowserEngineTabs.ts @@ -0,0 +1,375 @@ +import { randomUUID } from "node:crypto"; +import type { AgentPresenceSnapshot, BrowserActor, BrowserTabSnapshot } from "../../../shared/contracts.ts"; +import type { BrowserAutomationService } from "./BrowserAutomationService.ts"; +import type { BrowserCoreTab } from "./BrowserCore.ts"; +import type { EngineFallbackReason } from "./BrowserErrors.ts"; +import { CdpTabDriver, type CdpSocketFactory } from "./CdpTabDriver.ts"; +import { CHROMIUM_ENGINE } from "./TabDriver.ts"; + +/** A running, trusted plugin service that contributes a browser engine (`browser:engine`). */ +export interface BrowserEngineProvider { + pluginId: string; + pluginName: string; + serviceId: string; + /** The id agents pass as `engine` to browser_new_tab. */ + engineId: string; + title: string; + /** The engine lays pages out for real (default false: clicks go through the DOM). */ + layout: boolean; +} + +export interface BrowserEngineTabsHost { + automation: BrowserAutomationService; + /** Engines that may take tabs now: plugin enabled, native code trusted, service running. */ + providers(): BrowserEngineProvider[]; + /** `canvastty.browserEngine.openTab` `{ engineId, tabId }` → `{ webSocketUrl }`: a local CDP endpoint for this tab. */ + openEngineTab(provider: BrowserEngineProvider, tabId: string): Promise; + /** `canvastty.browserEngine.closeTab` `{ engineId, tabId }` (a notification). */ + closeEngineTab(provider: BrowserEngineProvider, tabId: string): void; + /** + * Opens a Chromium tab with this id at this document revision without showing it. `waitForLoad` resolves once its + * page loaded (bounded by the host); otherwise at once. + */ + openChromiumTab(tabId: string, url: string, revision: number, options: { waitForLoad: boolean }): Promise; + /** Something the person's tab list shows changed. */ + changed(): void; + connect?: CdpSocketFactory; + now?: () => number; +} + +export interface EngineChoice { + provider: BrowserEngineProvider | null; + /** Said to the agent when it asked for an engine it does not get. */ + notice?: string; +} + +interface EngineTab { + id: string; + provider: BrowserEngineProvider; + driver: CdpTabDriver; + revision: number; + /** The last http(s) URL of the tab: what Chromium opens when the tab moves. */ + lastUrl: string; + status: BrowserCoreTab["status"]; + /** + * A navigation the core started (open, navigate, back/forward, reload) already advanced the revision; its commit + * does not advance it again, so a command that raced the commit is not refused as stale. + */ + pendingCommit: boolean; + unlisten: () => void; +} + +/** Sites remembered per app session, most recent last. */ +const MAX_REMEMBERED_SITES = 200; +/** An engine that could not open a tab is skipped by `auto` for this long. */ +const ENGINE_RETRY_AFTER_MS = 60_000; +/** Reasons that say something about the site itself: its next tabs go straight to Chromium. */ +const SITE_REASONS = new Set(["bot-wall", "thin-text", "unsupported-method", "engine-disconnected"]); +const ENGINE_ID = /^[a-z0-9](?:[a-z0-9._-]{0,62}[a-z0-9])?$/; + +export function isEngineRequest(value: unknown): value is string { + return typeof value === "string" && (value === "auto" || ENGINE_ID.test(value)); +} + +/** + * Tabs driven by plugin-contributed engines (EP: browser engines). Policy stays in the core: only an agent's new tab + * may use one, never a tab the person opens; such a tab is never shown (showing it moves it to Chromium first); it + * gets no cookies and no browser profile; and whenever the engine cannot serve it (a bot wall, thin text, a missing + * CDP method, a screenshot, a crash) it moves to Chromium under the same tab id and the site is remembered for the + * rest of the session. + */ +export class BrowserEngineTabs { + private readonly host: BrowserEngineTabsHost; + private readonly tabs = new Map(); + private readonly moving = new Map; url: string; revision: number }>(); + private readonly rememberedSites = new Map(); + private readonly unavailableUntil = new Map(); + private readonly now: () => number; + + constructor(host: BrowserEngineTabsHost) { + this.host = host; + this.now = host.now ?? Date.now; + } + + get size(): number { + return this.tabs.size + this.moving.size; + } + + has(tabId: string): boolean { + return this.tabs.has(tabId); + } + + /** The contributed engine driving a tab, or null (Chromium or unknown). */ + engineOf(tabId: string): string | null { + return this.tabs.get(tabId)?.provider.engineId ?? null; + } + + coreTab(tabId: string): BrowserCoreTab | null { + const tab = this.tabs.get(tabId); + if (tab) return { id: tab.id, url: this.urlOf(tab), documentRevision: tab.revision, status: tab.status }; + const moving = this.moving.get(tabId); + return moving ? { id: tabId, url: moving.url, documentRevision: moving.revision, status: "loading" } : null; + } + + snapshots(agents: readonly AgentPresenceSnapshot[]): BrowserTabSnapshot[] { + return [...this.tabs.values()].map((tab) => { + const url = this.urlOf(tab); + return { + id: tab.id, + url, + title: tab.driver.title() || hostOf(url) || url, + loading: tab.status === "loading" || tab.driver.isLoading(), + canGoBack: false, + canGoForward: false, + documentRevision: tab.revision, + status: tab.status, + favicon: null, + agents: agents.filter((presence) => presence.currentTabId === tab.id).map((presence) => structuredClone(presence)), + crashState: null, + engine: tab.provider.engineId + }; + }); + } + + /** Sites that needed Chromium in this session (for Settings and tests). */ + rememberedSitesList(): string[] { + return [...this.rememberedSites.keys()]; + } + + /** + * The engine for a new tab. A person's tab and `chromium` always get Chromium. `auto` (the default) takes the first + * running engine unless the site needed Chromium before or the engine just failed to open a tab; a named engine + * that is missing, or a remembered site, get Chromium with a notice. + */ + choose(request: { engine?: string; actor: BrowserActor; url: string }): EngineChoice { + if (request.actor.kind !== "agent") return { provider: null }; + const requested = request.engine ?? "auto"; + if (requested === CHROMIUM_ENGINE) return { provider: null }; + const providers = this.host.providers() + .filter((provider) => ENGINE_ID.test(provider.engineId) && provider.engineId !== CHROMIUM_ENGINE) + .sort((left, right) => left.pluginId.localeCompare(right.pluginId) || left.engineId.localeCompare(right.engineId)); + const site = hostOf(request.url); + if (requested === "auto") { + if (site && this.rememberedSites.has(site)) return { provider: null }; + const now = this.now(); + return { provider: providers.find((provider) => (this.unavailableUntil.get(providerKey(provider)) ?? 0) <= now) ?? null }; + } + const provider = providers.find((candidate) => candidate.engineId === requested); + if (!provider) { + return { provider: null, notice: `Browser engine "${requested}" is not installed or not running; the tab opened in Chromium.` }; + } + if (site && this.rememberedSites.has(site)) { + return { provider: null, notice: `${site} needed Chromium earlier in this session; the tab opened in Chromium.` }; + } + return { provider }; + } + + /** Opens an engine tab (never shown) and starts loading the URL. Rejects when the engine cannot take it. */ + async open(provider: BrowserEngineProvider, url: string, tabId: string = randomUUID()): Promise { + let driver: CdpTabDriver | null = null; + try { + const endpoint = await this.host.openEngineTab(provider, tabId); + const webSocketUrl = endpoint && typeof endpoint === "object" ? (endpoint as { webSocketUrl?: unknown }).webSocketUrl : undefined; + if (typeof webSocketUrl !== "string") throw new Error("Browser engine did not return a CDP endpoint."); + driver = await CdpTabDriver.open({ + url: webSocketUrl, + engine: provider.engineId, + layout: provider.layout, + ...(this.host.connect ? { connect: this.host.connect } : {}) + }); + const opened = driver; + const tab: EngineTab = { + id: tabId, + provider, + driver: opened, + revision: 0, + lastUrl: url, + status: "loading", + pendingCommit: false, + unlisten: () => undefined + }; + tab.unlisten = opened.listen({ + message: (method, params) => this.onEvent(tab, method, params), + detach: () => this.onDisconnect(tab) + }); + this.tabs.set(tabId, tab); + await this.host.automation.registerDriver(tabId, opened, tab.revision); + this.host.changed(); + void this.load(tab, url); + return tabId; + } catch (error) { + const tab = this.tabs.get(tabId); + if (tab) { + this.forget(tab); + } else { + driver?.close(); + this.host.closeEngineTab(provider, tabId); + } + this.unavailableUntil.set(providerKey(provider), this.now() + ENGINE_RETRY_AFTER_MS); + throw error; + } + } + + async navigate(tabId: string, url: string): Promise { + const tab = this.require(tabId); + const site = hostOf(url); + if (site && this.rememberedSites.has(site)) { + await this.moveToChromium(tabId, "remembered-site", { url, waitForLoad: false }); + return; + } + this.advance(tab); + await this.load(tab, url); + } + + async history(tabId: string, delta: -1 | 1): Promise { + const tab = this.require(tabId); + if (!await tab.driver.canGo(delta)) return; + this.advance(tab); + tab.pendingCommit = true; + await tab.driver.history(delta).catch(() => this.markError(tab)); + } + + async reload(tabId: string): Promise { + const tab = this.require(tabId); + this.advance(tab); + tab.pendingCommit = true; + await tab.driver.reload().catch(() => this.markError(tab)); + } + + close(tabId: string): void { + const tab = this.tabs.get(tabId); + if (!tab) return; + this.forget(tab); + this.host.changed(); + } + + /** + * Moves the tab to Chromium: same tab id, the next document revision, the same URL (or `url`). Sites the engine + * failed on are remembered for the session. Runs once per tab; later calls wait for the same move. + */ + moveToChromium( + tabId: string, + reason: EngineFallbackReason, + options: { url?: string; waitForLoad?: boolean } = {} + ): Promise { + const pending = this.moving.get(tabId); + if (pending) return pending.promise; + const tab = this.tabs.get(tabId); + if (!tab) return Promise.resolve(); + const url = options.url ?? this.urlOf(tab); + const revision = tab.revision + 1; + const currentSite = hostOf(this.urlOf(tab)); + if (SITE_REASONS.has(reason) && currentSite) this.remember(currentSite, reason); + this.forget(tab); + const promise = (async () => { + try { + await this.host.openChromiumTab(tabId, url, revision, { waitForLoad: options.waitForLoad ?? true }); + } finally { + this.moving.delete(tabId); + this.host.changed(); + } + })(); + this.moving.set(tabId, { promise, url, revision }); + return promise; + } + + dispose(): void { + for (const tab of [...this.tabs.values()]) this.forget(tab); + } + + private async load(tab: EngineTab, url: string): Promise { + tab.status = "loading"; + tab.lastUrl = url; + tab.pendingCommit = true; + try { + await tab.driver.navigate(url); + } catch { + this.markError(tab); + } + } + + private advance(tab: EngineTab): void { + tab.revision += 1; + tab.status = "loading"; + this.host.automation.updateRevision(tab.id, tab.revision); + this.host.changed(); + } + + private markError(tab: EngineTab): void { + if (this.tabs.get(tab.id) !== tab) return; + tab.pendingCommit = false; + tab.status = "error"; + this.host.changed(); + } + + private onEvent(tab: EngineTab, method: string, params: unknown): void { + if (this.tabs.get(tab.id) !== tab) return; + if (method === "Page.frameNavigated") { + const frame = ((params ?? {}) as { frame?: { parentId?: unknown; url?: unknown } }).frame; + if (!frame || frame.parentId) return; + if (typeof frame.url === "string" && isHttpUrl(frame.url)) tab.lastUrl = frame.url; + if (tab.pendingCommit) { + tab.pendingCommit = false; + this.host.changed(); + } else { + // The page navigated by itself (a link, a script): refs from before are stale. + this.advance(tab); + } + } else if (method === "Page.loadEventFired") { + tab.status = "ready"; + this.host.changed(); + } + } + + private onDisconnect(tab: EngineTab): void { + if (this.tabs.get(tab.id) !== tab) return; + // The engine went away (crash, restart, plugin stopped): the tab continues in Chromium. + void this.moveToChromium(tab.id, "engine-disconnected", { waitForLoad: false }).catch(() => undefined); + } + + private forget(tab: EngineTab): void { + if (this.tabs.get(tab.id) === tab) this.tabs.delete(tab.id); + tab.unlisten(); + this.host.automation.unregister(tab.id); + tab.driver.close(); + this.host.closeEngineTab(tab.provider, tab.id); + } + + private remember(site: string, reason: EngineFallbackReason): void { + this.rememberedSites.delete(site); + this.rememberedSites.set(site, reason); + while (this.rememberedSites.size > MAX_REMEMBERED_SITES) { + this.rememberedSites.delete(this.rememberedSites.keys().next().value!); + } + } + + private urlOf(tab: EngineTab): string { + const current = tab.driver.url(); + return isHttpUrl(current) ? current : tab.lastUrl; + } + + private require(tabId: string): EngineTab { + const tab = this.tabs.get(tabId); + if (!tab) throw new Error("Browser engine tab is unavailable."); + return tab; + } +} + +function providerKey(provider: BrowserEngineProvider): string { + return `${provider.pluginId}\u0000${provider.serviceId}\u0000${provider.engineId}`; +} + +function isHttpUrl(value: string): boolean { + return /^https?:\/\//i.test(value); +} + +/** The site a URL belongs to for the fallback memory: its host name, lowercased, without `www.`. */ +export function hostOf(value: string): string | null { + try { + const url = new URL(value); + if (url.protocol !== "http:" && url.protocol !== "https:") return null; + return url.hostname.toLowerCase().replace(/^www\./, "") || null; + } catch { + return null; + } +} diff --git a/src/main/services/browser/BrowserErrors.ts b/src/main/services/browser/BrowserErrors.ts index 165031fd..df0c1c9c 100644 --- a/src/main/services/browser/BrowserErrors.ts +++ b/src/main/services/browser/BrowserErrors.ts @@ -22,6 +22,35 @@ export class BrowserKernelError extends Error { } } +/** + * Why a tab driven by a contributed engine moves to Chromium: the page looked like a bot wall, its text came out thin + * for its size, the engine lacks a CDP method or an action, a screenshot was asked for, the engine went away, the + * tab became visible to the person, or it navigated to a site that already needed Chromium in this session. + */ +export type EngineFallbackReason = + | "bot-wall" + | "thin-text" + | "unsupported-method" + | "unsupported-action" + | "screenshot" + | "engine-disconnected" + | "revealed" + | "remembered-site"; + +/** + * Thrown by automation on a tab a contributed engine drives when that tab has to continue in Chromium. BrowserCore + * moves the tab (same tab id, next document revision) and runs the command again there; it never reaches an agent. + */ +export class EngineFallbackRequired extends Error { + readonly reason: EngineFallbackReason; + + constructor(reason: EngineFallbackReason, message = `Browser engine cannot continue: ${reason}.`) { + super(message); + this.name = "EngineFallbackRequired"; + this.reason = reason; + } +} + export function browserError(error: unknown): BrowserError { if (error instanceof BrowserKernelError) { return { diff --git a/src/main/services/browser/BrowserTabLifecycle.ts b/src/main/services/browser/BrowserTabLifecycle.ts new file mode 100644 index 00000000..d6ca1966 --- /dev/null +++ b/src/main/services/browser/BrowserTabLifecycle.ts @@ -0,0 +1,337 @@ +/** + * When a hidden browser tab is paused (frozen) or put to sleep (discarded), and when it is woken again. + * + * Hidden tabs already run with Chromium background throttling, but throttling leaves requestAnimationFrame loops and + * a timer wake-up every second running. A tab that stays hidden and undriven for `freezeAfterMs` is frozen (Chromium + * stops its timers, rAF and tasks); one that stays so for `discardAfterMs`, or the least recently used ones when more + * than `maxLiveHiddenTabs` hidden tabs are alive, is discarded (its WebContents is closed; the host keeps what it needs + * to bring it back). Showing a tab, or any command for it, wakes it first: `ensureLive` resolves only once the tab + * runs again, so no automation reaches a frozen page. + * + * Every transition of one tab runs on that tab's own queue, so a wake that arrives while a freeze or discard is in + * flight waits for it and then undoes it. The host decides what blocks a transition (playing media, a download, an + * open dialog, a beforeunload handler, an agent on the tab); a blocked tab is retried later, never forced. + * This class holds no Electron objects: BrowserService is the host, and tests drive it with fake timers. + */ + +export type TabLifecycleState = "active" | "frozen" | "discarded"; + +export interface TabLifecycleHost { + /** Why the tab must not be frozen now, or null. */ + freezeBlocker(tabId: string): string | null; + /** Why the tab must not be discarded now, or null. May consult the page (a beforeunload handler). */ + discardBlocker(tabId: string): Promise; + freeze(tabId: string): Promise; + resume(tabId: string): Promise; + /** False when the tab could not be discarded after all (it was shown or driven meanwhile); it then stays as it was. */ + discard(tabId: string): Promise; + restore(tabId: string): Promise; + stateChanged(tabId: string, state: TabLifecycleState): void; +} + +export interface TabLifecycleOptions { + enabled?: boolean; + freezeAfterMs?: number; + discardAfterMs?: number; + maxLiveHiddenTabs?: number; + /** How long a blocked or failed freeze or discard waits before it is tried again. */ + retryAfterMs?: number; + now?: () => number; + setTimer?: (callback: () => void, ms: number) => unknown; + clearTimer?: (handle: unknown) => void; + onError?: (error: unknown) => void; +} + +export const TAB_FREEZE_AFTER_MS = 30_000; +export const TAB_DISCARD_AFTER_MS = 10 * 60_000; +export const MAX_LIVE_HIDDEN_TABS = 6; + +interface Entry { + id: string; + state: TabLifecycleState; + visible: boolean; + busy: boolean; + /** When the tab was last shown, driven or woken; idle time counts from the later of this and hiding. */ + lastUsedAt: number; + hiddenSince: number | null; + freezeNotBefore: number; + discardNotBefore: number; + /** A discard for the hidden-tab limit is queued; the tab no longer counts as live. */ + discardQueued: boolean; + timer: unknown; + queue: Promise; +} + +export class BrowserTabLifecycle { + private readonly host: TabLifecycleHost; + private readonly entries = new Map(); + private readonly freezeAfterMs: number; + private readonly discardAfterMs: number; + private readonly maxLiveHiddenTabs: number; + private readonly retryAfterMs: number; + private readonly now: () => number; + private readonly setTimer: (callback: () => void, ms: number) => unknown; + private readonly clearTimer: (handle: unknown) => void; + private readonly onError: (error: unknown) => void; + private enabled: boolean; + private disposed = false; + + constructor(host: TabLifecycleHost, options: TabLifecycleOptions = {}) { + this.host = host; + this.enabled = options.enabled ?? true; + this.freezeAfterMs = options.freezeAfterMs ?? TAB_FREEZE_AFTER_MS; + this.discardAfterMs = options.discardAfterMs ?? TAB_DISCARD_AFTER_MS; + this.maxLiveHiddenTabs = options.maxLiveHiddenTabs ?? MAX_LIVE_HIDDEN_TABS; + this.retryAfterMs = options.retryAfterMs ?? this.freezeAfterMs; + this.now = options.now ?? Date.now; + this.setTimer = options.setTimer ?? ((callback, ms) => { + const timer = setTimeout(callback, ms); + timer.unref?.(); + return timer; + }); + this.clearTimer = options.clearTimer ?? ((handle) => clearTimeout(handle as NodeJS.Timeout)); + this.onError = options.onError ?? ((error) => console.warn("CanvasTTY browser tab lifecycle step failed.", error)); + } + + get isEnabled(): boolean { + return this.enabled; + } + + state(tabId: string): TabLifecycleState { + return this.entries.get(tabId)?.state ?? "active"; + } + + /** Starts tracking a tab (a no-op for one already tracked, such as a tab being restored). */ + track(tabId: string, visible: boolean): void { + if (this.disposed || this.entries.has(tabId)) return; + const now = this.now(); + const entry: Entry = { + id: tabId, + state: "active", + visible, + busy: false, + lastUsedAt: now, + hiddenSince: visible ? null : now, + freezeNotBefore: 0, + discardNotBefore: 0, + discardQueued: false, + timer: null, + queue: Promise.resolve() + }; + this.entries.set(tabId, entry); + this.schedule(entry); + } + + untrack(tabId: string): void { + const entry = this.entries.get(tabId); + if (!entry) return; + this.cancelTimer(entry); + this.entries.delete(tabId); + } + + setVisible(tabId: string, visible: boolean): void { + const entry = this.entries.get(tabId); + if (!entry || entry.visible === visible) return; + entry.visible = visible; + if (visible) { + entry.hiddenSince = null; + this.touch(entry); + if (entry.state !== "active") void this.ensureLive(tabId).catch(this.onError); + return; + } + entry.hiddenSince = this.now(); + this.resetRetries(entry); + this.schedule(entry); + } + + /** Automation started or stopped driving the tab (BrowserAutomationService's busy window). */ + setBusy(tabId: string, busy: boolean): void { + const entry = this.entries.get(tabId); + if (!entry || entry.busy === busy) return; + entry.busy = busy; + this.touch(entry); + if (!busy) this.schedule(entry); + } + + /** + * Resolves once the tab runs: a frozen tab is resumed, a discarded one restored (`reloaded`). Waits for any + * transition of the tab already in flight, so a wake never overtakes the freeze or discard it has to undo. + */ + ensureLive(tabId: string): Promise<{ reloaded: boolean }> { + const entry = this.entries.get(tabId); + if (!entry) return Promise.resolve({ reloaded: false }); + this.touch(entry); + const result = this.enqueue(entry, async () => { + if (this.entries.get(tabId) !== entry) return { reloaded: false }; + if (entry.state === "frozen") { + try { + await this.host.resume(tabId); + } finally { + // A resume that failed (the automation channel went away) must not leave the tab marked paused forever. + this.setState(entry, "active"); + } + return { reloaded: false }; + } + if (entry.state === "discarded") { + await this.host.restore(tabId); + this.setState(entry, "active"); + return { reloaded: true }; + } + return { reloaded: false }; + }); + return result.finally(() => this.schedule(entry)); + } + + /** Off: nothing new is paused and paused tabs resume; sleeping tabs wake when they are next shown or used. */ + setEnabled(enabled: boolean): void { + if (this.enabled === enabled) return; + this.enabled = enabled; + for (const entry of this.entries.values()) { + this.resetRetries(entry); + if (!enabled) { + this.cancelTimer(entry); + if (entry.state === "frozen") void this.ensureLive(entry.id).catch(this.onError); + } else { + // Idle time counts from now: turning the setting on must not put long-hidden tabs to sleep at once. + entry.lastUsedAt = this.now(); + this.schedule(entry); + } + } + } + + dispose(): void { + this.disposed = true; + for (const entry of this.entries.values()) this.cancelTimer(entry); + this.entries.clear(); + } + + private touch(entry: Entry): void { + entry.lastUsedAt = this.now(); + this.resetRetries(entry); + this.cancelTimer(entry); + } + + private resetRetries(entry: Entry): void { + entry.freezeNotBefore = 0; + entry.discardNotBefore = 0; + } + + private deferRetries(entry: Entry): void { + const retryAt = this.now() + this.retryAfterMs; + // Either deadline can already be due, including one that expired while the failed transition was in flight. + entry.freezeNotBefore = Math.max(entry.freezeNotBefore, retryAt); + entry.discardNotBefore = Math.max(entry.discardNotBefore, retryAt); + } + + private idle(entry: Entry): boolean { + return this.enabled && !this.disposed && !entry.visible && !entry.busy && this.entries.get(entry.id) === entry; + } + + private idleSince(entry: Entry): number { + return Math.max(entry.hiddenSince ?? entry.lastUsedAt, entry.lastUsedAt); + } + + private freezeAt(entry: Entry): number { + return Math.max(this.idleSince(entry) + this.freezeAfterMs, entry.freezeNotBefore); + } + + private discardAt(entry: Entry): number { + return Math.max(this.idleSince(entry) + this.discardAfterMs, entry.discardNotBefore); + } + + private schedule(entry: Entry): void { + this.cancelTimer(entry); + if (!this.idle(entry) || entry.state === "discarded") return; + const due = entry.state === "active" + ? Math.min(this.freezeAt(entry), this.discardAt(entry)) + : this.discardAt(entry); + entry.timer = this.setTimer(() => { + entry.timer = null; + void this.enqueue(entry, () => this.step(entry)).catch(this.onError); + }, Math.max(0, due - this.now())); + } + + private cancelTimer(entry: Entry): void { + if (entry.timer === null) return; + this.clearTimer(entry.timer); + entry.timer = null; + } + + private async step(entry: Entry): Promise { + try { + if (!this.idle(entry)) return; + const now = this.now(); + if (entry.state === "active" && now >= this.freezeAt(entry)) { + const blocker = this.host.freezeBlocker(entry.id); + if (blocker) entry.freezeNotBefore = now + this.retryAfterMs; + else { + await this.host.freeze(entry.id); + if (this.entries.get(entry.id) === entry) this.setState(entry, "frozen"); + } + } + if (entry.state !== "discarded" && this.idle(entry) && this.now() >= this.discardAt(entry)) { + await this.tryDiscard(entry); + } + this.enforceHiddenLimit(); + } catch (error) { + this.deferRetries(entry); + throw error; + } finally { + this.schedule(entry); + } + } + + private async tryDiscard(entry: Entry): Promise { + if (!this.idle(entry) || entry.state === "discarded") return; + const blocker = await this.host.discardBlocker(entry.id); + // Shown, driven or untracked while the page was asked: leave it. + if (!this.idle(entry) || this.state(entry.id) === "discarded") return; + if (blocker) { + entry.discardNotBefore = this.now() + this.retryAfterMs; + return; + } + const discarded = await this.host.discard(entry.id); + if (discarded && this.entries.get(entry.id) === entry) this.setState(entry, "discarded"); + } + + /** More than maxLiveHiddenTabs hidden tabs alive: discard the least recently used that have been idle a while. */ + private enforceHiddenLimit(): void { + const live = [...this.entries.values()].filter((entry) => ( + !entry.visible && entry.state !== "discarded" && !entry.discardQueued + )); + let excess = live.length - this.maxLiveHiddenTabs; + if (excess <= 0) return; + const now = this.now(); + const candidates = live + .filter((entry) => this.idle(entry) && now - this.idleSince(entry) >= this.freezeAfterMs && now >= entry.discardNotBefore) + .sort((left, right) => left.lastUsedAt - right.lastUsedAt); + for (const entry of candidates) { + if (excess <= 0) break; + excess -= 1; + entry.discardQueued = true; + void this.enqueue(entry, async () => { + try { + await this.tryDiscard(entry); + } catch (error) { + this.deferRetries(entry); + throw error; + } finally { + entry.discardQueued = false; + this.schedule(entry); + } + }).catch(this.onError); + } + } + + private setState(entry: Entry, state: TabLifecycleState): void { + if (entry.state === state) return; + entry.state = state; + this.host.stateChanged(entry.id, state); + } + + private enqueue(entry: Entry, task: () => Promise): Promise { + const run = entry.queue.then(task, task); + entry.queue = run.catch(() => undefined); + return run; + } +} diff --git a/src/main/services/browser/CdpTabDriver.ts b/src/main/services/browser/CdpTabDriver.ts new file mode 100644 index 00000000..60cb670c --- /dev/null +++ b/src/main/services/browser/CdpTabDriver.ts @@ -0,0 +1,383 @@ +import type { TabDriver, TabDriverListeners } from "./TabDriver.ts"; + +/** One text-frame connection to a local CDP endpoint. */ +export interface CdpSocket { + send(text: string): void; + close(): void; +} + +export interface CdpSocketHandlers { + message(text: string): void; + close(): void; +} + +export type CdpSocketFactory = (url: string, handlers: CdpSocketHandlers) => Promise; + +export interface CdpTabDriverOptions { + /** The engine's CDP WebSocket for this tab, from the plugin service. Loopback only. */ + url: string; + engine: string; + layout: boolean; + connect?: CdpSocketFactory; + /** Connecting and opening the page (default 10 s). */ + openTimeoutMs?: number; + /** One CDP command (default 30 s). */ + commandTimeoutMs?: number; +} + +const DEFAULT_OPEN_TIMEOUT_MS = 10_000; +const DEFAULT_COMMAND_TIMEOUT_MS = 30_000; +const MAX_PENDING_COMMANDS = 256; +/** A page that never reports its load event stops counting as loading after this long. */ +const MAX_LOADING_MS = 45_000; +const LOOPBACK_HOSTS = new Set(["127.0.0.1", "[::1]", "localhost"]); + +/** + * The CDP WebSocket a plugin handed over must be a plain `ws://` endpoint on this computer's loopback interface with + * an explicit port: the core never connects a tab to another host, and never sends credentials to it. + */ +export function assertLoopbackCdpUrl(value: unknown): string { + if (typeof value !== "string" || value.length > 512) throw new Error("Browser engine endpoint is invalid."); + let url: URL; + try { + url = new URL(value); + } catch { + throw new Error("Browser engine endpoint is invalid."); + } + if (url.protocol !== "ws:" || !LOOPBACK_HOSTS.has(url.hostname) || !url.port + || url.username || url.password || url.hash) { + throw new Error("Browser engine endpoint must be a ws:// address on 127.0.0.1, [::1] or localhost with a port."); + } + return url.toString(); +} + +/** Node's and Electron's built-in WebSocket client. */ +export const connectWebSocket: CdpSocketFactory = (url, handlers) => new Promise((resolve, reject) => { + const socket = new WebSocket(url); + socket.binaryType = "arraybuffer"; + let opened = false; + let closed = false; + const decoder = new TextDecoder(); + socket.addEventListener("open", () => { + opened = true; + resolve({ send: (text) => socket.send(text), close: () => socket.close() }); + }); + socket.addEventListener("message", (event) => { + const data: unknown = event.data; + if (typeof data === "string") handlers.message(data); + else if (data instanceof ArrayBuffer) handlers.message(decoder.decode(data)); + }); + const end = (): void => { + if (closed) return; + closed = true; + if (!opened) reject(new Error("Browser engine endpoint refused the connection.")); + else handlers.close(); + }; + socket.addEventListener("close", end); + socket.addEventListener("error", end); +}); + +interface PendingCommand { + method: string; + resolve(value: unknown): void; + reject(error: Error): void; + timer: NodeJS.Timeout; +} + +export class CdpCommandError extends Error { + readonly code: number | null; + + constructor(method: string, message: string, code: number | null) { + super(`${method}: ${message}`.slice(0, 400)); + this.name = "CdpCommandError"; + this.code = code; + } +} + +/** + * A tab in a contributed engine, over one CDP WebSocket of its own (engines like Lightpanda serve one page per + * connection). The driver creates the page (`Target.createTarget`), attaches a flat session to it and sends every + * command on that session. It keeps the URL, title and loading state from page events, since `url()` and `title()` + * are synchronous. Nothing here reads or sets cookies: the page starts empty and nothing of the person's browser + * profile reaches it. + */ +export class CdpTabDriver implements TabDriver { + readonly engine: string; + readonly layout: boolean; + private socket: CdpSocket | null = null; + private sessionId: string | null = null; + private targetId: string | null = null; + private nextId = 1; + private readonly pending = new Map(); + private readonly listeners = new Set(); + private readonly commandTimeoutMs: number; + private destroyed = false; + private attached = false; + private mainFrameId: string | null = null; + private currentUrl = "about:blank"; + private currentTitle = ""; + private loadingSince: number | null = null; + + private constructor(options: CdpTabDriverOptions) { + this.engine = options.engine; + this.layout = options.layout; + this.commandTimeoutMs = options.commandTimeoutMs ?? DEFAULT_COMMAND_TIMEOUT_MS; + } + + /** Connects, creates the tab's page and attaches to it. Rejects (and closes) when any step fails or is too slow. */ + static async open(options: CdpTabDriverOptions): Promise { + const url = assertLoopbackCdpUrl(options.url); + const driver = new CdpTabDriver(options); + const connect = options.connect ?? connectWebSocket; + const opening = (async () => { + const socket = await connect(url, { + message: (text) => driver.receive(text), + close: () => driver.closed("engine-closed") + }); + if (driver.destroyed) { + socket.close(); + throw new Error("Browser engine closed the connection."); + } + driver.socket = socket; + const created = await driver.call("Target.createTarget", { url: "about:blank" }, null) as { targetId?: unknown }; + if (typeof created.targetId !== "string" || !created.targetId) throw new Error("Browser engine did not create a page."); + driver.targetId = created.targetId; + const attached = await driver.call("Target.attachToTarget", { targetId: created.targetId, flatten: true }, null) as { + sessionId?: unknown; + }; + if (typeof attached.sessionId !== "string" || !attached.sessionId) throw new Error("Browser engine did not attach to the page."); + driver.sessionId = attached.sessionId; + return driver; + })(); + let timer: NodeJS.Timeout | undefined; + try { + return await Promise.race([ + opening, + new Promise((_resolve, reject) => { + timer = setTimeout(() => reject(new Error("Browser engine did not open a page in time.")), options.openTimeoutMs ?? DEFAULT_OPEN_TIMEOUT_MS); + timer.unref(); + }) + ]); + } catch (error) { + driver.close(); + opening.catch(() => undefined); + throw error; + } finally { + if (timer) clearTimeout(timer); + } + } + + isDestroyed(): boolean { + return this.destroyed; + } + + isAttached(): boolean { + return this.attached && !this.destroyed; + } + + /** The connection is the attachment: nothing more to do than to remember it. */ + attach(): void { + if (this.destroyed) throw new Error("Browser engine tab is closed."); + this.attached = true; + } + + detach(): void { + this.close(); + } + + send(method: string, params?: Record): Promise { + if (!this.sessionId) return Promise.reject(new Error("Browser engine tab is not open.")); + return this.call(method, params ?? {}, this.sessionId); + } + + listen(listeners: TabDriverListeners): () => void { + this.listeners.add(listeners); + return () => this.listeners.delete(listeners); + } + + url(): string { + return this.currentUrl; + } + + title(): string { + return this.currentTitle; + } + + isLoading(): boolean { + return this.loadingSince !== null && Date.now() - this.loadingSince < MAX_LOADING_MS; + } + + /** Loads a URL in the tab. Resolves once the engine accepted the navigation; a refused one rejects. */ + async navigate(url: string): Promise { + this.loadingSince = Date.now(); + try { + const result = await this.send("Page.navigate", { url }) as { errorText?: unknown }; + if (typeof result?.errorText === "string" && result.errorText) throw new Error(result.errorText.slice(0, 200)); + } catch (error) { + this.loadingSince = null; + throw error; + } + } + + /** Back (-1) or forward (+1) in the tab's history; false when there is no such entry. */ + async history(delta: -1 | 1): Promise { + const history = await this.send("Page.getNavigationHistory") as { + currentIndex?: unknown; + entries?: Array<{ id?: unknown }>; + }; + const index = typeof history.currentIndex === "number" ? history.currentIndex + delta : -1; + const entry = Array.isArray(history.entries) ? history.entries[index] : undefined; + if (!entry || typeof entry.id !== "number") return false; + this.loadingSince = Date.now(); + await this.send("Page.navigateToHistoryEntry", { entryId: entry.id }); + return true; + } + + async canGo(delta: -1 | 1): Promise { + try { + const history = await this.send("Page.getNavigationHistory") as { currentIndex?: unknown; entries?: unknown[] }; + const index = typeof history.currentIndex === "number" ? history.currentIndex + delta : -1; + return Array.isArray(history.entries) && index >= 0 && index < history.entries.length; + } catch { + return false; + } + } + + async reload(): Promise { + this.loadingSince = Date.now(); + await this.send("Page.reload"); + } + + /** Closes the page and the connection. Safe to call more than once. */ + close(): void { + if (this.socket && this.targetId && !this.destroyed) { + try { + this.socket.send(JSON.stringify({ id: this.nextId++, method: "Target.closeTarget", params: { targetId: this.targetId } })); + } catch { + // The connection is going away anyway. + } + } + const socket = this.socket; + this.closed("closed"); + try { + socket?.close(); + } catch { + // Already closed. + } + } + + private call(method: string, params: Record, sessionId: string | null): Promise { + if (this.destroyed || !this.socket) return Promise.reject(new Error("Browser engine disconnected.")); + if (this.pending.size >= MAX_PENDING_COMMANDS) return Promise.reject(new Error("Browser engine is busy.")); + const id = this.nextId++; + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { + this.pending.delete(id); + reject(new Error(`${method}: the browser engine did not answer in time.`)); + }, this.commandTimeoutMs); + timer.unref(); + this.pending.set(id, { method, resolve, reject, timer }); + try { + this.socket!.send(JSON.stringify({ id, method, params, ...(sessionId ? { sessionId } : {}) })); + } catch (error) { + clearTimeout(timer); + this.pending.delete(id); + reject(error instanceof Error ? error : new Error(String(error))); + } + }); + } + + private receive(text: string): void { + let message: Record; + try { + const parsed: unknown = JSON.parse(text); + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) return; + message = parsed as Record; + } catch { + return; + } + if (typeof message.id === "number") { + const waiter = this.pending.get(message.id); + if (!waiter) return; + this.pending.delete(message.id); + clearTimeout(waiter.timer); + const error = message.error as { message?: unknown; code?: unknown } | undefined; + if (error && typeof error === "object") { + waiter.reject(new CdpCommandError( + waiter.method, + typeof error.message === "string" ? error.message : "command failed", + typeof error.code === "number" ? error.code : null + )); + } else { + waiter.resolve(message.result ?? {}); + } + return; + } + if (typeof message.method !== "string") return; + const params = message.params; + if (message.method === "Target.detachedFromTarget" || message.method === "Target.targetDestroyed") { + const source = (params ?? {}) as { sessionId?: unknown; targetId?: unknown }; + if (source.sessionId === this.sessionId || source.targetId === this.targetId) this.closed("target-closed"); + return; + } + if (!this.sessionId || message.sessionId !== this.sessionId) return; + this.track(message.method, params); + for (const listener of [...this.listeners]) { + try { + listener.message(message.method, params); + } catch { + // One listener's failure must not stop the others. + } + } + } + + private track(method: string, params: unknown): void { + const source = (params ?? {}) as Record; + if (method === "Page.frameNavigated") { + const frame = (source.frame ?? {}) as { id?: unknown; parentId?: unknown; url?: unknown }; + if (frame.parentId) return; + if (typeof frame.id === "string") this.mainFrameId = frame.id; + if (typeof frame.url === "string") this.currentUrl = frame.url; + this.loadingSince ??= Date.now(); + this.refreshTitle(); + } else if (method === "Page.navigatedWithinDocument") { + if (typeof source.url === "string" && (!this.mainFrameId || source.frameId === this.mainFrameId)) this.currentUrl = source.url; + } else if (method === "Page.frameStartedLoading") { + if (!this.mainFrameId || source.frameId === this.mainFrameId) this.loadingSince ??= Date.now(); + } else if (method === "Page.loadEventFired") { + this.loadingSince = null; + this.refreshTitle(); + } else if (method === "Page.frameStoppedLoading") { + if (this.mainFrameId && source.frameId === this.mainFrameId) this.loadingSince = null; + } else if (method === "Page.domContentEventFired") { + this.refreshTitle(); + } + } + + private refreshTitle(): void { + void this.send("Runtime.evaluate", { expression: "document.title", returnByValue: true, silent: true }).then((value) => { + const title = (value as { result?: { value?: unknown } }).result?.value; + if (typeof title === "string") this.currentTitle = title.slice(0, 1_000); + }, () => undefined); + } + + private closed(reason: string): void { + if (this.destroyed) return; + this.destroyed = true; + this.attached = false; + this.loadingSince = null; + for (const [id, waiter] of this.pending) { + clearTimeout(waiter.timer); + waiter.reject(new Error("Browser engine disconnected.")); + this.pending.delete(id); + } + for (const listener of [...this.listeners]) { + try { + listener.detach(reason); + } catch { + // Ignore listener failures while closing. + } + } + this.listeners.clear(); + } +} diff --git a/src/main/services/browser/TabDriver.ts b/src/main/services/browser/TabDriver.ts new file mode 100644 index 00000000..e1dddafc --- /dev/null +++ b/src/main/services/browser/TabDriver.ts @@ -0,0 +1,88 @@ +import type { WebContents } from "electron"; + +/** CanvasTTY's own engine: the Electron WebContents behind a Browser card tab. */ +export const CHROMIUM_ENGINE = "chromium"; + +export interface TabDriverListeners { + message(method: string, params: unknown): void; + detach(reason: string): void; +} + +/** + * What BrowserAutomationService needs from one tab: a CDP channel, its events, and the page's URL, title and loading + * state. The Electron WebContents is the default driver; a plugin-contributed engine drives a tab over its own local + * CDP endpoint (CdpTabDriver). + */ +export interface TabDriver { + /** `chromium` or the contributed engine's id. */ + readonly engine: string; + /** + * The engine lays pages out for real: element boxes and viewport metrics mean something. An engine without layout + * gets clicks by DOM (`element.click()`), and observation skips the geometry filter. + */ + readonly layout: boolean; + isDestroyed(): boolean; + isAttached(): boolean; + attach(protocolVersion: string): void; + detach(): void; + send(method: string, params?: Record): Promise; + /** Returns the function that removes these listeners. */ + listen(listeners: TabDriverListeners): () => void; + url(): string; + title(): string; + isLoading(): boolean; +} + +/** The Electron tab: every call goes to the WebContents and its debugger exactly as before drivers existed. */ +export class ElectronTabDriver implements TabDriver { + readonly engine = CHROMIUM_ENGINE; + readonly layout = true; + readonly contents: WebContents; + + constructor(contents: WebContents) { + this.contents = contents; + } + + isDestroyed(): boolean { + return this.contents.isDestroyed(); + } + + isAttached(): boolean { + return this.contents.debugger.isAttached(); + } + + attach(protocolVersion: string): void { + this.contents.debugger.attach(protocolVersion); + } + + detach(): void { + this.contents.debugger.detach(); + } + + send(method: string, params?: Record): Promise { + return this.contents.debugger.sendCommand(method, params); + } + + listen(listeners: TabDriverListeners): () => void { + const message = (_event: unknown, method: string, params: unknown): void => listeners.message(method, params); + const detach = (_event: unknown, reason: string): void => listeners.detach(reason); + this.contents.debugger.on("message", message); + this.contents.debugger.on("detach", detach); + return () => { + this.contents.debugger.removeListener("message", message); + this.contents.debugger.removeListener("detach", detach); + }; + } + + url(): string { + return this.contents.getURL(); + } + + title(): string { + return this.contents.getTitle(); + } + + isLoading(): boolean { + return this.contents.isLoading(); + } +} diff --git a/src/main/services/isolation/AgentIsolation.ts b/src/main/services/isolation/AgentIsolation.ts index c18cc54b..5d648c29 100644 --- a/src/main/services/isolation/AgentIsolation.ts +++ b/src/main/services/isolation/AgentIsolation.ts @@ -1,3 +1,4 @@ +import { spawnSync } from "node:child_process"; import { chmodSync, existsSync, mkdirSync, mkdtempSync, readdirSync, realpathSync, rmdirSync, rmSync, statSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { delimiter, dirname, join } from "node:path"; @@ -14,6 +15,10 @@ export const SANDBOX_EXEC = "/usr/bin/sandbox-exec"; export const ISOLATION_FOLDER_PREFIX = "ctty-iso-"; /** Tells the agent inside the layer what it may do (its value is ISOLATION_NOTE). */ export const ISOLATION_ENV = "CANVASTTY_ISOLATION"; +/** Where the docs say how to let bubblewrap create its namespaces (Ubuntu 24.04 and later restrict them). */ +export const BUBBLEWRAP_USERNS_DOCS = "docs/installing-and-security.md#linux-when-bubblewrap-cannot-start"; +/** A failed bubblewrap check is repeated after this long, so allowing it takes effect without a restart. */ +const BUBBLEWRAP_PROBE_RETRY_MS = 60_000; export const ISOLATION_NOTE = "CanvasTTY agent isolation: files can be written only inside the project folder, $TMPDIR and this CLI's own folders; SSH/cloud keys, other agents' credentials and CanvasTTY's tokens cannot be read; other processes, apps and daemons are out of reach. \"Operation not permitted\" outside that is this rule: do the work inside the project, or tell the person what you need."; export interface AgentIsolationOptions { @@ -32,6 +37,13 @@ export interface AgentIsolationOptions { exists?: (path: string) => boolean; /** Linux: the host folders and placeholders bubblewrap needs (shared across launches); tests pass their own. */ linuxHostPaths?: LinuxHostPaths; + /** + * Linux: runs bubblewrap once with the namespaces a launch uses and returns null when it works, or what it said. + * bubblewrap can be installed and still unable to create an unprivileged user namespace (Ubuntu 24.04's AppArmor + * `kernel.apparmor_restrict_unprivileged_userns=1`). Tests pass their own. + */ + bubblewrapProbe?: (bwrap: string) => string | null; + now?: () => number; } export interface IsolationDecisionInput { @@ -87,6 +99,8 @@ export class AgentIsolation { private readonly hostEnvironment: Readonly>; private bubblewrap: string | null | undefined; private readonly linuxHostPaths: LinuxHostPaths; + /** The last bubblewrap check: a working one is kept, a failed one is repeated after BUBBLEWRAP_PROBE_RETRY_MS. */ + private bubblewrapCheck: { path: string; failure: string | null; at: number } | null = null; constructor(options: AgentIsolationOptions) { this.options = options; @@ -106,7 +120,11 @@ export class AgentIsolation { } if (this.platform === "linux") { if (this.bubblewrap === undefined) this.bubblewrap = findOnPath("bwrap", exists); - return this.bubblewrap ? { layer: "bubblewrap" } : { reason: "bubblewrap (bwrap) is not installed; install it to isolate agents on Linux." }; + if (!this.bubblewrap) return { reason: "bubblewrap (bwrap) is not installed; install it to isolate agents on Linux." }; + const failure = this.bubblewrapFailure(this.bubblewrap); + return failure === null + ? { layer: "bubblewrap" } + : { reason: `bubblewrap (bwrap) is installed but cannot create its sandbox here (${failure}); Ubuntu 24.04 and later block unprivileged user namespaces through AppArmor. ${BUBBLEWRAP_USERNS_DOCS} says how to allow it.` }; } if (this.platform === "win32") return { reason: "CanvasTTY has no agent isolation layer on Windows yet." }; return { reason: `CanvasTTY has no agent isolation layer on ${this.platform}.` }; @@ -214,6 +232,17 @@ export class AgentIsolation { } } + /** Null when bubblewrap can start here; cached, so a launch costs one check at most once a minute. */ + private bubblewrapFailure(bwrap: string): string | null { + const now = (this.options.now ?? Date.now)(); + const last = this.bubblewrapCheck; + if (last && last.path === bwrap && (last.failure === null || now - last.at < BUBBLEWRAP_PROBE_RETRY_MS)) return last.failure; + let failure: string | null; + try { failure = (this.options.bubblewrapProbe ?? probeBubblewrap)(bwrap); } catch (error) { failure = error instanceof Error ? error.message : String(error); } + this.bubblewrapCheck = { path: bwrap, failure, at: now }; + return failure; + } + private enabled(): boolean { try { return this.options.enabled() !== false; } catch { return true; } } @@ -230,6 +259,19 @@ function removeMountPoint(hooks: string): void { } catch { /* not empty, or already gone */ } } +/** Starts `true` under bubblewrap with the namespaces a launch gets: null when it runs, else bubblewrap's first line. */ +export function probeBubblewrap(bwrap: string): string | null { + const result = spawnSync(bwrap, ["--die-with-parent", "--unshare-pid", "--unshare-ipc", "--ro-bind", "/", "/", "--dev", "/dev", "--proc", "/proc", "true"], { + stdio: ["ignore", "ignore", "pipe"], + encoding: "utf8", + timeout: 5_000 + }); + if (result.status === 0) return null; + const said = (result.stderr ?? "").split("\n").map((line) => line.trim()).find(Boolean); + const detail = said ?? result.error?.message ?? (result.signal ? `stopped by ${result.signal}` : `exit code ${String(result.status)}`); + return detail.length > 200 ? `${detail.slice(0, 199)}…` : detail; +} + function findOnPath(name: string, exists: (path: string) => boolean): string | null { for (const folder of (process.env.PATH ?? "").split(delimiter)) { if (!folder) continue; diff --git a/src/renderer/src/App.tsx b/src/renderer/src/App.tsx index 6d36491a..c7b4a429 100644 --- a/src/renderer/src/App.tsx +++ b/src/renderer/src/App.tsx @@ -153,6 +153,7 @@ const FALLBACK_SETTINGS: AppSettings = { browserAgentAccess: true, browserShowAgentPresence: true, browserRestoreTabs: true, + browserPauseHiddenTabs: true, attentionNotifications: true, attentionQueueVisible: true, attentionQueuePlacement: "bottom-right", diff --git a/src/renderer/src/features/browser/BrowserCard.tsx b/src/renderer/src/features/browser/BrowserCard.tsx index 4a1f80a9..457a772e 100644 --- a/src/renderer/src/features/browser/BrowserCard.tsx +++ b/src/renderer/src/features/browser/BrowserCard.tsx @@ -127,6 +127,8 @@ export function BrowserCard({ const freezeFrameDataUrl = freezeFrame && freezeFrame.tabId === activeTab?.id ? freezeFrame.dataUrl : null; + // A sleeping tab has no page behind the card: its last picture stands in until it reloads. + const sleepPreview = activeTab?.lifecycle === "sleeping" ? safeFavicon(activeTab.preview ?? null) : null; useEffect(() => { liveBounds.current = bounds; @@ -456,6 +458,11 @@ export function BrowserCard({ > {tab.title || t(locale, "newTab")} + {tab.lifecycle && ( + + {t(locale, tab.lifecycle === "paused" ? "browserTabPaused" : "browserTabSleeping")} + + )} {showAgentPresence && }