diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f906e9..1c59b25 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht ### Added +- **Browser-WebSocket transport: attach to a Chrome that serves no `/json`.** A Chrome that had debugging turned on at runtime, through the toggle on `chrome://inspect/#remote-debugging`, serves only the browser-level socket — measured on Chrome 153, `/devtools/browser/` returns 101 while `/devtools/page/` returns 403 and `/json/version`+`/json/list` both return 404. That is the transport a user gets when they attach to the browser they were already using rather than relaunching it, and the toolkit could not reach it at all. Discovery now falls back to `Target.getTargets` over that one socket, and each call reaches its page through a flat `Target.attachToTarget` session on it. **The HTTP path is byte-for-byte unchanged**: `listTargets` issues its original `GET /json/list` first and unconditionally, and only a request that did not answer reaches the fallback — no added round-trip, no added failure mode, and the transport is cached after the first listing. The one-target-per-call guarantee holds: `Target.getTargets` is metadata-only and attaches to nothing, so a browser with hundreds of tabs costs one round trip rather than one attach per tab. Nothing is auto-discovered — a runtime-toggled Chrome raises a modal consent prompt for each new client, so the browser is named explicitly with `CDP_BROWSER_WS` (the socket URL) or `CDP_USER_DATA_DIR` (the profile whose `DevToolsActivePort` holds it), and the default profile is never scanned. (`src/cdp/endpoint.ts`, `src/cdp/session.ts`, `src/client.ts`, `src/cdp/driver.ts`) +- **`bun run browser-ws:smoke` and `bun run browser-ws:wedge`.** Both spawn their own disposable headless Chrome and front it with a proxy that removes exactly what the runtime-toggle mode removes (`test/browser-ws-only-proxy.ts`), because that mode cannot be requested with a flag and puts a consent dialog in front of every client. The smoke drives discovery and per-tab work against a confirmed `/json/version=404 /json/list=404 page-ws=403` endpoint, with every phase capped at the real 60s MCP tool timeout: at `TABS=200` it listed 201 pages (410 targets across 6 types) in 0.08s, evaluated in 0.09s, snapshotted in 0.03s, and drove 5 consecutive tabs in 0.03s. The wedge proves the timeout bound survives socket sharing, which `bench:wedge` cannot — that benchmark gives every page its own socket, where a hung renderer can only block the one socket dialed into it, while this transport puts every page on ONE socket. Observed against a 5s bound: the blackholed tab rejected at 5.00s while a witness tab, `list_pages`, and a freshly opened tab all answered in under 1ms on the same socket. (`test/browser-ws-smoke.ts`, `test/browser-ws-wedge.ts`) - **Wedge benchmark (`bun run bench:wedge`).** A runnable, CI-able proof of the core claim: an isolated headless Chrome drives a throwaway page into an endpoint that accepts the connection and never responds, 8 times, and measures the whole stuck-page lifecycle — the wedged `navigate` rejects at the configured bound (15.03s observed against the default 15s bound), a witness tab keeps answering at healthy latency mid-brick (p95 3ms), and closing the bricked tab plus reopening evaluates at p95 89ms with zero `/mcp` restarts. Also reports, as an observation rather than a failure, that the stuck page itself stays unresponsive while its load is pending: the blast radius is the tab, not the server. (`scripts/wedge-bench.ts`) ## [2.7.0] - 2026-09-15 diff --git a/README.md b/README.md index 161de6f..b12e8f1 100644 --- a/README.md +++ b/README.md @@ -235,6 +235,8 @@ await TOOLS.navigate_page({ target: "index:0", url: "https://example.com" }); |---|---|---| | `CDP_BASE` | `http://127.0.0.1:9222` | DevTools HTTP origin (drives discovery + the lighthouse `--port`). | | `CDP_TIMEOUT_MS` | `15000` | Per-command timeout. | +| `CDP_BROWSER_WS` | unset | The `ws://…/devtools/browser/` URL, for a Chrome that serves no `/json`. Only consulted after an HTTP listing has already failed. See "Attaching to a Chrome that serves no `/json`" below. | +| `CDP_USER_DATA_DIR` | unset | Profile directory to read `DevToolsActivePort` from, to find that browser socket without naming its uuid. Opt-in on purpose: nothing is scanned unless you name it. | | `CDP_ARTIFACT_DIR` | `/tmp/cdp-toolkit` | Screenshots, traces, heap snapshots, lighthouse reports, recorder buffers. | | `CDP_STATE_DIR` | `/tmp/cdp-toolkit` | `select_page` selected-target file, in-flight trace state. | | `CDP_EXTRACT_BASE_URL` | `http://127.0.0.1:8090/v1` | `extract_page` only. Base URL of the OpenAI-compatible extraction endpoint (the server appends `/chat/completions`). The default is loopback llm-ferry, so page content never leaves the machine; overriding it sends page content wherever the operator points. | @@ -317,6 +319,42 @@ The server serves **both** eras off the same stdio transport, and the era is pin Measured 2026-09-01: Claude Code 2.1.258 and Codex CLI 0.147.0 both connect and both pin the **legacy** era — their binaries carry the 2026-07-28 client strings, but neither opens with `server/discover` by default, and the MCP SDK's own `Client` behaves the same unless it opts in. So serving both eras is load-bearing, not courtesy, and today the `ttlMs`/`cacheScope` hints reach only clients that ask for the modern era. The static listing pays off on either one — no mid-session tool churn, and no prompt-cache invalidation. +## Attaching to a Chrome that serves no `/json` + +Chrome has two ways in, and until recently every build served both: (A) the HTTP discovery endpoints `/json/version` and `/json/list`, plus a per-target socket at `/devtools/page/`; and (B) one browser-level socket at `/devtools/browser/`, over which `Target.getTargets` lists and `Target.attachToTarget` reaches a page. + +A Chrome started with `--remote-debugging-port` serves both, and nothing below is reached on one. A Chrome that had debugging turned on **at runtime**, through the toggle on `chrome://inspect/#remote-debugging`, serves only (B) — measured on Chrome 153: + +| Request | `--remote-debugging-port` Chrome | runtime-toggled Chrome | +|---|---|---| +| `ws://127.0.0.1:/devtools/browser/` | 101 | **101** | +| `ws://127.0.0.1:/devtools/page/` | 101 | **403** | +| `http://127.0.0.1:/json/version`, `/json/list` | 200 | **404** | + +That is the transport a user gets when they attach to the browser they were *already using* rather than relaunching it — a logged-in profile with their real tabs — so it is worth supporting rather than working around. cdp-toolkit reaches it with `Target.getTargets` for discovery and one flat `Target.attachToTarget` session per call for the work, over a single refcounted browser socket. + +**The HTTP path is untouched.** `list_pages` issues its original `GET /json/list` first and unconditionally; only a request that did *not* answer reaches the fallback. On a Chrome that serves (A) the code path is the one that shipped, with no added round-trip and no added failure mode. The transport is then cached, so the 404 is paid once per process. + +**Nothing is auto-discovered — you name the browser.** Connecting to a DevTools endpoint is not a neutral act: a runtime-toggled Chrome raises a modal *"Allow remote debugging?"* prompt on the user's screen for each new client, and that prompt grants full access to a logged-in session. So cdp-toolkit never scans the default profile looking for a browser socket. Point it at one explicitly: + +```bash +# the browser socket directly (its uuid is minted fresh on every Chrome start) +CDP_BROWSER_WS=ws://127.0.0.1:9222/devtools/browser/ cdp list_pages + +# or name the profile, and the uuid is read from its DevToolsActivePort file +CDP_USER_DATA_DIR="$HOME/Library/Application Support/Google/Chrome" cdp list_pages +``` + +**The one-target-per-call guarantee still holds.** Discovery is one `Target.getTargets` — metadata only, it attaches to nothing — so a browser with hundreds of tabs costs one round trip, not one attach per tab. Each call then attaches to the one target it named and detaches on close. Every command still carries `CDP_TIMEOUT_MS`, per command id rather than per socket, so a wedged tab rejects at the bound and the shared socket stays clear for everyone else. `bun run browser-ws:wedge` proves exactly that: it blackholes one tab, then measures a witness tab, `list_pages`, and a freshly opened tab on the same socket. + +```bash +bun run browser-ws:smoke # drives discovery + per-tab work on a browser-ws-only endpoint +TABS=200 bun run browser-ws:smoke +bun run browser-ws:wedge # one stuck tab must not wedge the shared socket +``` + +Both spawn their own disposable headless Chrome and front it with a proxy that removes exactly what the runtime-toggle mode removes (`test/browser-ws-only-proxy.ts`), because the real mode cannot be requested with a flag and puts a consent dialog in front of every client. They never look for a Chrome you are already running. + ## Structured page extraction (`extract_page`) `extract_page` turns a page into **schema-conformant JSON** instead of prose: it grabs the target page's HTML, cleans it (scripts, styles, hidden elements and other non-content noise stripped), and sends it to an OpenAI-compatible `/chat/completions` endpoint along with your JSON Schema, returning JSON that matches the schema — fields you name, not a paragraph you have to parse. A `selector` scopes extraction to one subtree (piercing open shadow roots), so you can extract the article or the table rather than the whole page. diff --git a/package.json b/package.json index 49ef4f4..7966872 100644 --- a/package.json +++ b/package.json @@ -88,7 +88,9 @@ "input:smoke": "bun run ./test/input-smoke.ts", "extension:smoke": "bun run ./test/extension-smoke.ts", "focus:smoke": "bun run ./test/focus-emulation-smoke.ts", - "extract:smoke": "bun run ./test/extract-smoke.ts" + "extract:smoke": "bun run ./test/extract-smoke.ts", + "browser-ws:smoke": "bun run ./test/browser-ws-smoke.ts", + "browser-ws:wedge": "bun run ./test/browser-ws-wedge.ts" }, "devDependencies": { "@modelcontextprotocol/client": "^2.0.0", diff --git a/src/cdp/driver.ts b/src/cdp/driver.ts index 8db15a0..e198a7f 100644 --- a/src/cdp/driver.ts +++ b/src/cdp/driver.ts @@ -5,7 +5,8 @@ * one connection via openPage and holds it for the PageDriver's life; release() closes it. Uid * codec (scheme "cdp"): `cdp:`, or a bare legacy numeric uid, per THE UID CODEC block in ../driver.ts. */ -import { CdpConnection, CdpError, listTargets, openBrowser, openPage, resolveTarget } from "../client.ts"; +import { CdpConnection, CdpError, listTargets, openBrowser, openPage, resolveTarget, type PageConnection } from "../client.ts"; +import { openSession } from "./session.ts"; import type { Target, TargetSelector } from "../types.ts"; import { LeaseConflictError } from "../leases.ts"; import { BEACON_READ_EXPRESSION, BEACON_SOURCE, BeaconSessions, RENDERER_PROBE_EXPRESSION, RENDERER_PROBE_TIMEOUT_MS, sendInput } from "../activity.ts"; @@ -81,7 +82,7 @@ export function cdpSameSite(sameSite?: "strict" | "lax" | "none" | "default"): " /* ---------------------- copied helpers, see attributions below ---------------------- */ // Copied verbatim from src/tools/input.ts `centerOf`: scroll into view, read viewport-space center. -async function centerOf(conn: CdpConnection, objectId: string): Promise<{ x: number; y: number }> { +async function centerOf(conn: PageConnection, objectId: string): Promise<{ x: number; y: number }> { const fn = "function(){this.scrollIntoView({block:'center',inline:'center'});const r=this.getBoundingClientRect();if(r.width===0&&r.height===0)return null;return {x:r.left+r.width/2,y:r.top+r.height/2};}"; const { result, exceptionDetails } = await conn.send<{ result: { value?: { x: number; y: number } | null }; exceptionDetails?: { text?: string } }>( "Runtime.callFunctionOn", { objectId, functionDeclaration: fn, returnByValue: true }, @@ -90,7 +91,7 @@ async function centerOf(conn: CdpConnection, objectId: string): Promise<{ x: num if (!result.value) throw driverError("page-error", "element has zero size / is not visible; cannot compute a click point"); return result.value; } -async function focusElement(conn: CdpConnection, objectId: string): Promise { +async function focusElement(conn: PageConnection, objectId: string): Promise { await conn.send("Runtime.callFunctionOn", { objectId, functionDeclaration: "function(){this.focus&&this.focus();}", returnByValue: true }); } /** @@ -112,10 +113,10 @@ async function focusElement(conn: CdpConnection, objectId: string): Promise{clearTimeout(t);t=setTimeout(done,60);};window.addEventListener('scroll',on,true);t=setTimeout(done,60);setTimeout(done,500);});})()"; -async function armScrollSettleWatch(conn: CdpConnection): Promise { +async function armScrollSettleWatch(conn: PageConnection): Promise { await conn.send("Runtime.evaluate", { expression: ARM_SCROLL_SETTLE_WATCH, returnByValue: true }); } -async function awaitScrollSettle(conn: CdpConnection): Promise { +async function awaitScrollSettle(conn: PageConnection): Promise { // Best-effort: a navigation racing the wheel event could tear down window.__cdpScrollSettle // before this reads it, and that is not a scroll failure worth surfacing as one. await conn.send("Runtime.evaluate", { expression: "window.__cdpScrollSettle", awaitPromise: true, returnByValue: true }).catch(() => undefined); @@ -129,7 +130,7 @@ async function awaitScrollSettle(conn: CdpConnection): Promise { // It is a parameter rather than a `conn.send` here because the choke point is per-PAGE // (it knows the target id) and this helper only has a connection. async function setValueOnObject( - conn: CdpConnection, + conn: PageConnection, objectId: string, value: string, sendInput: (method: string, params: Record) => Promise, @@ -245,7 +246,7 @@ function axExtras(node: AxNode): Record { } /* ------------------------------- locator helpers ------------------------------- */ // css branch copied from src/tools/input.ts `resolveElement`; uid/text/xpath branches are new. -async function resolveElementLocator(conn: CdpConnection, loc: ElementLocator): Promise<{ objectId: string }> { +async function resolveElementLocator(conn: PageConnection, loc: ElementLocator): Promise<{ objectId: string }> { if ("uid" in loc) { return resolveUid(conn, decodeUid(loc.uid)).catch(() => { throw driverError("stale-uid", `uid does not resolve: ${loc.uid}`); @@ -270,7 +271,7 @@ async function resolveElementLocator(conn: CdpConnection, loc: ElementLocator): * XPath expression and disambiguates them itself. Not copied from any tools/ module (none * implement this); built only on the client primitives per CONTRACT.md rule 2. */ -async function searchLocate(conn: CdpConnection, query: string): Promise { +async function searchLocate(conn: PageConnection, query: string): Promise { await conn.send("DOM.getDocument", { depth: 0 }).catch(() => undefined); const { searchId, resultCount } = await conn.send<{ searchId: string; resultCount: number }>("DOM.performSearch", { query }); try { @@ -285,7 +286,7 @@ async function searchLocate(conn: CdpConnection, query: string): Promise } } -async function backendNodeIdOf(conn: CdpConnection, loc: ElementLocator): Promise { +async function backendNodeIdOf(conn: PageConnection, loc: ElementLocator): Promise { if ("uid" in loc) return decodeUid(loc.uid); if ("css" in loc) { const { objectId } = await resolveElementLocator(conn, loc); @@ -323,7 +324,7 @@ interface LayoutMetrics { cssVisualViewport?: { pageX?: number; pageY?: number; clientWidth?: number; clientHeight?: number }; cssContentSize?: { width: number; height: number }; } -async function layoutMetrics(conn: CdpConnection): Promise { +async function layoutMetrics(conn: PageConnection): Promise { return conn.send("Page.getLayoutMetrics"); } /** @@ -554,7 +555,7 @@ export function planTiledCapture( * document-frame correction below is what buys correctness. Keeping it also means the offset this * scroll CREATES must be read after it settles — see capture()'s ordering. */ -async function elementBox(conn: CdpConnection, loc: ElementLocator): Promise<{ x: number; y: number; width: number; height: number }> { +async function elementBox(conn: PageConnection, loc: ElementLocator): Promise<{ x: number; y: number; width: number; height: number }> { const backendNodeId = await backendNodeIdOf(conn, loc); await conn.send("DOM.scrollIntoViewIfNeeded", { backendNodeId }).catch(() => undefined); const box = await conn.send<{ model: { content: number[] } }>("DOM.getBoxModel", { backendNodeId }); @@ -663,7 +664,7 @@ class CdpPageDriver implements PageDriver { // side effect) it did not have before this migration. See the per-method comments below. private readonly enabledDomains = new Set(); private released = false; - constructor(private readonly conn: CdpConnection, target: Target, readonly browser: BrowserDriver) { + constructor(private readonly conn: PageConnection, target: Target, readonly browser: BrowserDriver) { this.info = { id: target.id, url: target.url, title: target.title, type: target.type }; } private async ensureDomain(name: string): Promise { @@ -1595,12 +1596,24 @@ class CdpPageDriver implements PageDriver { * process, which degrades the beacon to current-document-only — correct, since * a CLI process cannot hold anything across calls by construction. */ -const beaconSessions = new BeaconSessions((conn) => conn.close()); +const beaconSessions = new BeaconSessions((conn) => conn.close()); -/** The page endpoint for a target id, or undefined if the browser no longer has it. */ -async function targetWsUrl(targetId: string): Promise { +/** + * A connection to one target by id, or undefined if the browser no longer has + * it. The beacon paths' counterpart of client.ts's `openPage`, and deliberately + * NOT that function: these three callers are gate-free by contract (see + * ../driver.ts) and must not go through the lease gate `resolveTarget` applies. + * + * Transport-routed like `openPage`, for the same reason: on a browser-ws + * endpoint the listing carries no per-target URL to dial. + */ +async function openTargetConn(targetId: string, opts: { timeoutMs?: number } = {}): Promise { const hit = (await listTargets()).find((t) => t.id === targetId); - return hit?.webSocketDebuggerUrl; + if (!hit) return undefined; + // Same marker openPage reads: no per-target URL means this endpoint serves + // none, so the target is reached by session instead. + if (!hit.webSocketDebuggerUrl) return openSession(targetId, opts).catch(() => undefined); + return new CdpConnection(hit.webSocketDebuggerUrl, opts).connect().catch(() => undefined); } /** @@ -1612,7 +1625,7 @@ async function targetWsUrl(targetId: string): Promise { * its tab look exactly like a live tab nobody has touched, and the held session * would never be dropped or retried. */ -async function readBeaconOver(conn: CdpConnection): Promise { +async function readBeaconOver(conn: PageConnection): Promise { try { const res = await conn.send<{ result: { value?: unknown } }>("Runtime.evaluate", { expression: BEACON_READ_EXPRESSION, @@ -1634,7 +1647,7 @@ async function readBeaconOver(conn: CdpConnection): Promise { +async function probeOver(conn: PageConnection, timeoutMs: number): Promise<{ responsive: boolean; beaconTs: number | null }> { try { const res = await conn.send<{ result: { value?: unknown } }>( "Runtime.evaluate", @@ -1783,9 +1796,7 @@ class CdpBrowserDriver implements BrowserDriver { // build a fresh one rather than reporting a beacon that is not there. beaconSessions.drop(targetId); } - const wsUrl = await targetWsUrl(targetId).catch(() => undefined); - if (!wsUrl) return false; - const conn = await new CdpConnection(wsUrl).connect().catch(() => undefined); + const conn = await openTargetConn(targetId).catch(() => undefined); if (!conn) return false; try { await conn.send("Page.enable"); @@ -1816,9 +1827,7 @@ class CdpBrowserDriver implements BrowserDriver { if (value !== undefined) return value; beaconSessions.drop(targetId); } - const wsUrl = await targetWsUrl(targetId).catch(() => undefined); - if (!wsUrl) return null; - const conn = await new CdpConnection(wsUrl).connect().catch(() => undefined); + const conn = await openTargetConn(targetId).catch(() => undefined); if (!conn) return null; try { return (await readBeaconOver(conn)) ?? null; @@ -1844,14 +1853,8 @@ class CdpBrowserDriver implements BrowserDriver { async probeRenderer(targetId: string, timeoutMs: number = RENDERER_PROBE_TIMEOUT_MS): Promise<{ responsive: boolean; beaconTs: number | null }> { const held = beaconSessions.get(targetId); if (held) return probeOver(held, timeoutMs); - const wsUrl = await targetWsUrl(targetId).catch(() => undefined); - if (!wsUrl) return { responsive: false, beaconTs: null }; - const conn = new CdpConnection(wsUrl, { timeoutMs }); - try { - await conn.connect(); - } catch { - return { responsive: false, beaconTs: null }; - } + const conn = await openTargetConn(targetId, { timeoutMs }).catch(() => undefined); + if (!conn) return { responsive: false, beaconTs: null }; try { return await probeOver(conn, timeoutMs); } finally { diff --git a/src/cdp/endpoint.ts b/src/cdp/endpoint.ts new file mode 100644 index 0000000..b4dd27c --- /dev/null +++ b/src/cdp/endpoint.ts @@ -0,0 +1,176 @@ +/** + * Where the DevTools endpoint is, and which of its two transports this Chrome + * actually serves. + * + * THE FACT THIS MODULE EXISTS FOR. Chrome has always had two ways in, and + * until recently every build served both: + * + * A. the HTTP discovery endpoints — `/json/version`, `/json/list` — plus a + * per-target socket at `/devtools/page/`; + * B. one browser-level socket at `/devtools/browser/`, over which + * `Target.getTargets` lists and `Target.attachToTarget` reaches a page. + * + * A Chrome started with `--remote-debugging-port` serves both. A Chrome that + * had debugging turned on at RUNTIME, through the toggle on + * `chrome://inspect/#remote-debugging`, serves only (B) — measured on Chrome + * 153 (the same handshake on a `--remote-debugging-port` instance returns 101 + * for all three): + * + * ws://127.0.0.1:/devtools/browser/ -> HTTP 101 + * ws://127.0.0.1:/devtools/page/ -> HTTP 403 + * http://127.0.0.1:/json/version, /json/list -> HTTP 404 + * + * That is not a degraded mode to work around, it is the transport a user gets + * when they attach to the browser they were already using rather than + * relaunching it. Everything the toolkit needs is reachable over (B). + * + * WHY DETECTED AND NOT CONFIGURED. Making the user declare the transport would + * make them diagnose a 404 first. The detection is cached per process. + * + * WHY THE FALLBACK IS DRIVEN BY A FAILED OPERATION, not by a probe that runs + * first. `listTargets` still issues its original `/json/list` request before + * anything here is consulted, and only a request that did NOT answer reaches + * the fallback. So on a Chrome that serves (A) the code path is the one that + * shipped, byte for byte, and the new transport cannot regress it — there is + * no added round-trip, no added failure mode, and no ordering question. The + * cost of (B) is one 404 on the first listing, then the decision is cached. + * + * WHY THE PORT FILE. The browser socket's URL contains a uuid that is minted + * fresh on every Chrome start, so it cannot be written into a config once. The + * uuid and the port are the two lines of `DevToolsActivePort` in the profile + * directory, which is where `/json/version` would otherwise have supplied it. + * Resolved at connect time for that reason, never memoized across a restart. + */ +import { readFile } from "node:fs/promises"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +/** Base HTTP origin of the DevTools endpoint. Override with CDP_BASE. */ +export const BASE = process.env.CDP_BASE ?? "http://127.0.0.1:9222"; + +/** + * Which transport this endpoint serves. + * "http" — `/json/*` answers; per-target sockets exist. The original path. + * "browser-ws" — only the browser socket answers; pages are reached by session. + */ +export type CdpTransport = "http" | "browser-ws"; + +export interface EndpointInfo { + transport: CdpTransport; + /** The browser-level WebSocket URL. Always present: both transports have one. */ + browserWsUrl: string; +} + +/** + * The browser socket URL from a profile's `DevToolsActivePort`. + * + * The file is exactly two lines — port, then the path component including the + * uuid — and Chrome rewrites it on every start. A profile that is not running + * leaves a STALE file behind rather than deleting it, so a URL from here is a + * candidate to be handshaked, never a fact; `detectEndpoint` treats a failed + * connect as "not this profile" and moves on. + */ +async function browserWsFromPortFile(dir: string): Promise { + const raw = await readFile(join(dir, "DevToolsActivePort"), "utf8").catch(() => undefined); + if (!raw) return undefined; + const [port, path] = raw.split("\n"); + if (!port?.trim() || !path?.trim()) return undefined; + const host = new URL(BASE).hostname; + return `ws://${host}:${port.trim()}${path.trim()}`; +} + +/** + * The browser socket for an explicitly named profile, and ONLY an explicitly + * named one. + * + * NEVER SCANS THE DEFAULT PROFILE, and that restraint is the point rather than + * an omission. Connecting to a DevTools endpoint is not a neutral act: a Chrome + * with runtime debugging enabled raises a modal "Allow remote debugging?" + * consent prompt on the user's screen for each new client, and that prompt + * grants full access to a logged-in browsing session. A library that guesses + * its way to a browser nobody named would raise that prompt from a unit-test + * run — which is exactly how this restriction was found, by doing it. + * + * So the profile is opt-in: `CDP_USER_DATA_DIR` names it, or nothing is tried. + * Pointing this toolkit at a browser stays a decision the user makes. + */ +async function discoverBrowserWs(): Promise { + const dir = process.env.CDP_USER_DATA_DIR; + return dir ? browserWsFromPortFile(dir) : undefined; +} + +/** + * A bounded WebSocket handshake. The port file can name a Chrome that exited, + * and a dead port would otherwise stall the whole detection on a connect that + * never resolves. + */ +function handshake(url: string, timeoutMs: number): Promise { + return new Promise((resolve) => { + let settled = false; + const done = (ok: boolean) => { + if (settled) return; + settled = true; + try { + ws.close(); + } catch { + /* ignore */ + } + resolve(ok); + }; + const ws = new WebSocket(url); + ws.onopen = () => done(true); + ws.onerror = () => done(false); + setTimeout(() => done(false), timeoutMs); + }); +} + +let cached: Promise | undefined; + +/** + * Locate the browser socket for an endpoint that serves no `/json`. + * + * Called only after an HTTP request has already failed, so it never runs at all + * against a Chrome started with `--remote-debugging-port`. `CDP_BROWSER_WS` + * short-circuits the profile lookup, for an endpoint the port file cannot name + * (a container, a forwarded port, a profile outside the defaults). + */ +export function detectEndpoint(): Promise { + // Only a resolved detection is worth keeping. A failure here is usually + // transient — Chrome still starting, a port file mid-rewrite — and caching + // the rejection would make the first unlucky call poison every later one. + return (cached ??= (async (): Promise => { + const forced = process.env.CDP_BROWSER_WS; + if (forced) return { transport: "browser-ws", browserWsUrl: forced }; + + const discovered = await discoverBrowserWs(); + if (discovered && (await handshake(discovered, 5_000))) { + return { transport: "browser-ws", browserWsUrl: discovered }; + } + + // Bounded for the same reason the handshake above is: a host that accepts + // the connection and never answers would otherwise stall detection forever. + const version = await fetch(`${BASE}/json/version`, { signal: AbortSignal.timeout(5_000) }) + .then((r) => (r.ok ? (r.json() as Promise<{ webSocketDebuggerUrl?: string }>) : undefined)) + .catch(() => undefined); + if (version?.webSocketDebuggerUrl) { + return { transport: "http", browserWsUrl: version.webSocketDebuggerUrl }; + } + + const dir = process.env.CDP_USER_DATA_DIR; + throw new Error( + `no DevTools endpoint at ${BASE}: GET /json/version did not answer and ` + + (dir + ? `no live browser socket was named by ${dir}/DevToolsActivePort. ` + : `no profile was searched — CDP_USER_DATA_DIR is unset, and no profile is ever scanned unless you name one. `) + + `Set CDP_BROWSER_WS to the ws://.../devtools/browser/ URL, or CDP_USER_DATA_DIR to the profile in use.`, + ); + })().catch((error: unknown) => { + cached = undefined; + throw error; + })); +} + +/** Test seam: drop the per-process detection cache. */ +export function resetEndpointCache(): void { + cached = undefined; +} diff --git a/src/cdp/session.ts b/src/cdp/session.ts new file mode 100644 index 0000000..caa1b1a --- /dev/null +++ b/src/cdp/session.ts @@ -0,0 +1,233 @@ +/** + * A page connection carried by a flat `Target.attachToTarget` session on the + * browser socket, for the endpoints that serve no per-target socket (see + * ./endpoint.ts for why those exist). + * + * WHAT THIS IS. `SessionConnection` presents the same surface as + * `CdpConnection` — `send`, `on`, `waitFor`, `close`, with the same + * per-command timeout — but instead of owning a socket it owns a sessionId and + * borrows the shared browser socket. Every tool module in the toolkit is typed + * against that surface, so none of them has to know which one it holds. The + * one-target-per-call guarantee is unchanged: a call attaches to the ONE + * target it named and detaches when it is done. + * + * WHY FLAT SESSIONS. With `flatten: true` every message for an attached target + * carries its `sessionId` on the same socket, so routing is a map lookup and + * two sessions never see each other's events. The nested alternative wraps + * each message in `Target.receivedMessageFromTarget`, which would mean + * re-implementing correlation a second time. + * + * WHY THE SOCKET IS SHARED AND REFERENCE-COUNTED. Chrome mints a NEW session + * per attach, and sessions on one socket are independent — measured: attaching + * twice to one target yields two distinct ids, and detaching one leaves the + * other working. So concurrent calls can and should share a single browser + * socket. What they must not do is close it while another call is still using + * it, hence the refcount: the socket opens on the first borrower and closes + * after the last one releases. + * + * THE WEDGED-TAB PROPERTY IS PRESERVED, and it is the reason the toolkit is + * worth pointing at a browser with a wedged tab in it. A command to a hung + * renderer times out in `CdpConnection.send` exactly as before — the timer is + * per command id, not per socket — and because this transport never attaches + * to a target it was not asked for, a wedged tab is only ever reached by a + * call that named it. + */ +import { CdpConnection, CdpError, DEFAULT_TIMEOUT_MS } from "../client.ts"; +import { detectEndpoint } from "./endpoint.ts"; + +type EventHandler = (params: Record, sessionId?: string) => void; + +/* ----------------------------- shared browser socket ----------------------------- */ + +let shared: { conn: CdpConnection; refs: number; url: string } | undefined; +let opening: Promise | undefined; + +/** + * Borrow the shared browser socket, opening it if nobody holds one. + * + * `opening` deduplicates a concurrent first borrow: two calls that arrive + * together must end up on ONE socket, not race to open two and leak the loser. + */ +async function acquireBrowserConn(): Promise { + if (shared) { + shared.refs++; + return shared.conn; + } + if (!opening) { + opening = (async () => { + const { browserWsUrl } = await detectEndpoint(); + const conn = await new CdpConnection(browserWsUrl).connect(); + shared = { conn, refs: 0, url: browserWsUrl }; + return conn; + })().finally(() => { + opening = undefined; + }); + } + const conn = await opening; + // `shared` is set by the block above before it resolves; a concurrent + // release cannot have run in between, because releasing requires a ref and + // this borrower has not taken one yet. + shared!.refs++; + return conn; +} + +/** Return a borrow. The socket closes when the last holder lets go. */ +function releaseBrowserConn(): void { + if (!shared) return; + shared.refs--; + if (shared.refs <= 0) { + const { conn } = shared; + shared = undefined; + conn.close(); + } +} + +/** + * Run `fn` on the shared browser socket. The browser-domain counterpart of + * client.ts's `withPage`, and the only way this module opens a socket. + */ +export async function withBrowserSocket(fn: (conn: CdpConnection) => Promise): Promise { + const conn = await acquireBrowserConn(); + try { + return await fn(conn); + } finally { + releaseBrowserConn(); + } +} + +/* -------------------------------- page sessions -------------------------------- */ + +/** + * One attached target, shaped like a `CdpConnection`. + * + * Structural, not a subclass: `CdpConnection` owns a socket in its constructor + * and this owns a sessionId, so there is no state to inherit. Callers hold + * them through the shared `PageConnection` type below. + */ +export class SessionConnection { + private detached = false; + private readonly offs: Array<() => void> = []; + + constructor( + private readonly browser: CdpConnection, + readonly sessionId: string, + private readonly opts: { timeoutMs?: number } = {}, + ) {} + + /** Send over this session. Same timeout semantics as CdpConnection.send. */ + send>( + method: string, + params: Record = {}, + opts: { timeoutMs?: number; sessionId?: string } = {}, + ): Promise { + if (this.detached) return Promise.reject(new CdpError("connection not open")); + // An explicit sessionId wins, so a caller that attached a NESTED session of + // its own (cdp/workers.ts does this for ServiceWorker) still addresses it. + return this.browser.send(method, params, { + timeoutMs: opts.timeoutMs ?? this.opts.timeoutMs, + sessionId: opts.sessionId ?? this.sessionId, + }); + } + + /** + * Subscribe, filtered to THIS session. + * + * The filter is what makes one socket behave like many: the browser socket + * carries every attached target's events, and a handler registered here must + * see only its own. An event with no sessionId is browser-level (a + * `Target.*` notification) and is delivered too, matching what a per-target + * socket shows. + */ + on(method: string, handler: EventHandler): () => void { + const off = this.browser.on(method, (params, sessionId) => { + if (sessionId && sessionId !== this.sessionId) return; + handler(params, sessionId); + }); + this.offs.push(off); + return off; + } + + waitFor

>( + method: string, + predicate?: (params: P) => boolean, + timeoutMs = this.opts.timeoutMs ?? DEFAULT_TIMEOUT_MS, + ): Promise

{ + return new Promise

((resolve, reject) => { + const timer = setTimeout(() => { + off(); + reject(new CdpError(`waitFor('${method}') timed out after ${timeoutMs}ms`)); + }, timeoutMs); + const off = this.on(method, (params) => { + if (!predicate || predicate(params as P)) { + clearTimeout(timer); + off(); + resolve(params as P); + } + }); + }); + } + + /** + * Detach and return the socket borrow. + * + * Named `close` because that is the method every tool module already calls in + * its `finally`. The detach is fire-and-forget: the borrow must be returned + * even when the target died first, and a failed detach on a target that is + * already gone is not a caller's problem. + */ + close(): void { + if (this.detached) return; + this.detached = true; + for (const off of this.offs) off(); + this.offs.length = 0; + void this.browser.send("Target.detachFromTarget", { sessionId: this.sessionId }).catch(() => { + /* target already gone */ + }); + releaseBrowserConn(); + } +} + +/** + * What every tool module actually holds: a `CdpConnection` when the endpoint + * serves per-target sockets, a `SessionConnection` when it does not. + * + * AN INTERFACE, NOT A UNION, on purpose. A union of the two classes would make + * every tool module narrow before it could send, for a distinction none of them + * cares about. This is the surface they were already using — four methods — + * so both implementations satisfy it structurally and `withPage` hands over + * whichever one the transport produced. + */ +export interface PageConnection { + send>( + method: string, + params?: Record, + opts?: { timeoutMs?: number; sessionId?: string }, + ): Promise; + on(method: string, handler: EventHandler): () => void; + waitFor

>( + method: string, + predicate?: (params: P) => boolean, + timeoutMs?: number, + ): Promise

; + close(): void; +} + +/** Attach to one target and wrap the session. The caller closes it. */ +export async function openSession( + targetId: string, + opts: { timeoutMs?: number } = {}, +): Promise { + const browser = await acquireBrowserConn(); + try { + const { sessionId } = await browser.send<{ sessionId?: string }>( + "Target.attachToTarget", + { targetId, flatten: true }, + { timeoutMs: opts.timeoutMs }, + ); + if (!sessionId) throw new CdpError(`Target.attachToTarget returned no sessionId for ${targetId}`); + return new SessionConnection(browser, sessionId, opts); + } catch (e) { + releaseBrowserConn(); + throw e; + } +} diff --git a/src/client.ts b/src/client.ts index 5fea822..6586bff 100644 --- a/src/client.ts +++ b/src/client.ts @@ -24,8 +24,10 @@ import { FRAME_EMPTY_NEEDLE_MESSAGE, } from "./frames.ts"; -/** Base HTTP origin of the DevTools endpoint. Override with CDP_BASE. */ -export const BASE = process.env.CDP_BASE ?? "http://127.0.0.1:9222"; +export { BASE } from "./cdp/endpoint.ts"; +import { BASE, detectEndpoint, resetEndpointCache, type CdpTransport } from "./cdp/endpoint.ts"; +import { openSession, withBrowserSocket, type PageConnection } from "./cdp/session.ts"; +export { SessionConnection, withBrowserSocket, type PageConnection } from "./cdp/session.ts"; /** Default per-command timeout (ms). Override with CDP_TIMEOUT_MS. */ export const DEFAULT_TIMEOUT_MS = Number(process.env.CDP_TIMEOUT_MS ?? 15_000); @@ -193,21 +195,93 @@ export class CdpConnection { /* ----------------------------- endpoint discovery ----------------------------- */ +/** + * Which transport this endpoint turned out to serve, learned from the first + * listing that succeeded. Undefined until then. + */ +let knownTransport: CdpTransport | undefined; + +/** Test seam: forget the learned transport. */ +export function resetTransportCache(): void { + knownTransport = undefined; + resetEndpointCache(); +} + async function httpJson(path: string): Promise { const res = await fetch(`${BASE}${path}`); if (!res.ok) throw new CdpError(`${path} -> HTTP ${res.status}`); return (await res.json()) as T; } -/** All targets (GET /json/list). */ -export function listTargets(): Promise { - return httpJson("/json/list"); +/** + * `Target.getTargets` reshaped into the `/json/list` record every caller + * already reads. + * + * TWO DIFFERENCES from the HTTP listing, both deliberate: + * + * `filter: [{}]` is passed because the DEFAULT filter is not "everything" — it + * omits types the empty filter includes. Measured on Chrome 153: default + * returned 4 targets (page, browser_ui), `[{}]` returned 6, and the missing + * ones were exactly the iframe/worker types that the `frame:` and `worker:` + * selector arms resolve against. Taking the default would silently delete two + * documented features. + * + * `webSocketDebuggerUrl` is EMPTY, and it has to be: on this transport there is + * no per-target socket to name, and inventing the `/devtools/page/` URL + * would hand callers a string that 403s. Nothing reads the field on this path — + * `openPage` branches on transport before it would — and the emptiness is what + * makes a caller that ignored the branch fail loudly instead of hanging. + */ +async function getTargetsOverBrowserWs(): Promise { + const { targetInfos } = await withBrowserSocket((conn) => + conn.send<{ targetInfos: Array<{ targetId: string; type: string; title: string; url: string; openerId?: string }> }>( + "Target.getTargets", + { filter: [{}] }, + ), + ); + return targetInfos.map((t) => ({ + id: t.targetId, + type: t.type, + title: t.title, + url: t.url, + webSocketDebuggerUrl: "", + ...(t.openerId ? { parentId: t.openerId } : {}), + })); +} + +/** + * All targets: `GET /json/list`, falling back to `Target.getTargets` on the + * browser socket when this Chrome does not serve it. + * + * THE HTTP REQUEST GOES FIRST AND UNCONDITIONALLY, which is the whole + * no-regression story: against a Chrome that serves `/json/list` this function + * does exactly what it did before the fallback existed, and the fallback is + * unreachable. Only a request that failed opens a socket. + * + * Once a transport is known it is cached, so the 404 is paid once per process + * rather than on every listing. + */ +export async function listTargets(): Promise { + if (knownTransport === "browser-ws") return getTargetsOverBrowserWs(); + try { + const listing = await httpJson("/json/list"); + knownTransport = "http"; + return listing; + } catch (httpError) { + const targets = await getTargetsOverBrowserWs().catch(() => { + // The HTTP failure is the one worth reporting: it names the endpoint the + // user configured, where the fallback's failure only says a profile + // lookup came up empty. + throw httpError; + }); + knownTransport = "browser-ws"; + return targets; + } } -/** Browser-level WebSocket URL (GET /json/version), for Target.* / Browser.*. */ +/** Browser-level WebSocket URL, for Target.* / Browser.*. */ export async function browserWsUrl(): Promise { - const v = await httpJson<{ webSocketDebuggerUrl: string }>("/json/version"); - return v.webSocketDebuggerUrl; + return (await detectEndpoint()).browserWsUrl; } /** Resolve a TargetSelector to a concrete page target. See types.ts for grammar. @@ -317,13 +391,25 @@ export async function openBrowser(opts: { timeoutMs?: number } = {}): Promise { +): Promise<{ conn: PageConnection; target: Target }> { + // resolveTarget listed first, so the transport is already known here: an + // empty webSocketDebuggerUrl is the browser-ws listing's own marker that this + // target has no socket to dial (see getTargetsOverBrowserWs). const target = await resolveTarget(selector, { lease: opts.lease }); - const conn = await new CdpConnection(target.webSocketDebuggerUrl, opts).connect(); + const conn = target.webSocketDebuggerUrl + ? await new CdpConnection(target.webSocketDebuggerUrl, opts).connect() + : await openSession(target.id, opts); return { conn, target }; } @@ -333,7 +419,7 @@ export async function openPage( */ export async function withPage( selector: TargetSelector, - fn: (conn: CdpConnection, target: Target) => Promise, + fn: (conn: PageConnection, target: Target) => Promise, opts: { timeoutMs?: number; lease?: string } = {}, ): Promise { const { conn, target } = await openPage(selector, opts); diff --git a/src/tools/evaluate.ts b/src/tools/evaluate.ts index ed64848..479e1d9 100644 --- a/src/tools/evaluate.ts +++ b/src/tools/evaluate.ts @@ -15,7 +15,7 @@ * `CdpError` carrying the exception/description text. */ import { CdpError, withPage } from "../client.ts"; -import type { CdpConnection } from "../client.ts"; +import type { PageConnection } from "../client.ts"; import type { TargetSelector } from "../types.ts"; /** A CDP RemoteObject as returned by Runtime.evaluate / Runtime.callFunctionOn. */ @@ -78,7 +78,7 @@ function unwrap(obj: RemoteObject): unknown { return obj.description ?? null; } -async function evaluateExpression(conn: CdpConnection, expression: string, awaitPromise: boolean): Promise { +async function evaluateExpression(conn: PageConnection, expression: string, awaitPromise: boolean): Promise { const res = await conn.send("Runtime.evaluate", { expression, returnByValue: true, @@ -94,7 +94,7 @@ async function evaluateExpression(conn: CdpConnection, expression: string, await } async function evaluateFunction( - conn: CdpConnection, + conn: PageConnection, expression: string, awaitPromise: boolean, args: unknown[], diff --git a/src/tools/navigation.ts b/src/tools/navigation.ts index 11f12de..2d0f64f 100644 --- a/src/tools/navigation.ts +++ b/src/tools/navigation.ts @@ -12,7 +12,7 @@ * on a fixed interval until the text appears or the timeout elapses. */ import { CdpError, withPage } from "../client.ts"; -import type { CdpConnection } from "../client.ts"; +import type { PageConnection } from "../client.ts"; import type { Target, TargetSelector } from "../types.ts"; export interface NavigatePageArgs { @@ -65,7 +65,7 @@ export async function navigatePage(args: NavigatePageArgs): Promise => { + async (conn: PageConnection): Promise => { await conn.send("Page.enable"); // Subscribe to the load milestone BEFORE acting so a fast page that loads @@ -156,7 +156,7 @@ export async function waitForText(args: WaitForArgs): Promise { return withPage( args.target, - async (conn: CdpConnection, _target: Target): Promise => { + async (conn: PageConnection, _target: Target): Promise => { await conn.send("Runtime.enable"); const expr = `(() => { const b = document.body; return !!b && typeof b.innerText === 'string' && b.innerText.includes(${JSON.stringify(args.text)}); })()`; diff --git a/src/tools/network_mock.ts b/src/tools/network_mock.ts index b7a8368..c1a8117 100644 --- a/src/tools/network_mock.ts +++ b/src/tools/network_mock.ts @@ -11,7 +11,7 @@ * CDP connection per target; integration-tested in test/mock-smoke.ts. */ import { CdpError, openPage, resolveTarget } from "../client.ts"; -import type { CdpConnection } from "../client.ts"; +import type { PageConnection } from "../client.ts"; import type { Target, TargetSelector } from "../types.ts"; /* ============================================================================ @@ -143,7 +143,7 @@ interface MockRule { } interface MockSession { - conn: CdpConnection; + conn: PageConnection; target: Target; rules: MockRule[]; intercepted: Array<{ url: string; method: string; action: MockAction }>; diff --git a/src/tools/performance.ts b/src/tools/performance.ts index 8dac655..f05aeb4 100644 --- a/src/tools/performance.ts +++ b/src/tools/performance.ts @@ -35,7 +35,7 @@ * path and the recommended entry point. */ import type { Target, TargetSelector } from "../types.ts"; -import type { CdpConnection } from "../client.ts"; +import type { PageConnection } from "../client.ts"; import { CdpError, openPage } from "../client.ts"; import { mkdir, readFile, writeFile, stat, unlink } from "node:fs/promises"; import { join } from "node:path"; @@ -75,7 +75,7 @@ interface TraceEvent { /** A traced-page registry entry held in-process between start and stop. */ interface LiveTrace { - conn: CdpConnection; + conn: PageConnection; target: Target; events: TraceEvent[]; startedAt: number; @@ -108,7 +108,7 @@ function stateFilePath(): string { } /** Subscribe to Tracing.dataCollected and buffer every event into `sink`. */ -function bufferTraceData(conn: CdpConnection, sink: TraceEvent[]): () => void { +function bufferTraceData(conn: PageConnection, sink: TraceEvent[]): () => void { return conn.on("Tracing.dataCollected", (params: Record) => { const value = (params as { value?: TraceEvent[] }).value; if (Array.isArray(value)) sink.push(...value); @@ -116,7 +116,7 @@ function bufferTraceData(conn: CdpConnection, sink: TraceEvent[]): () => void { } /** Begin a trace on an already-open page connection. */ -async function beginTrace(conn: CdpConnection, categories: string[]): Promise { +async function beginTrace(conn: PageConnection, categories: string[]): Promise { await conn.send("Tracing.start", { traceConfig: { includedCategories: categories }, transferMode: "ReportEvents", @@ -127,7 +127,7 @@ async function beginTrace(conn: CdpConnection, categories: string[]): Promise { +async function endTrace(conn: PageConnection, sink: TraceEvent[], timeoutMs = 30_000): Promise { const complete = conn.waitFor("Tracing.tracingComplete", undefined, timeoutMs); await conn.send("Tracing.end"); await complete; diff --git a/src/tools/recorder.ts b/src/tools/recorder.ts index 29c6b76..071e6c7 100644 --- a/src/tools/recorder.ts +++ b/src/tools/recorder.ts @@ -66,7 +66,7 @@ */ import { mkdir, appendFile, writeFile, copyFile } from "node:fs/promises"; import { CdpError, listTargets, openPage, resolveTarget } from "../client.ts"; -import type { CdpConnection } from "../client.ts"; +import type { PageConnection } from "../client.ts"; import { resolveWorkerSelector } from "../cdp/workers.ts"; import type { Target, TargetSelector } from "../types.ts"; import { @@ -189,7 +189,7 @@ export interface RecorderHandle { /** The resolved target this recorder is attached to. */ target: Target; /** The live connection (exposed so a one-shot caller can drive Page.reload on it). */ - conn: CdpConnection; + conn: PageConnection; /** Count of buffer-write failures observed (0 on a healthy capture). */ droppedWrites(): number; } @@ -286,7 +286,7 @@ export interface CaptureWindow { /** The unique per-capture buffer file (read this for isolated results). */ file: string; /** Live connection (open until stop), for Network.getResponseBody etc. */ - conn: CdpConnection; + conn: PageConnection; /** Resolved target identity. */ resolved: { id: string; url: string; title: string }; /** How the window was driven: "reload" for a page, "listen" for a worker. */ diff --git a/src/tools/screencast.ts b/src/tools/screencast.ts index 5ad677c..f17d2ad 100644 --- a/src/tools/screencast.ts +++ b/src/tools/screencast.ts @@ -54,7 +54,7 @@ import { spawn } from "node:child_process"; import { mkdir, rm, stat, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { CdpError, openPage } from "../client.ts"; -import type { CdpConnection } from "../client.ts"; +import type { PageConnection } from "../client.ts"; import type { Target, TargetSelector } from "../types.ts"; const ARTIFACT_DIR = process.env.CDP_ARTIFACT_DIR ?? "/tmp/cdp-toolkit"; @@ -367,7 +367,7 @@ async function probeEncoder(): Promise { /** A live screencast held in-process between start and stop, keyed by targetId. */ interface LiveRecording { - conn: CdpConnection; + conn: PageConnection; target: Target; spoolDir: string; ledger: FrameLedgerEntry[]; diff --git a/src/tools/snapshot.ts b/src/tools/snapshot.ts index 3c1740d..37fa56e 100644 --- a/src/tools/snapshot.ts +++ b/src/tools/snapshot.ts @@ -8,7 +8,7 @@ * `DOM.resolveNode({ backendNodeId: uid })`. There is no server-side ref * table to drift or expire. */ -import type { CdpConnection } from "../client.ts"; +import type { PageConnection } from "../client.ts"; import { withPage } from "../client.ts"; import type { Target, TargetSelector, Uid } from "../types.ts"; @@ -175,7 +175,7 @@ export async function takeSnapshot(args: TakeSnapshotArgs = {}): Promise { +export async function resolveUid(conn: PageConnection, uid: Uid): Promise<{ objectId: string }> { const { object } = await conn.send<{ object: { objectId?: string } }>("DOM.resolveNode", { backendNodeId: uid, }); diff --git a/test/browser-ws-only-proxy.ts b/test/browser-ws-only-proxy.ts new file mode 100644 index 0000000..09d6b71 --- /dev/null +++ b/test/browser-ws-only-proxy.ts @@ -0,0 +1,121 @@ +/** + * A DevTools endpoint that serves ONLY the browser socket, in front of a normal + * Chrome. + * + * WHY THIS EXISTS. The transport this fixture reproduces is not something you + * can ask Chrome for with a flag. It is what a Chrome exposes when debugging + * was enabled at RUNTIME, through the toggle on `chrome://inspect/#remote-debugging`, + * rather than with `--remote-debugging-port`. A Chrome in that state ALSO puts + * a modal consent dialog in front of every new client, so it is not something a + * test suite may dial by itself. + * + * So instead of a real one, this proxies a disposable `--remote-debugging-port` + * Chrome and removes exactly what that mode removes, with the responses it was + * measured to give (Chrome 153): + * + * GET /json/version, /json/list -> 404 + * WS /devtools/page/ -> 403 + * WS /devtools/browser/ -> 101, proxied verbatim + * + * A test that passes through here therefore proves the transport works with no + * HTTP discovery and no per-target socket available — which is the claim — on a + * browser the suite owns and may connect to freely. + */ +declare const Bun: { + serve(opts: { + port: number; + fetch(req: Request, server: { upgrade(req: Request, opts?: { data?: unknown }): boolean }): Response | Promise | undefined; + websocket: { + open(ws: BunSocket): void | Promise; + message(ws: BunSocket, msg: string | Buffer): void; + close(ws: BunSocket): void; + }; + }): { port: number; stop(closeActive?: boolean): void }; +}; +interface BunSocket { + data: { upstream?: WebSocket; queue: string[]; ready: boolean }; + send(msg: string): void; + close(): void; +} + +export interface BrowserWsOnlyEndpoint { + /** The base URL to hand the toolkit as CDP_BASE. Its /json/* all 404. */ + base: string; + /** The one URL that upgrades, to hand the toolkit as CDP_BROWSER_WS. */ + browserWsUrl: string; + stop(): void; +} + +/** + * Front `upstreamBase` (a normal Chrome) with a browser-ws-only endpoint. + * + * The upstream browser socket is resolved ONCE here, through the upstream's own + * `/json/version` — that is the fixture's own plumbing, not part of what is + * under test, and it is why the toolkit can be given a `CDP_BROWSER_WS` without + * any `/json` of its own. + */ +export async function startBrowserWsOnlyProxy(upstreamBase: string): Promise { + const { webSocketDebuggerUrl } = (await fetch(`${upstreamBase}/json/version`).then((r) => r.json())) as { + webSocketDebuggerUrl: string; + }; + + const server = Bun.serve({ + port: 0, + fetch(req, srv) { + const { pathname } = new URL(req.url); + // The browser socket is the ONLY thing that upgrades. + if (pathname.startsWith("/devtools/browser/")) { + if (srv.upgrade(req, { data: { queue: [], ready: false } })) return undefined; + return new Response("upgrade failed", { status: 400 }); + } + // What the runtime-toggle mode returns for a per-target socket. + if (pathname.startsWith("/devtools/page/") || pathname.startsWith("/devtools/frame/")) { + return new Response("Forbidden", { status: 403 }); + } + // ...and for HTTP discovery. + return new Response("Not Found", { status: 404 }); + }, + websocket: { + async open(ws) { + const upstream = new WebSocket(webSocketDebuggerUrl); + ws.data.upstream = upstream; + upstream.onmessage = (ev: MessageEvent) => ws.send(String(ev.data)); + upstream.onclose = () => ws.close(); + upstream.onerror = () => ws.close(); + // Settle on failure too: resolving only on open would leave a failed + // upstream handshake queueing client messages forever with no error. + await new Promise((resolve) => { + upstream.onopen = () => resolve(); + upstream.addEventListener("error", () => resolve()); + upstream.addEventListener("close", () => resolve()); + }); + ws.data.ready = true; + // Anything the client sent during the upstream handshake, in order. + for (const queued of ws.data.queue) upstream.send(queued); + ws.data.queue.length = 0; + }, + message(ws, msg) { + const text = String(msg); + if (!ws.data.ready) { + ws.data.queue.push(text); + return; + } + ws.data.upstream?.send(text); + }, + close(ws) { + try { + ws.data.upstream?.close(); + } catch { + /* ignore */ + } + }, + }, + }); + + const path = new URL(webSocketDebuggerUrl).pathname; + return { + base: `http://127.0.0.1:${server.port}`, + browserWsUrl: `ws://127.0.0.1:${server.port}${path}`, + stop: () => server.stop(true), + }; +} diff --git a/test/browser-ws-smoke.ts b/test/browser-ws-smoke.ts new file mode 100644 index 0000000..9c27242 --- /dev/null +++ b/test/browser-ws-smoke.ts @@ -0,0 +1,172 @@ +/** + * End-to-end proof that the toolkit drives a browser-ws-only endpoint. + * + * Owns its browser: spawns a disposable headless Chrome on an ephemeral port + * with a throwaway profile, fronts it with the browser-ws-only proxy (see + * ./browser-ws-only-proxy.ts), and kills it on the way out. It never looks for + * a Chrome you are already running, by design — see the consent-prompt note in + * src/cdp/endpoint.ts. + * + * Run: bun run browser-ws:smoke + * Scale the tab count with TABS=200 to reproduce a many-tab browser. + */ +import { spawn } from "node:child_process"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { startBrowserWsOnlyProxy } from "./browser-ws-only-proxy.ts"; + +const TABS = Number(process.env.TABS ?? 40); +/** Every phase is capped at the real MCP tool timeout. */ +const TOOL_TIMEOUT_MS = 60_000; + +function chromeBinary(): string { + if (process.env.CHROME_BIN) return process.env.CHROME_BIN; + if (process.platform === "darwin") return "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"; + return "google-chrome"; +} + +/** Throws rather than exits, so the catch below still kills Chrome and its profile. */ +function fail(msg: string): never { + throw new Error(msg); +} + +/** Run `label` under the 60s tool budget, printing what it actually took. */ +async function phase(label: string, fn: () => Promise): Promise { + const t0 = Date.now(); + const timer = new Promise((_, reject) => + setTimeout(() => reject(new Error(`exceeded ${TOOL_TIMEOUT_MS}ms tool budget`)), TOOL_TIMEOUT_MS), + ); + try { + const out = await Promise.race([fn(), timer]); + console.log(` ok ${label} — ${((Date.now() - t0) / 1000).toFixed(2)}s`); + return out as T; + } catch (e) { + console.log(` FAIL ${label} — ${((Date.now() - t0) / 1000).toFixed(2)}s: ${(e as Error).message}`); + throw e; + } +} + +const profile = await mkdtemp(join(tmpdir(), "cdp-browser-ws-smoke-")); +const chrome = spawn( + chromeBinary(), + [ + "--headless=new", + `--user-data-dir=${profile}`, + "--remote-debugging-port=0", + "--no-first-run", + "--no-default-browser-check", + "--disable-sync", + "--disable-gpu", + ], + { stdio: ["ignore", "ignore", "pipe"] }, +); + +/** Chrome prints `DevTools listening on ws://...` to stderr once it is up. */ +const upstreamWs = await new Promise((resolve, reject) => { + let buf = ""; + const t = setTimeout(() => reject(new Error("Chrome did not report a DevTools endpoint in 30s")), 30_000); + chrome.stderr.on("data", (d: Buffer) => { + buf += d.toString(); + const m = buf.match(/DevTools listening on (ws:\/\/\S+)/); + if (m?.[1]) { + clearTimeout(t); + resolve(m[1]); + } + }); + chrome.on("exit", (code) => reject(new Error(`Chrome exited (${code}) before reporting an endpoint`))); +}); +const upstreamBase = `http://${new URL(upstreamWs).host}`; +console.log(`disposable Chrome: ${upstreamBase} (profile ${profile})`); + +const proxy = await startBrowserWsOnlyProxy(upstreamBase); +console.log(`browser-ws-only endpoint: ${proxy.base}`); + +async function cleanup(): Promise { + proxy.stop(); + chrome.kill("SIGKILL"); + await rm(profile, { recursive: true, force: true }).catch(() => {}); +} + +try { + /* ---- 0. the endpoint really is browser-ws-only ------------------------- */ + const version = await fetch(`${proxy.base}/json/version`).then((r) => r.status); + const list = await fetch(`${proxy.base}/json/list`).then((r) => r.status); + const pageWs = await new Promise((resolve) => { + const ws = new WebSocket(`${proxy.base.replace("http", "ws")}/devtools/page/DEADBEEF`); + ws.onopen = () => { + ws.close(); + resolve("101"); + }; + ws.onerror = () => resolve("refused (403)"); + }); + console.log(`\nendpoint shape: /json/version=${version} /json/list=${list} page-ws=${pageWs}`); + if (version !== 404 || list !== 404) fail("fixture is not browser-ws-only: /json answered"); + if (pageWs !== "refused (403)") fail(`fixture is not browser-ws-only: /devtools/page/* upgraded (${pageWs})`); + + /* ---- point the toolkit at it ------------------------------------------- */ + process.env.CDP_BASE = proxy.base; + process.env.CDP_BROWSER_WS = proxy.browserWsUrl; + process.env.CDP_REQUIRE_LEASE = "0"; + const { listTargets } = await import("../src/client.ts"); + const { TOOLS } = await import("../src/index.ts"); + + /* ---- open the tabs ------------------------------------------------------ */ + const { withBrowserSocket } = await import("../src/cdp/session.ts"); + await withBrowserSocket(async (conn) => { + for (let i = 0; i < TABS; i++) { + await conn.send("Target.createTarget", { + url: `data:text/html,Tab ${i}

tab ${i} heading

`, + }); + } + }); + console.log(`\nopened ${TABS} tabs\n`); + + /* ---- 1. discovery ------------------------------------------------------- */ + const pages = await phase(`list_pages over ${TABS}+ tabs`, async () => { + const out = (await TOOLS.list_pages({})) as { pages: Array<{ id: string; url: string; title: string }> }; + if (!Array.isArray(out.pages)) fail("list_pages returned no pages array"); + if (out.pages.length < TABS) fail(`list_pages returned ${out.pages.length}, expected >= ${TABS}`); + return out.pages; + }); + console.log(` -> ${pages.length} pages`); + + /* ---- 2. drive an ordinary tab ------------------------------------------ */ + const ordinary = pages.find((p) => p.url.includes("Tab%205") || p.title.includes("Tab 5")) ?? pages[pages.length - 1]; + if (!ordinary) fail("no ordinary tab to drive"); + await phase(`evaluate_script on an ordinary tab (${ordinary.title})`, async () => { + const v = (await TOOLS.evaluate_script({ target: ordinary.id, expression: "document.querySelector('#h').textContent" })) as unknown; + if (typeof v !== "string" || !v.includes("heading")) fail(`unexpected value: ${JSON.stringify(v)}`); + console.log(` -> ${JSON.stringify(v)}`); + }); + await phase(`take_snapshot on the same tab`, async () => { + const snap = (await TOOLS.take_snapshot({ target: ordinary.id })) as { snapshot: string; nodeCount: number }; + if (!snap.snapshot?.includes("heading")) fail("snapshot did not contain the heading"); + console.log(` -> ${snap.nodeCount} a11y nodes`); + }); + + /* ---- 3. the unfiltered listing still carries non-page targets ---------- */ + await phase("listTargets carries non-page types (frame:/worker: arms intact)", async () => { + const all = await listTargets(); + const types = new Set(all.map((t) => t.type)); + console.log(` -> ${all.length} targets, types: ${[...types].join(", ")}`); + if (!types.has("page")) fail("no page targets in the unfiltered listing"); + }); + + /* ---- 4. repeated drives, no state bleed -------------------------------- */ + await phase("5 consecutive drives across different tabs", async () => { + for (let i = 0; i < 5; i++) { + const p = pages[i % pages.length]!; + const v = (await TOOLS.evaluate_script({ target: p.id, expression: "document.title" })) as unknown; + if (typeof v !== "string") fail(`drive ${i} returned ${JSON.stringify(v)}`); + } + }); + + console.log("\nPASS — browser-ws-only transport drives discovery and per-tab work under the 60s budget."); + await cleanup(); + process.exit(0); +} catch (e) { + console.error(`\nFAILED: ${(e as Error).message}`); + await cleanup(); + process.exit(1); +} diff --git a/test/browser-ws-wedge.ts b/test/browser-ws-wedge.ts new file mode 100644 index 0000000..09a13b2 --- /dev/null +++ b/test/browser-ws-wedge.ts @@ -0,0 +1,145 @@ +/** + * The wedged-tab claim, over the browser-ws-only transport. + * + * WHY THIS IS A SEPARATE CHECK from scripts/wedge-bench.ts. That benchmark + * proves the timeout bound holds when every page has its own socket, where a + * hung renderer can only ever block the one socket dialed into it. This + * transport puts every page on ONE shared socket, which is exactly the + * arrangement where a naive implementation would let one stuck tab block all + * the others — so the property has to be re-proven, not inherited. + * + * What it measures, against a disposable Chrome fronted by the browser-ws-only + * proxy (see ./browser-ws-only-proxy.ts): + * + * 1. a tab navigated to a socket that accepts and never answers is driven + * anyway, and the call REJECTS at the bound instead of hanging; + * 2. while that tab is stuck, a witness tab stays fast — the shared socket + * is not head-of-line blocked; + * 3. discovery (list_pages) still answers fast with the stuck tab open, + * which is the whole point: a wedged tab must not cost you the listing. + * + * Run: bun run browser-ws:wedge + */ +import { spawn } from "node:child_process"; +import { createServer } from "node:net"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { startBrowserWsOnlyProxy } from "./browser-ws-only-proxy.ts"; + +const BOUND_MS = Number(process.env.CDP_TIMEOUT_MS ?? 5_000); +const SLACK_MS = 2_000; +const FAST_BUDGET_MS = 1_500; + +process.env.CDP_TIMEOUT_MS = String(BOUND_MS); + +function chromeBinary(): string { + if (process.env.CHROME_BIN) return process.env.CHROME_BIN; + if (process.platform === "darwin") return "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"; + return "google-chrome"; +} + +async function timed(fn: () => Promise): Promise<{ ms: number; result?: T; error?: Error }> { + const t0 = performance.now(); + try { + return { ms: performance.now() - t0, result: await fn() }; + } catch (error) { + return { ms: performance.now() - t0, error: error as Error }; + } +} +const fmt = (n: number): string => (n >= 1000 ? `${(n / 1000).toFixed(2)}s` : `${n.toFixed(0)}ms`); + +/** A server that completes the TCP accept and then never writes a byte. */ +const blackhole = createServer((socket) => { + socket.on("error", () => {}); + // deliberately no response, ever +}); +await new Promise((resolve) => blackhole.listen(0, "127.0.0.1", resolve)); +const blackholeUrl = `http://127.0.0.1:${(blackhole.address() as { port: number }).port}/hang`; + +const profile = await mkdtemp(join(tmpdir(), "cdp-ws-wedge-")); +const chrome = spawn( + chromeBinary(), + [ + "--headless=new", + `--user-data-dir=${profile}`, + "--remote-debugging-port=0", + "--no-first-run", + "--no-default-browser-check", + "--disable-sync", + "--disable-gpu", + ], + { stdio: ["ignore", "ignore", "pipe"] }, +); +const upstreamWs = await new Promise((resolve, reject) => { + let buf = ""; + const t = setTimeout(() => reject(new Error("Chrome did not report an endpoint in 30s")), 30_000); + chrome.stderr.on("data", (d: Buffer) => { + buf += d.toString(); + const m = buf.match(/DevTools listening on (ws:\/\/\S+)/); + if (m?.[1]) { + clearTimeout(t); + resolve(m[1]); + } + }); + chrome.on("exit", (code) => reject(new Error(`Chrome exited (${code}) before reporting an endpoint`))); +}); +const proxy = await startBrowserWsOnlyProxy(`http://${new URL(upstreamWs).host}`); + +process.env.CDP_BASE = proxy.base; +process.env.CDP_BROWSER_WS = proxy.browserWsUrl; +process.env.CDP_REQUIRE_LEASE = "0"; + +let failures = 0; +const check = (ok: boolean, label: string, detail: string): void => { + if (!ok) failures++; + console.log(` ${ok ? "ok " : "FAIL"} ${label} — ${detail}`); +}; + +try { + const { TOOLS } = await import("../src/index.ts"); + console.log(`browser-ws-only endpoint ${proxy.base}, bound ${BOUND_MS}ms\n`); + + const witness = (await TOOLS.new_page({ url: "data:text/html,witness" })) as { targetId: string }; + const stuck = (await TOOLS.new_page({ url: "about:blank" })) as { targetId: string }; + + // Navigate the victim into the blackhole. The navigate itself is expected to + // reject at the bound; what matters is everything after it. + const nav = await timed(() => TOOLS.navigate_page({ target: stuck.targetId, url: blackholeUrl })); + console.log(` (victim navigate returned in ${fmt(nav.ms)}${nav.error ? ` — ${nav.error.message}` : ""})\n`); + + // 1. driving the stuck tab rejects at the bound, never hangs. + const drive = await timed(() => TOOLS.evaluate_script({ target: stuck.targetId, expression: "1+1" })); + check( + !!drive.error && drive.ms < BOUND_MS + SLACK_MS, + "stuck tab rejects at the bound (no hang)", + `${fmt(drive.ms)}, ${drive.error ? `rejected: ${drive.error.message.slice(0, 60)}` : "RESOLVED — expected a rejection"}`, + ); + + // 2. the witness tab is unaffected: the shared socket is not head-of-line blocked. + const w = await timed(() => TOOLS.evaluate_script({ target: witness.targetId, expression: "document.title" })); + check( + !w.error && w.result === "witness" && w.ms < FAST_BUDGET_MS, + "witness tab stays fast while the other is stuck", + `${fmt(w.ms)}, value=${JSON.stringify(w.result)}`, + ); + + // 3. discovery still answers fast. + const l = await timed(() => TOOLS.list_pages({})); + const pageCount = (l.result as { pages: unknown[] } | undefined)?.pages.length ?? 0; + check(!l.error && l.ms < FAST_BUDGET_MS, "list_pages stays fast with a stuck tab open", `${fmt(l.ms)}, ${pageCount} pages`); + + // 4. recovery: close the bricked tab, drive a fresh one at healthy latency. + await TOOLS.close_page({ target: stuck.targetId }).catch(() => {}); + const fresh = (await TOOLS.new_page({ url: "data:text/html,fresh" })) as { targetId: string }; + const r = await timed(() => TOOLS.evaluate_script({ target: fresh.targetId, expression: "document.title" })); + check(!r.error && r.result === "fresh" && r.ms < FAST_BUDGET_MS, "recovery after closing the bricked tab", `${fmt(r.ms)}`); + + console.log(failures === 0 ? "\nPASS — one stuck tab does not wedge the shared browser socket." : `\nFAILED (${failures})`); +} finally { + proxy.stop(); + chrome.kill("SIGKILL"); + blackhole.close(); + await rm(profile, { recursive: true, force: true }).catch(() => {}); +} +process.exit(failures === 0 ? 0 : 1);