feat(sparsekernel): expand brokered browser actions

Author: AbrahamGreenman

docs/architecture/browser-broker.md

@@ -15,7 +15,7 @@ V0 also has a concrete local-browser attachment point: callers may register and
 
 The `@openclaw/sparsekernel-browser-broker` adapter materializes the ledger lease into a real CDP browser context with `Target.createBrowserContext`, creates a target for the task/session, captures screenshots through `Page.captureScreenshot`, captures PDFs through `Page.printToPDF`, records console events, and writes screenshots/downloads/PDFs through the SparseKernel artifact API. Callers can construct it directly with a kernel-shaped client or use `createSparseKernelCdpBrowserBroker` to wire it to `sparsekerneld`. Download capture uses CDP download events and the content-addressed artifact store rather than exposing arbitrary download paths to agents.
 
-When `OPENCLAW_RUNTIME_BROWSER_BROKER=cdp` and `OPENCLAW_SPARSEKERNEL_BROWSER_CDP_ENDPOINT=<loopback endpoint>` are set, the embedded OpenClaw browser tool receives an internal SparseKernel proxy instead of raw CDP access. Set `OPENCLAW_RUNTIME_BROWSER_BROKER=managed` to let SparseKernel ask the existing OpenClaw browser control service to start the managed browser and return its loopback CDP endpoint; `OPENCLAW_SPARSEKERNEL_BROWSER_CONTROL_URL` overrides the default `http://127.0.0.1:18791` control URL. Supported v0 actions (`status`, `doctor`, `profiles`, `tabs`, `open`, `navigate`, `focus`, `close`, `snapshot`, `console`, `screenshot`, `pdf`, direct file-input `upload`, `dialog`, and basic `act`) operate against the leased CDP context. Snapshots use a bounded CDP `Runtime.evaluate` DOM read, basic actions resolve refs from the latest brokered snapshot, console output is captured from CDP runtime/log events, and screenshot/PDF output is captured as SparseKernel artifacts, read back through artifact access, and converted to existing tool result formats for compatibility. The context is retained for the active embedded run and released during broker cleanup, not opened and closed for every browser tool call.
+When `OPENCLAW_RUNTIME_BROWSER_BROKER=cdp` and `OPENCLAW_SPARSEKERNEL_BROWSER_CDP_ENDPOINT=<loopback endpoint>` are set, the embedded OpenClaw browser tool receives an internal SparseKernel proxy instead of raw CDP access. Set `OPENCLAW_RUNTIME_BROWSER_BROKER=managed` to let SparseKernel ask the existing OpenClaw browser control service to start the managed browser and return its loopback CDP endpoint; `OPENCLAW_SPARSEKERNEL_BROWSER_CONTROL_URL` overrides the default `http://127.0.0.1:18791` control URL. Supported v0 actions (`status`, `doctor`, `profiles`, `tabs`, `open`, `navigate`, `focus`, `close`, `snapshot`, `console`, `screenshot`, `pdf`, direct file-input `upload`, `dialog`, and brokered `act`) operate against the leased CDP context. Brokered `act` covers the OpenClaw action contract for click, coordinate click, type, press, hover, scroll, drag, select, fill, resize, wait, evaluate, close, and batch using CDP input events plus bounded DOM evaluation. Snapshots use a bounded CDP `Runtime.evaluate` DOM read, actions resolve refs from the latest brokered snapshot where needed, console output is captured from CDP runtime/log events, and screenshot/PDF output is captured as SparseKernel artifacts, read back through artifact access, and converted to existing tool result formats for compatibility. The context is retained for the active embedded run and released during broker cleanup, not opened and closed for every browser tool call.
 
 The current daemon can ask the existing OpenClaw browser control service to own the browser process, but it does not yet launch or supervise SparseKernel-native process pools by trust zone. The next backend should:
 
@@ -26,3 +26,5 @@ The current daemon can ask the existing OpenClaw browser control service to own
 - enforce max context counts through leases.
 
 BrowserContext isolation is session isolation, not host isolation. Playwright route blocking is useful request control, not a hard security boundary.
+
+The brokered CDP action engine is intentionally implemented inside the broker boundary, but it is not a byte-for-byte Playwright clone. Some browser behaviors that depend on Playwright's locator retry model or network-idle accounting may still differ and should be covered by targeted tests before relying on them for critical automations.

docs/architecture/local-agent-kernel.md

@@ -96,7 +96,7 @@ The browser broker model is:
 
 Important boundary: BrowserContext isolation is session isolation, not host isolation. Playwright route blocking and SSRF guards are useful controls, but they are not hard security boundaries.
 
-The broker applies configured trust-zone network policy to explicit allowed origins before allocating a context. This is an egress guard for brokered contexts, not a kernel or VM boundary. Set `OPENCLAW_RUNTIME_BROWSER_BROKER=cdp` and `OPENCLAW_SPARSEKERNEL_BROWSER_CDP_ENDPOINT=<loopback endpoint>` to make the OpenClaw browser tool acquire a real SparseKernel CDP context for the active run. Set `OPENCLAW_RUNTIME_BROWSER_BROKER=managed` to use the existing OpenClaw browser control service as the managed process owner and let SparseKernel lease CDP contexts from its reported endpoint. The runtime injects an internal browser proxy for supported navigation, tab, snapshot, console, screenshot, PDF, direct file-input upload, dialog, and basic action routes instead of exposing raw CDP to the agent. Screenshot and PDF outputs go through the artifact store.
+The broker applies configured trust-zone network policy to explicit allowed origins before allocating a context. This is an egress guard for brokered contexts, not a kernel or VM boundary. Set `OPENCLAW_RUNTIME_BROWSER_BROKER=cdp` and `OPENCLAW_SPARSEKERNEL_BROWSER_CDP_ENDPOINT=<loopback endpoint>` to make the OpenClaw browser tool acquire a real SparseKernel CDP context for the active run. Set `OPENCLAW_RUNTIME_BROWSER_BROKER=managed` to use the existing OpenClaw browser control service as the managed process owner and let SparseKernel lease CDP contexts from its reported endpoint. The runtime injects an internal browser proxy for supported navigation, tab, snapshot, console, screenshot, PDF, direct file-input upload, dialog, and action routes instead of exposing raw CDP to the agent. Brokered actions cover the OpenClaw action contract with CDP input events and bounded DOM evaluation; screenshot and PDF outputs go through the artifact store.
 
 ## Sandbox broker
 

packages/browser-broker/src/index.test.ts

@@ -108,33 +108,9 @@ class FakeCdpTransport implements CdpTransport {
         this.handleNavigate(String(message.params?.url ?? ""));
         break;
       case "Runtime.evaluate":
-        this.respond(message.id, {
-          result: {
-            value: JSON.stringify({
-              title: this.currentUrl.includes("example.com") ? "Example" : "",
-              url: this.currentUrl,
-              text: "Example page body",
-              links: [
-                {
-                  text: "Example link",
-                  href: "https://example.com/link",
-                  selector: "body > a:nth-of-type(1)",
-                },
-              ],
-              buttons: [{ text: "Submit", selector: "body > button:nth-of-type(1)" }],
-              inputs: [
-                {
-                  label: "Search",
-                  name: "q",
-                  type: "text",
-                  selector: "body > input:nth-of-type(1)",
-                },
-              ],
-              truncated: false,
-            }),
-          },
-        });
+        this.respondRuntimeEvaluate(message);
         break;
+      case "Input.dispatchMouseEvent":
       case "Input.dispatchKeyEvent":
       case "Emulation.setDeviceMetricsOverride":
       case "Page.handleJavaScriptDialog":
@@ -196,6 +172,59 @@ class FakeCdpTransport implements CdpTransport {
     });
   }
 
+  private respondRuntimeEvaluate(message: { id: number; params?: Record<string, unknown> }): void {
+    const expression = String(message.params?.expression ?? "");
+    if (expression.includes("const links =")) {
+      this.respond(message.id, {
+        result: {
+          value: JSON.stringify({
+            title: this.currentUrl.includes("example.com") ? "Example" : "",
+            url: this.currentUrl,
+            text: "Example page body",
+            links: [
+              {
+                text: "Example link",
+                href: "https://example.com/link",
+                selector: "body > a:nth-of-type(1)",
+              },
+            ],
+            buttons: [{ text: "Submit", selector: "body > button:nth-of-type(1)" }],
+            inputs: [
+              {
+                label: "Search",
+                name: "q",
+                type: "text",
+                selector: "body > input:nth-of-type(1)",
+              },
+            ],
+            truncated: false,
+          }),
+        },
+      });
+      return;
+    }
+    if (expression.includes("title: document.title")) {
+      this.respond(message.id, {
+        result: {
+          value: JSON.stringify({
+            title: this.currentUrl.includes("example.com") ? "Example" : "",
+            url: this.currentUrl,
+          }),
+        },
+      });
+      return;
+    }
+    if (expression.includes("() => 42")) {
+      this.respond(message.id, {
+        result: { value: 42 },
+      });
+      return;
+    }
+    this.respond(message.id, {
+      result: { value: { ok: true } },
+    });
+  }
+
   private respond(id: number, result: Record<string, unknown>): void {
     this.emit({ id, result });
   }
@@ -577,6 +606,93 @@ describe("@openclaw/sparsekernel-browser-broker", () => {
     );
   });
 
+  it("performs the broader brokered action contract against the leased context", async () => {
+    const kernel = new FakeKernel();
+    const transport = new FakeCdpTransport();
+    const broker = new SparseKernelCdpBrowserBroker({
+      kernel,
+      fetchImpl: async () =>
+        Response.json({
+          webSocketDebuggerUrl: "ws://127.0.0.1/devtools/browser/test",
+        }),
+      transportFactory: async () => transport,
+    });
+
+    const context = await broker.acquireContext({
+      trust_zone_id: "public_web",
+      cdp_endpoint: "http://127.0.0.1:9222",
+      initial_url: "https://example.com/",
+    });
+    await broker.snapshotContext(context.ledger_context.id);
+
+    await broker.actContext(context.ledger_context.id, {
+      kind: "clickCoords",
+      x: 10,
+      y: 20,
+      doubleClick: true,
+    });
+    await broker.actContext(context.ledger_context.id, {
+      kind: "scrollIntoView",
+      ref: "e1",
+    });
+    await broker.actContext(context.ledger_context.id, {
+      kind: "drag",
+      startRef: "e1",
+      endRef: "e2",
+    });
+    await broker.actContext(context.ledger_context.id, {
+      kind: "select",
+      ref: "e3",
+      values: ["choice"],
+    });
+    await broker.actContext(context.ledger_context.id, {
+      kind: "fill",
+      fields: [{ ref: "e3", type: "text", value: "filled" }],
+    });
+    const evaluated = await broker.actContext(context.ledger_context.id, {
+      kind: "evaluate",
+      fn: "() => 42",
+    });
+    const batch = await broker.actContext(context.ledger_context.id, {
+      kind: "batch",
+      actions: [
+        { kind: "press", key: "Enter" },
+        { kind: "wait", selector: "body", url: "example.com", loadState: "load" },
+      ],
+    });
+
+    expect(evaluated.value).toBe(42);
+    expect(batch.value).toMatchObject({
+      results: [
+        { ok: true, kind: "press" },
+        { ok: true, kind: "wait" },
+      ],
+    });
+    expect(transport.sent).toEqual(
+      expect.arrayContaining([
+        expect.objectContaining({ method: "Input.dispatchMouseEvent" }),
+        expect.objectContaining({
+          method: "Runtime.evaluate",
+          params: expect.objectContaining({
+            expression: expect.stringContaining("scrollIntoView"),
+          }),
+        }),
+        expect.objectContaining({
+          method: "Runtime.evaluate",
+          params: expect.objectContaining({
+            expression: expect.stringContaining("DataTransfer"),
+          }),
+        }),
+        expect.objectContaining({
+          method: "Runtime.evaluate",
+          params: expect.objectContaining({
+            expression: expect.stringContaining("filled"),
+          }),
+        }),
+      ]),
+    );
+  });
+
   it("constructs from the SparseKernel daemon client", async () => {
     const calls: Array<{ url: string; body?: unknown }> = [];
     const transport = new FakeCdpTransport();