Repository navigation
feat(agents): add a pi-durable browserTool for a persistent browser #2484
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "agents": minor | ||
| --- | ||
|
|
||
| Add `browserTool` to `agents/browser/pi`, the pi-durable version of the persistent browser tool. Screenshots come back as images the model can see. See [Persistent browser](https://github.com/cloudflare/agents/blob/main/docs/agents/browse-the-web.md#persistent-browser). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| import { | ||
| Type, | ||
| type ImageContent, | ||
| type TextContent | ||
| } from "@earendil-works/pi-ai"; | ||
| import type { | ||
| ToolExecutionResult, | ||
| ToolRegistration | ||
| } from "@earendil-works/pi-durable"; | ||
| import { | ||
| createBrowserToolCore, | ||
| type BrowserToolOptions, | ||
| type BrowserToolOutput | ||
| } from "./browser-tool"; | ||
| import type { BrowserNewTab } from "./session-connector"; | ||
| import { | ||
| browserExecuteModelOutput, | ||
| browserScreenshotOutput | ||
| } from "./tool-helpers"; | ||
|
|
||
| export type { | ||
| BrowserToolInput, | ||
| BrowserToolOptions, | ||
| BrowserToolOutput | ||
| } from "./browser-tool"; | ||
| export type { BrowserNewTab, BrowserSource } from "./session-connector"; | ||
|
|
||
| export interface PiBrowserToolOptions< | ||
| TName extends string = "browser" | ||
| > extends BrowserToolOptions { | ||
| /** The tool's name. Default `"browser"`. */ | ||
| name?: TName; | ||
| } | ||
|
|
||
| const browserToolParameters = Type.Object({ | ||
| code: Type.String({ | ||
| description: "An async arrow function that drives the browser with `cdp`" | ||
| }) | ||
| }); | ||
|
|
||
| /** | ||
| * What a UI can show about a run without reading the content: the run's | ||
| * outcome and what happened to the browser. | ||
| */ | ||
| export type PiBrowserToolDetails = { | ||
| executionId: string; | ||
| status: BrowserToolOutput["status"]; | ||
| restarted?: true; | ||
| newTabs?: { targetId: string; url?: string; title?: string }[]; | ||
| }; | ||
|
|
||
| export type PiBrowserTool<TName extends string = "browser"> = ToolRegistration< | ||
| typeof browserToolParameters, | ||
| PiBrowserToolDetails | ||
| > & { | ||
| readonly name: TName; | ||
| }; | ||
|
|
||
| function details(output: BrowserToolOutput): PiBrowserToolDetails { | ||
| return { | ||
| executionId: output.executionId, | ||
| status: output.status, | ||
| ...(output.restarted ? { restarted: true as const } : {}), | ||
| ...(output.newTabs | ||
| ? { | ||
| newTabs: output.newTabs.map((tab: BrowserNewTab) => ({ | ||
| targetId: tab.targetId, | ||
| ...(tab.url === undefined ? {} : { url: tab.url }), | ||
| ...(tab.title === undefined ? {} : { title: tab.title }) | ||
| })) | ||
| } | ||
| : {}) | ||
| }; | ||
| } | ||
|
|
||
| function text(value: unknown): TextContent { | ||
| return { | ||
| type: "text", | ||
| text: typeof value === "string" ? value : JSON.stringify(value) | ||
| }; | ||
| } | ||
|
|
||
| /** | ||
| * The tool result pi stores and sends: the model projection of the run as | ||
| * JSON (no `calls` log, bounded `logs`), and a screenshot as an image part | ||
| * after it. pi-ai only sends the image to models that accept images. | ||
| */ | ||
| function browserToolResult( | ||
| output: BrowserToolOutput | ||
| ): ToolExecutionResult<PiBrowserToolDetails> { | ||
| const isError = output.status === "error"; | ||
| const screenshot = browserScreenshotOutput(output); | ||
| if (!screenshot) { | ||
| return { | ||
| content: [text(browserExecuteModelOutput(output).value)], | ||
| details: details(output), | ||
| ...(isError ? { isError } : {}) | ||
| }; | ||
| } | ||
|
|
||
| // No size check: a screenshot comes from one cdp.send result, which | ||
| // codemode caps at 1 MB, well under what model providers accept. | ||
| const bytes = Math.floor((screenshot.data.length * 3) / 4).toLocaleString(); | ||
| // Keep the rest of the result (status, restarted, notice, newTabs) around | ||
| // the sentence that replaces the screenshot. | ||
| const summary = browserExecuteModelOutput({ | ||
| ...output, | ||
| result: `Screenshot attached as an image (${screenshot.mediaType}, approximately ${bytes} bytes). If you can't see it, read the page with Runtime.evaluate instead.` | ||
| }).value; | ||
| const image: ImageContent = { | ||
| type: "image", | ||
| data: screenshot.data, | ||
| mimeType: screenshot.mediaType | ||
| }; | ||
| return { | ||
| content: [text(summary), image], | ||
| details: details(output) | ||
| }; | ||
| } | ||
|
|
||
| /** | ||
| * Create a pi-durable tool that lets the model drive a persistent browser | ||
| * with JavaScript and the Chrome DevTools Protocol. | ||
| * | ||
| * Works like `browserTool` in `agents/browser/ai-sdk`: tabs, cookies, and | ||
| * logins carry over between runs, `sessionId: "active"` addresses the tab the | ||
| * model last worked in, and a replaced browser is reported as | ||
| * `restarted: true`. A returned screenshot comes back as an image part that | ||
| * the model sees, when its model accepts images. | ||
| * | ||
| * The tool runs its calls one at a time, since they share the active tab, | ||
| * and doesn't rerun after an eviction (its code may have clicked or | ||
| * submitted something): pi gives the model an interrupted result instead. | ||
| * Pass `ctx` unless the tool is built inside an Agent. | ||
| * | ||
| * @example | ||
| * ```ts | ||
| * import { Browser, browserRun } from "agents/browser"; | ||
| * import { browserTool } from "agents/browser/pi"; | ||
| * | ||
| * export class MyAgent extends DurableObject<Env> { | ||
| * readonly browser = new Browser({ provider: browserRun(this.env.BROWSER) }); | ||
| * readonly registry = createRegistry(); | ||
| * readonly harness = new PiHarness({ | ||
| * harness: ({ storage, context }) => { | ||
| * this.registry.install({ | ||
| * name: "browser", | ||
| * tools: [ | ||
| * browserTool({ | ||
| * ctx: this.ctx, | ||
| * browser: this.browser, | ||
| * loader: this.env.LOADER | ||
| * }) | ||
| * ] | ||
| * }); | ||
| * return Harness.open(storage, { models, registry: this.registry }, context); | ||
| * } | ||
| * }); | ||
| * readonly lifecycle = Lifecycle.install(this) | ||
| * .use(this.browser) | ||
| * .use(this.harness); | ||
| * } | ||
| * ``` | ||
| */ | ||
| export function browserTool<TName extends string = "browser">( | ||
| options: PiBrowserToolOptions<TName> | ||
| ): PiBrowserTool<TName> { | ||
| const core = createBrowserToolCore(options, { | ||
| screenshotHint: | ||
| "To see a screenshot, return { type: 'browser_screenshot', mediaType, data } with data from Page.captureScreenshot and mediaType 'image/png', or 'image/jpeg' if you captured with format: 'jpeg'. The image is attached to the result." | ||
| }); | ||
| return { | ||
| name: options.name ?? ("browser" as TName), | ||
| description: core.description, | ||
| parameters: browserToolParameters, | ||
| // Calls share the active tab, so one round's calls must not interleave. | ||
| executionMode: "sequential", | ||
|
Contributor
There was a problem hiding this comment. 🔴 Concurrent conversations lose their active tab When separate conversations call Learn morePi runs different conversations concurrently; Example: Conversation A creates tab A, and conversation B creates tab B before A's next call. B's pass saves tab B as active. A's Recommended fix: Decide whether the browser is intended to be shared between conversations. If not, assign a distinct named Browser to each conversation. If it is shared, track the active target per conversation and pass the conversation identity into the connector; a global execution lock alone cannot retain separate active tabs between turns. Was this helpful? React with 👍 or 👎 to provide feedback. |
||
| // The code may have clicked or submitted something; don't run it twice. | ||
| replay: "unsafe", | ||
| async execute(args) { | ||
| return browserToolResult(await core.execute({ code: args.code })); | ||
| } | ||
| }; | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.