Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/agent-native-dialogs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@e2e-dev/web": patch
---

Answer native dialogs an agent step's own action opens instead of failing its next action. In `agent.act` and its replays, `e2e explore`, and the action tools of an `e2e mcp` session, an alert, a confirm, and a leave-page dialog with no handler are accepted, and a prompt is dismissed. The app log names each one. A dialog belongs to the newest action before it, so one a test's own step opens still fails the next step until a `browser.onDialog` handler takes it, even when an agent step reads the screen in between.
5 changes: 5 additions & 0 deletions .changeset/operation-agent-step.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

`OperationContext.agentStep` tells an engine that an operation belongs to an agent step, including an action the replay cache repeats with the `test` origin. The web engine uses it to answer native dialogs for the agent.
12 changes: 9 additions & 3 deletions apps/web-benchmark/tests-agent/scenarios.e2e.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
* accessibility tree. Each step gets the catalog description as its goal and
* reads the rest off the page; the deterministic check on the scenario's
* success message decides the test. Scenarios that need pixels (canvas, image
* and CSS-content UIs, the aria-hidden flow, the onboarding wizard), native
* dialogs, a second tab, or a file the agent first downloads are left out:
* the grammar has no verb for them yet.
* and CSS-content UIs, the aria-hidden flow, the onboarding wizard), a typed
* answer to a native prompt, a second tab, or a file the agent first
* downloads are left out: the grammar has no verb for them yet.
*/

import { test } from '@e2e-dev/web';
Expand Down Expand Up @@ -130,6 +130,12 @@ const SCENARIOS: readonly Scenario[] = [
goal: 'buy a gift card for friend@example.com and complete the purchase',
success: 'Gift card sent to friend@example.com',
},
{
// The confirm opens 900 ms after the click, and the engine accepts it for the agent, replay included.
slug: 'deferred-dialog',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This scenario has no committed recording under apps/web-benchmark/.e2e/cache/. In CI the step will call the model instead of replaying, so the suite loses its deterministic, no-cost gate and fails on PRs without the key. Re-record it locally with test:agent and commit the new cache entry in this PR.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At apps/web-benchmark/tests-agent/scenarios.e2e.ts, line 135:

<comment>This scenario has no committed recording under `apps/web-benchmark/.e2e/cache/`. In CI the step will call the model instead of replaying, so the suite loses its deterministic, no-cost gate and fails on PRs without the key. Re-record it locally with `test:agent` and commit the new cache entry in this PR.</comment>

<file context>
@@ -130,6 +130,12 @@ const SCENARIOS: readonly Scenario[] = [
   },
+  {
+    // The confirm opens 900 ms after the click, and the engine accepts it for the agent, replay included.
+    slug: 'deferred-dialog',
+    goal: 'submit the report and accept the confirmation that follows',
+    success: 'Report submitted after deferred confirmation',
</file context>

goal: 'submit the report and accept the confirmation that follows',
success: 'Report submitted after deferred confirmation',
},
];

async function complete(
Expand Down
10 changes: 8 additions & 2 deletions docs/browser.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ expect(response.status).toBe(201);

## Handle a dialog

An unhandled `alert`, `confirm`, or `prompt` fails the next step. Register a
handler first:
An unhandled `alert`, `confirm`, `prompt`, or leave-page dialog that a test's
own step opens fails the next step. Register a handler first:

```ts
const off = await browser.onDialog('accept');
Expand All @@ -74,6 +74,12 @@ await off();
A function handler gets the dialog and must call `accept(text?)` or
`dismiss()`.

The agent can't see native dialogs, so one its own action opens is answered
for it when no handler is registered. This covers `agent.act` and its replays,
`e2e explore`, and the action tools of an `e2e mcp` session. An alert, a
confirm, and a leave-page dialog are accepted, and a prompt is dismissed. The
app log names each one.

## Download a file

```ts
Expand Down
8 changes: 8 additions & 0 deletions docs/reference/engine.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,14 @@ export interface OperationContext {
* transition to end first.
*/
readonly origin: 'test' | 'agent';
/**
* The operation is part of an agent step: the agent chose it just now, or
* the replay cache repeats a choice it made in an earlier run, whose
* `origin` is then `test`. An engine answers for the agent what the agent
* cannot see, such as a native dialog its action opens.
* When absent, use `origin === 'agent'`: older cores send only `origin`.
*/
readonly agentStep?: boolean;
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
}
```

Expand Down
6 changes: 6 additions & 0 deletions docs/reference/mcp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,12 @@ changed on screen.
`secrets` and the engine declares the `fill` action. The plaintext never
reaches the agent as text; see
[screenshots after a secret fill](/reference/security#screenshots-after-a-secret-fill).
- A native dialog an action opens is answered as in `agent.act`: an alert, a
confirm, and a leave-page dialog are accepted, and a prompt is dismissed.
The result doesn't say so, and the test line registers no handler, so a
test of that flow needs `browser.onDialog` first. `run_code` runs test
statements, so a dialog one opens fails the next statement, or the next
call when none follows.

#### `dismiss_keyboard`

Expand Down
8 changes: 7 additions & 1 deletion docs/reference/web.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -857,6 +857,12 @@ await off();
`Dialog`.
- A function handler must call `accept` or `dismiss`. One that does neither
has the dialog dismissed, as an unhandled dialog does.
- A dialog no handler takes is dismissed and fails the next step, or the
attempt at its end when no step follows, unless an agent step's own action
opened it, replayed from the cache or not. Then
it's answered for the agent: an alert, a confirm, and a leave-page dialog
are accepted, and a prompt is dismissed. The app log names each one. A
dialog belongs to the newest action before it, never to a read.
- A handler that throws fails the next step with its error, or the attempt
at its end when no step follows. One that throws before answering has its
dialog dismissed.
Expand All @@ -865,7 +871,7 @@ The errors surface on the next step, not on `onDialog`:

| Throws | When |
| --- | --- |
| `APP_NOT_OPEN` | A function handler answered neither way, or no handler took the dialog. |
| `INVALID_STATE` | A function handler answered neither way, or no handler took a dialog a test's own step opened. |
| `ASSERTION_FAILED` | The handler's `expect` failed. |
| `ENGINE_FAILURE` | The handler threw an unclassified error. |

Expand Down
11 changes: 6 additions & 5 deletions packages/e2e/src/agent/phases.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,18 @@ import type { StepActivity, StepEvent, StepRecorder } from '../run/steps.ts';
import { AgentError, toAgentError } from './error.ts';

/**
* One engine operation's budget: `actionTimeout`, capped by the step clock.
* Bounding each call independently is what keeps a single screen that never
* settles from consuming the whole step: the hang costs one action timeout and
* a clearly attributed failure, not the test budget.
* One engine operation of an agent step, with its budget: `actionTimeout`,
* capped by the step clock. Bounding each call independently is what keeps a
* single screen that never settles from consuming the whole step: the hang
* costs one action timeout and a clearly attributed failure, not the test
* budget.
*/
export function boundedOperation(
engine: LocatorEngine,
actionTimeout: number,
deadline: Deadline,
): OperationContext {
return { ...engine.operation(Math.max(1, Math.min(actionTimeout, deadline.remaining()))), origin: 'agent' };
return { ...engine.operation(Math.max(1, Math.min(actionTimeout, deadline.remaining()))), origin: 'agent', agentStep: true };
}

/** Records one policy decision as a child event of the current step. */
Expand Down
2 changes: 1 addition & 1 deletion packages/e2e/src/agent/step-accounting.ts
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ export class StepAccounting {
);
}

/** Every operation inside a replayed span is the test's, navigation included; outside it, the agent's. */
/** Every operation inside a replayed span acts with the test's origin, navigation included, and is still the agent step's; outside it, the agent's. */
private stamped(operation: OperationContext): OperationContext {
return this.inReplay ? { ...operation, origin: 'test' } : operation;
}
Expand Down
8 changes: 8 additions & 0 deletions packages/e2e/src/engine/contract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,14 @@ export interface OperationContext {
* transition to end first.
*/
readonly origin: 'test' | 'agent';
/**
* The operation is part of an agent step: the agent chose it just now, or
* the replay cache repeats a choice it made in an earlier run, whose
* `origin` is then `test`. An engine answers for the agent what the agent
* cannot see, such as a native dialog its action opens.
* When absent, use `origin === 'agent'`: older cores send only `origin`.
*/
readonly agentStep?: boolean;
}

export type TextPattern =
Expand Down
1 change: 1 addition & 0 deletions packages/e2e/src/engine/protocol/client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -584,6 +584,7 @@ function encodeOperation(link: Link, operation: OperationContext): WireOperation
runId: operation.runId,
attemptId: operation.attemptId,
origin: operation.origin,
...(operation.agentStep === true ? { agentStep: true as const } : {}),
};
}

Expand Down
1 change: 1 addition & 0 deletions packages/e2e/src/engine/protocol/protocol.ts
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ export interface WireOperation {
readonly runId: string;
readonly attemptId: string;
readonly origin: 'test' | 'agent';
readonly agentStep?: true;
}

/** `EngineCleanupContext` on the wire. */
Expand Down
1 change: 1 addition & 0 deletions packages/e2e/src/engine/protocol/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,7 @@ class EngineServer {
runId: wire.runId,
attemptId: wire.attemptId,
origin: wire.origin,
...(wire.agentStep === true ? { agentStep: true } : {}),
};
}

Expand Down
10 changes: 10 additions & 0 deletions packages/e2e/tests/unit/engine-protocol.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,16 @@ describe('engine protocol', () => {
expect(Object.keys(remote.session ?? {}).toSorted()).toEqual(['open', 'reset']);
});

it("carries whether an operation is an agent step's, and leaves it out when it is not", async () => {
const received: OperationContext[] = [];
const { remote } = overLoopback(toy({ actions: ['tap'], perform: async (_ref, _action, seen) => void received.push(seen) }));
await remote.init!(initInfo());
await remote.perform!({ id: 'n1', revision: '' }, { kind: 'tap' }, { ...operation(), agentStep: true });
await remote.perform!({ id: 'n1', revision: '' }, { kind: 'tap' }, operation());
expect(received.map((seen) => [seen.origin, seen.agentStep])).toEqual([['test', true], ['test', undefined]]);
expect('agentStep' in received[1]!).toBe(false);
});

it('carries a snapshot with its pixels byte for byte', async () => {
const data = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0, 255, 7]);
const { remote } = overLoopback(
Expand Down
6 changes: 5 additions & 1 deletion packages/e2e/tests/unit/step-accounting.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ function runtime(): AgentContext & { steps: StepRecorder } {
}

describe('step accounting operation contexts', () => {
it('marks actions as the agent\'s, and as the test\'s while a trace replays', () => {
it('marks actions as the agent\'s, and as the test\'s origin while a trace replays, always within the agent step', () => {
const accounting = new StepAccounting(runtime(), {
api: 'agent.act',
timeoutMs: 10_000,
Expand All @@ -37,11 +37,15 @@ describe('step accounting operation contexts', () => {
expect(accounting.replayingTrace).toBe(false);
expect(accounting.actionOperation().origin).toBe('agent');
expect(accounting.operation().origin).toBe('agent');
expect(accounting.actionOperation().agentStep).toBe(true);
accounting.replaying(true);
// Both contexts: targeted actions take actionOperation(), navigation takes operation().
expect(accounting.replayingTrace).toBe(true);
expect(accounting.actionOperation().origin).toBe('test');
expect(accounting.operation().origin).toBe('test');
// A replayed action is still the agent step's, so an engine answers what the agent cannot see for it.
expect(accounting.actionOperation().agentStep).toBe(true);
expect(accounting.operation().agentStep).toBe(true);
accounting.replaying(false);
expect(accounting.replayingTrace).toBe(false);
expect(accounting.actionOperation().origin).toBe('agent');
Expand Down
24 changes: 12 additions & 12 deletions packages/web/src/browser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -461,7 +461,7 @@ export function createBrowserFixture(surface: PlaywrightSurface, context: Engine
run: (operation: OperationContext) => Promise<void>,
): Promise<void> => {
const operation = context.operation(stepTimeout(options?.timeout, api) ?? context.timeouts.test);
return surface.guard(operation, 'navigation', run);
return surface.guardAction(operation, 'navigation', run);
};

/** An assertion-style budget: the given timeout or the assertion timeout, clamped to the test. */
Expand Down Expand Up @@ -519,7 +519,7 @@ export function createBrowserFixture(surface: PlaywrightSurface, context: Engine
const input: unknown = arg;
// A handle, not a value: the boundary resolves to a string, which Playwright
// keeps out of its in-page serializer when it is taken back from a handle.
const report = await surface.guard(context.operation(), 'evaluate', async (operation) => {
const report = await surface.guardAction(context.operation(), 'evaluate', async (operation) => {
const page = surface.requirePage();
try {
const handle = await page.evaluateHandle(evaluate, input);
Expand Down Expand Up @@ -743,7 +743,7 @@ export function createBrowserFixture(surface: PlaywrightSurface, context: Engine
return surface.guard(context.operation(), 'clearCookies', () => surface.requireContext().clearCookies(options));
},
setViewport: (size) =>
surface.guard(context.operation(), 'setViewport', async () => {
surface.guardAction(context.operation(), 'setViewport', async () => {
await surface.setViewport(size);
context.attachViewport({ width: size.width, height: size.height, scale: 1 });
}),
Expand All @@ -758,7 +758,7 @@ export function createBrowserFixture(surface: PlaywrightSurface, context: Engine
}
// Read once, so a test that changes its object later changes nothing here.
const requested = obj({ colorScheme: media.colorScheme, reducedMotion: media.reducedMotion });
await surface.guard(context.operation(), 'emulateMedia', () => surface.emulateMedia(requested));
await surface.guardAction(context.operation(), 'emulateMedia', () => surface.emulateMedia(requested));
},
// Async so the harness records the registration as a `browser.onDialog` step.
async onDialog(handler: DialogHandler) {
Expand All @@ -779,21 +779,21 @@ export function createBrowserFixture(surface: PlaywrightSurface, context: Engine
},
keyboard: {
down: (key) =>
surface.guard(context.operation(), 'keyboard.down', () => surface.requirePage().keyboard.down(key), classifyInputError),
surface.guardAction(context.operation(), 'keyboard.down', () => surface.requirePage().keyboard.down(key), classifyInputError),
up: (key) =>
surface.guard(context.operation(), 'keyboard.up', () => surface.requirePage().keyboard.up(key), classifyInputError),
surface.guardAction(context.operation(), 'keyboard.up', () => surface.requirePage().keyboard.up(key), classifyInputError),
press: (key) =>
surface.guard(context.operation(), 'keyboard.press', () => surface.requirePage().keyboard.press(key), classifyInputError),
surface.guardAction(context.operation(), 'keyboard.press', () => surface.requirePage().keyboard.press(key), classifyInputError),
type: (text) =>
surface.guard(context.operation(), 'keyboard.type', () => surface.requirePage().keyboard.type(text), classifyInputError),
surface.guardAction(context.operation(), 'keyboard.type', () => surface.requirePage().keyboard.type(text), classifyInputError),
},
mouse: {
move: (x, y) =>
surface.guard(context.operation(), 'mouse.move', () => surface.requirePage().mouse.move(x, y), classifyInputError),
surface.guardAction(context.operation(), 'mouse.move', () => surface.requirePage().mouse.move(x, y), classifyInputError),
wheel: (deltaX, deltaY) =>
surface.guard(context.operation(), 'mouse.wheel', () => surface.requirePage().mouse.wheel(deltaX, deltaY), classifyInputError),
down: () => surface.guard(context.operation(), 'mouse.down', () => surface.requirePage().mouse.down(), classifyInputError),
up: () => surface.guard(context.operation(), 'mouse.up', () => surface.requirePage().mouse.up(), classifyInputError),
surface.guardAction(context.operation(), 'mouse.wheel', () => surface.requirePage().mouse.wheel(deltaX, deltaY), classifyInputError),
down: () => surface.guardAction(context.operation(), 'mouse.down', () => surface.requirePage().mouse.down(), classifyInputError),
up: () => surface.guardAction(context.operation(), 'mouse.up', () => surface.requirePage().mouse.up(), classifyInputError),
},
};

Expand Down
52 changes: 49 additions & 3 deletions packages/web/src/dialogs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,17 @@
* Native dialog routing for one attempt. Handlers are registered by the `web`
* fixture; the surface owns the router so a dialog that fires before a test
* ever touches `web` is still accounted for instead of silently dismissed.
*
* A dialog no handler takes fails the test's next step, unless the agent
* acted last: native dialogs are outside the page, so the agent never sees
* one and cannot answer it. The router answers it for the agent the way the
* action that opened it asked (an alert or a confirm accepted, a page left
* when the agent navigated away from unsaved changes) and tells the app log.
* A prompt asks for text the agent never sees, so it is dismissed.
*/

import type { Dialog as PwDialog } from 'playwright-core';
import { EngineError } from 'e2e/engine';
import { EngineError, type AppLogEntry } from 'e2e/engine';
import { ErrorLatch, isClassified, message } from './support.ts';

/** A native dialog as a test's handler sees it. */
Expand All @@ -26,14 +33,26 @@ interface Registration {
readonly handler: DialogHandler;
}

/** What the router asks its surface about a dialog no handler takes. */
export interface UnhandledDialogs {
/** Whether the newest operation on the page was the agent's. */
readonly agentActed: () => boolean;
/** The attempt's app log, told each dialog answered for the agent. */
readonly log: (entry: AppLogEntry) => void;
}

export class DialogRouter {
private registrations: Registration[] = [];

/**
* `latch` is shared with the surface: a dialog failure and a route-handler
* failure surface through one check at the next step.
* failure surface through one check at the next step. Without `unhandled`,
* every dialog no handler takes fails the next step.
*/
constructor(private readonly latch: ErrorLatch = new ErrorLatch()) {}
constructor(
private readonly latch: ErrorLatch = new ErrorLatch(),
private readonly unhandled?: UnhandledDialogs,
) {}

/**
* Registers a handler (the newest wins) and returns its idempotent
Expand Down Expand Up @@ -62,6 +81,10 @@ export class DialogRouter {

private async route(dialog: PwDialog): Promise<void> {
const handler = this.registrations.at(-1)?.handler;
if (handler === undefined && this.unhandled?.agentActed() === true) {
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
await this.answerForAgent(dialog, this.unhandled);
return;
}
if (handler === undefined) {
this.latch.latch(
new EngineError(
Expand Down Expand Up @@ -122,4 +145,27 @@ export class DialogRouter {
if (!answered) await dialog.dismiss().catch(() => undefined);
}
}

/**
* Answers a dialog the agent's action opened and names it in the app log.
* A dialog the page closed first, or a page that went away, leaves nothing
* to answer and nothing to tell.
*/
private async answerForAgent(dialog: PwDialog, unhandled: UnhandledDialogs): Promise<void> {
const type = dialog.type();
const accepted = type !== 'prompt';
try {
if (accepted) await dialog.accept();
else await dialog.dismiss();
} catch {
return;
}
const text = dialog.message().replace(/\s+/g, ' ').trim();
const quoted = text === '' ? '' : ` ${JSON.stringify(text)}`;
unhandled.log({
source: 'system',
level: 'info',
text: `${type} dialog${quoted} ${accepted ? 'accepted' : 'dismissed'} for the agent`,
});
}
}
Loading
Loading