Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/mcp-fold-unchanged-subtrees.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

When an action changes most of the screen, large subtrees that did not change are shown as one line instead of being repeated.
33 changes: 15 additions & 18 deletions packages/core/src/agent/default-agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import type { ExecutorObservation, ExecutorObserveOptions, ReplayedPrefix, StepE
import { interactiveNodeCount } from './observation.ts';
import { testTitleLine } from './prompts.ts';
import { grammarTools } from './primitives.ts';
import { ScreenPresenter } from './screen-update.ts';
import { boundPresenter, presenterFor, type ScreenPresenter } from './screen-update.ts';
import { compactScreenHistory, compactScreenshotHistory } from './transcript-compaction.ts';
import { createToolLoopExecutor, type ToolLoopHelpers } from './tool-loop.ts';
import { toolAppliesTo, withToolContext } from './tool.ts';
Expand Down Expand Up @@ -51,18 +51,6 @@ const ASSERT_RULES = `You are judging exactly one assertion against the current
- Each scroll result contains a fresh observation. When screenshots are shown, each result includes fresh pixels too.
- Conclude passed when the collected evidence establishes the assertion, failed with ASSERTION_FAILED when it contradicts the assertion, and failed with ASSERTION_INCONCLUSIVE when the available evidence or remaining budget cannot establish an answer.`;

/** One presenter per dispatched step, shared by the opening prompt and the tools that follow it. */
const presenters = new WeakMap<StepExecutorContext, ScreenPresenter>();

function presenterFor(context: StepExecutorContext): ScreenPresenter {
let presenter = presenters.get(context);
if (presenter === undefined) {
presenter = new ScreenPresenter();
presenters.set(context, presenter);
}
return presenter;
}

/** What the built-in agent is built from: an agents entry's `system` and its validated `tools`. */
export interface BuiltInAgentOptions {
/** Appended to the built-in execution rules; read by act and exploring assert loops. */
Expand Down Expand Up @@ -90,6 +78,11 @@ function createActLoop(options: BuiltInAgentOptions): StepExecutor {
prepareMessages: (messages) => compactScreenshotHistory(compactScreenHistory(messages)),
tools: (context, helpers) => loopTools(context, helpers, options.tools),
buildPrompt: async (context) => {
// The run's presenter is read before the first await: another run on
// this context may bind its own presenter while this one waits on
// observe, and the opening prompt must read this run's, not the other
// run's.
const presenter = boundPresenter(context) ?? presenterFor(context);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The reordering that keeps an overlapping run's opening prompt on its own presenter has no regression test. The concurrent-run test uses custom buildPrompt functions, so the built-in act loop's buildPrompt is never exercised with two overlapping runs. Add a test with two createBuiltInAgent runs on one context whose observe resolves at different times, and check each opening prompt.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/core/src/agent/default-agent.ts, line 85:

<comment>The reordering that keeps an overlapping run's opening prompt on its own presenter has no regression test. The concurrent-run test uses custom `buildPrompt` functions, so the built-in act loop's `buildPrompt` is never exercised with two overlapping runs. Add a test with two `createBuiltInAgent` runs on one context whose `observe` resolves at different times, and check each opening prompt.</comment>

<file context>
@@ -78,6 +78,11 @@ function createActLoop(options: BuiltInAgentOptions): StepExecutor {
+      // this context may bind its own presenter while this one waits on
+      // observe, and the opening prompt must read this run's, not the other
+      // run's.
+      const presenter = boundPresenter(context) ?? presenterFor(context);
       // The opening look is tree-only, unless no listed interactive node
       // exists at all: then the opening prompt carries a screenshot, because the tree describes a canvas, a game,
</file context>

// The opening look is tree-only, unless no listed interactive node
// exists at all: then the opening prompt carries a screenshot, because the tree describes a canvas, a game,
// or a semantics-free native screen too poorly to act on, and the model
Expand All @@ -102,7 +95,7 @@ function createActLoop(options: BuiltInAgentOptions): StepExecutor {
const extra: string[] = [];
if (context.replayedPrefix !== undefined) extra.push(formatReplayedPrefix(context.replayedPrefix));
if (context.ledger !== '') extra.push(`Previously completed steps:\n${context.ledger}`);
return openingPrompt(context, `Execute this test step: ${context.step.instruction}`, extra, observation);
return openingPrompt(context, `Execute this test step: ${context.step.instruction}`, extra, observation, presenter);
},
});
}
Expand All @@ -118,8 +111,11 @@ function createAssertionLoop(options: BuiltInAgentOptions): StepExecutor {
system: rules(ASSERT_RULES, options.system),
inconclusiveOnOverflow: true,
tools: (context, helpers) => loopTools(context, helpers, options.tools),
buildPrompt: async (context) =>
openingPrompt(context, `Judge whether this assertion holds: ${context.step.instruction}`, [], await context.observe()),
buildPrompt: async (context) => {
// The run's presenter is read before the first await, as the act loop does.
const presenter = boundPresenter(context) ?? presenterFor(context);
return openingPrompt(context, `Judge whether this assertion holds: ${context.step.instruction}`, [], await context.observe(), presenter);
},
});
return { ...loop, runStep: (context) => loop.runStep(withVision(context)) };
}
Expand Down Expand Up @@ -155,7 +151,7 @@ function loopTools(
): ToolSet {
return {
...guardedTools(helpers, projectTools(context, tools)),
...grammarTools(context, { guard: helpers.guard, screen: presenterFor(context) }),
...grammarTools(context, { guard: helpers.guard }),
};
}

Expand All @@ -165,6 +161,7 @@ function openingPrompt(
lead: string,
extra: readonly string[],
observation: ExecutorObservation,
presenter: ScreenPresenter,
): string | ModelMessage[] {
const title = testTitleLine(context.attempt.titlePath);
const parts = [
Expand All @@ -173,7 +170,7 @@ function openingPrompt(
...(context.step.params === undefined ? [] : [`Step parameters:\n${JSON.stringify(context.step.params)}`]),
...extra,
];
const opening = presenterFor(context).open(observation);
const opening = presenter.open(observation);
if (typeof opening === 'string') return [...parts, opening].join('\n\n');
return [
{
Expand Down
15 changes: 9 additions & 6 deletions packages/core/src/agent/primitives.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import { cacheTokenFields, readCost } from './model/sdk.ts';
import { OperationQueue } from './operation-queue.ts';
import { imagePointToViewport, POINT_VERBS } from './point-tap.ts';
import { SAMPLE_NAMES, SAMPLE_PREFIX } from './sample-files.ts';
import { ScreenPresenter, type ScreenOutput, type ScreenUpdateOptions } from './screen-update.ts';
import { ScreenPresenter, boundPresenter, type ScreenOutput, type ScreenUpdateOptions } from './screen-update.ts';

/** Codes the model may pick when concluding; runtime codes are runtime-assigned. */
const MODEL_ERROR_CODES = [
Expand Down Expand Up @@ -227,9 +227,9 @@ interface GrammarToolSetup extends GrammarToolOptions {
/**
* Renders screens for the model and remembers what it has seen, so every
* result after the first reports the changes rather than the whole tree.
* Shared with the opening prompt by the built-in agent and with the session
* by `e2e mcp`; a fresh one per step otherwise, so the first look
* (`observe`) returns the whole screen.
* Passed by `e2e mcp` to share with the session; without one, the loop
* run's presenter when built inside a loop, else a fresh presenter of its
* own.
*/
readonly screen?: ScreenPresenter;
/**
Expand Down Expand Up @@ -898,7 +898,7 @@ function createPointTools(

/**
* What every verb runs through: the loop guard, the
* step's presenter, one queue for the tool bodies (an action and its look
* loop run's presenter, one queue for the tool bodies (an action and its look
* at the result run together, so a batched turn gets one coherent result
* per action), and the two moves each verb makes: present the screen after
* an action, and perform an action then present. A failed action still
Expand All @@ -912,7 +912,10 @@ function createPointTools(
*/
function verbKit(context: StepExecutorContext, options: GrammarToolSetup) {
const guard = options.guard ?? (<T>(body: () => Promise<T>) => body());
const screen: ScreenPresenter = options.screen ?? new ScreenPresenter();
// No presenter of its own: the loop run's when built inside one, else a
// fresh presenter per toolset, so tools used outside a loop never share
// screen state with a run or with each other.
const screen: ScreenPresenter = options.screen ?? boundPresenter(context) ?? new ScreenPresenter();
const queue = new OperationQueue();
const inOrder = <T>(body: () => Promise<T>): Promise<T> => queue.run(body);
/** The screen after an action: the changes since the one the model holds and, once the step shows pixels, a fresh screenshot. */
Expand Down
Loading
Loading