From 33ba409893cfea10f538130519604e94b615db47 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 19:52:02 +0000 Subject: [PATCH 01/13] Name the agent tools as upstream: complete_step, type, type_secret, and check with checked (#1630) Port upstream's agent-act "offers the default toolset and concludes through complete_step" on the web engine, with the full pointer, upload, resize_viewport and type toolset and the screen-changes result. The port's tool names now match upstream's: done is complete_step, fill is type, fill_secret is type_secret, and check takes checked in place of the uncheck and clear tools. type_secret is offered only when the step declares secrets. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1xnBhxbpAgMwCuPBUA --- .../src/content/docs/api/E2E/AgentTools.md | 7 ++- .../src/content/docs/api/E2E/DefaultAgent.md | 6 +-- .../content/docs/api/E2E/ExploreOptions.md | 2 +- .../content/docs/api/E2E/PreparedMessages.md | 4 +- .../src/content/docs/api/E2E/ToolLoop.md | 2 +- .../docs/api/E2E/ToolLoopExecutorOptions.md | 2 +- .../src/content/docs/api/E2E/VerdictTool.md | 4 +- docs-site/src/content/docs/api/E2E/index.md | 4 +- docs-site/src/content/docs/coding-agents.mdx | 2 +- docs-site/src/content/docs/features/acp.mdx | 6 +-- .../src/content/docs/reference/engine.mdx | 5 +- .../src/content/docs/reference/errors.mdx | 2 +- skills/e2e-dotnet/SKILL.md | 2 +- skills/e2e-dotnet/references/agent.md | 43 ++++++++------- skills/e2e-dotnet/references/bug-bash.md | 2 +- skills/e2e-dotnet/references/debugging.md | 4 +- src/E2E.Acp/AcpExecutor.cs | 2 +- src/E2E.Acp/StepTools.cs | 2 +- src/E2E.Cli/Mcp/SessionCatalog.cs | 19 ++++--- src/E2E/Agent/ActContext.cs | 2 +- src/E2E/Agent/ActTools.cs | 29 +++++----- src/E2E/Agent/ActionNames.cs | 4 +- src/E2E/Agent/Agent.cs | 4 +- src/E2E/Agent/AgentTools.cs | 54 +++++++++---------- src/E2E/Agent/DefaultAgent.cs | 8 +-- src/E2E/Agent/LoopGuards.cs | 2 +- src/E2E/Agent/Model.cs | 6 +-- src/E2E/Agent/Models/ToolChoice.cs | 2 +- src/E2E/Agent/ToolLoop.cs | 38 ++++++------- src/E2E/Explore/Body.cs | 2 +- src/E2E/Explore/Executor.cs | 8 ++- src/E2E/Explore/Explore.cs | 2 +- tests/E2E.Acp.Tests/Executor/AcpTests.cs | 34 ++++++------ .../Presets/AgentPresetsTests.cs | 2 +- ...conclusiveJudgmentAfterASecretFillTests.cs | 4 +- .../AgentAct/AgentActVerdictMappingTests.cs | 6 +-- ...ithTheDefaultToolLoopAgentExecutorTests.cs | 50 +++++++++-------- .../AgentAct/LoopGuardsAndTranscriptsTests.cs | 2 +- tests/E2E.Tests/AgentAct/TabsEngine.cs | 54 ------------------- .../AgentActPixelVerbsTests.Points.cs | 12 ++--- ...SecretFillPolicyUnderAHostileModelTests.cs | 6 +-- .../AgentAssertExploreTests.cs | 2 +- .../TheBuiltInAgentOptionsTests.cs | 13 ++--- .../AttachedTurnsTests.cs | 4 +- .../CreateGrammarToolsTests.cs | 2 +- .../TheGrammarToolsHaveClosedSchemasTests.cs | 47 ++++++++-------- .../ToolCallsOutsideTheVocabularyTests.cs | 8 +-- .../TapModifiersInTheGrammarToolsTests.cs | 2 +- tests/E2E.Tests/Explore/E2eExploreTests.cs | 6 +-- .../LoopGuards/FailureStreakTests.cs | 20 ++----- .../ModelRequestHeadersTests.cs | 2 +- .../SecondTurnOfAnOpenAiShapedStepTests.cs | 2 +- .../CreateToolLoopExecutorTests.cs | 4 +- .../ToolLoopPrefix/ToolLoopPrefixHelpers.cs | 2 +- ...dConclusionAnsweredWithAnotherToolTests.cs | 6 +-- .../ToolLoopForcedToolChoiceTests.cs | 4 +- .../TraceCacheReplay/TraceCacheFlows.cs | 2 +- .../TraceReplay/ReplayThroughTheActTests.cs | 2 +- ...SecretsCutShortByObservationLimitsTests.cs | 2 +- 59 files changed, 252 insertions(+), 329 deletions(-) delete mode 100644 tests/E2E.Tests/AgentAct/TabsEngine.cs diff --git a/docs-site/src/content/docs/api/E2E/AgentTools.md b/docs-site/src/content/docs/api/E2E/AgentTools.md index f04a775de..0d891fb27 100644 --- a/docs-site/src/content/docs/api/E2E/AgentTools.md +++ b/docs-site/src/content/docs/api/E2E/AgentTools.md @@ -34,13 +34,12 @@ The built-in agent's tool vocabulary (upstream's `agent/primitives.ts`). `Create ```csharp -public const string VerdictRules = "Finish with done when the step's goal is achieved, or you are certain it cannot be. done status is passed, failed, or blocked. blocked means credentials, the environment, or test setup prevented a verdict, and requires a code." +public const string VerdictRules = "Finish with complete_step when the step's goal is achieved, or you are certain it cannot be. complete_step status is passed, failed, or blocked. blocked means credentials, the environment, or test setup prevented a verdict, and requires a code." ``` Upstream's `VERDICT_RULES`: the guidance for [VerdictTool](/api/E2E/VerdictTool/). The tool loop appends it to the instructions of every -executor it builds, so an executor outside the loop gives it to its own model beside the tool. The port's tool is `done` -where upstream's is `complete_step`, and takes `code` where upstream's takes `errorCode`. +executor it builds, so an executor outside the loop gives it to its own model beside the tool. The tool takes `code` where upstream's takes `errorCode`. **Value:** [string](https://learn.microsoft.com/dotnet/api/system.string) @@ -83,7 +82,7 @@ public static VerdictTool CreateVerdictTool(ExecutorStepKind? kind = null) ``` -Upstream's `createVerdictTool`: the `done` tool with the closed blocked-code policy, and the verdict it collected. Make a +Upstream's `createVerdictTool`: the `complete_step` tool with the closed blocked-code policy, and the verdict it collected. Make a fresh one for each step: the first accepted verdict is final. **Parameters** diff --git a/docs-site/src/content/docs/api/E2E/DefaultAgent.md b/docs-site/src/content/docs/api/E2E/DefaultAgent.md index 97936da86..d2681de52 100644 --- a/docs-site/src/content/docs/api/E2E/DefaultAgent.md +++ b/docs-site/src/content/docs/api/E2E/DefaultAgent.md @@ -37,8 +37,8 @@ public const string AssertRules = "You are judging exactly one assertion against ``` -Assertion-specific rules (upstream's `ASSERT_RULES`): they allow evidence gathered across several viewports. The port's verdict tool is `done`, -which takes `code`. +Assertion-specific rules (upstream's `ASSERT_RULES`): they allow evidence gathered across several viewports. The verdict tool takes `code` +where upstream's takes `errorCode`. **Value:** [string](https://learn.microsoft.com/dotnet/api/system.string) @@ -48,7 +48,7 @@ which takes `code`. ```csharp -public const string BaseRules = "You are an e2e test agent. Act only by calling tools. The screen is a tree of nodes with stable ids like \"n42\", listed one per line as #id role \"name\": a node keeps its id for as long as it exists, across every screen and change in this conversation. Target a node by its id (ref), with or without the #. Never invent ids. The first screen is sent whole. Every action result and every observe then reports what changed since the screen you last received, one line per node: \"added\" (a new node), \"changed\" (with what it read before), or \"removed\" (the node is gone; never target it again). A node not listed as removed is still there under the id you have. When most of the screen changed, the result sends the whole new screen instead, headed \"Current screen\", and it replaces what you had: target only the ids it lists. Every action result already waited for the effect and contains the changes, so do not call observe after an action. Call observe only after waiting for something the last result showed in progress. Conclude passed only when the latest result shows the outcome the step asked for. A result that reports no change, or only the keyboard closing, means the decisive action did not land: act again or conclude failed. A result that says only that no listed node changed has not ruled out a drawn change: take a screenshot, when offered, before deciding. Never describe an unchanged screen as the outcome. Never type a secret value. Call fill_secret with the secret name. A secret looks like . If what you need is not on the screen, scroll to it. screenshot attaches the viewport's pixels when the tree lacks what you need (a shape on a canvas, a pin on a map, a region of an image, a control inside a system sheet) or contradicts what you expect; once you have one, every action result carries a fresh screenshot so you can see what the action did. tap_at, hover_at, drag_at, type_at, press_at, and select_at act at a point in the latest screenshot's pixel coordinates. tap_at, hover_at, and drag_at act on the bare points where the engine can, and on the listed controls under them by id where it cannot; type_at, press_at, and select_at act on a listed control under the point by its id. Prefer ids: tap, fill, press, and select by id whenever the screen lists the target, use the point tools only for a target the screen does not list, and take a screenshot before guessing a point rather than aiming from memory or from what you expect to be drawn." +public const string BaseRules = "You are an e2e test agent. Act only by calling tools. The screen is a tree of nodes with stable ids like \"n42\", listed one per line as #id role \"name\": a node keeps its id for as long as it exists, across every screen and change in this conversation. Target a node by its id (ref), with or without the #. Never invent ids. The first screen is sent whole. Every action result and every observe then reports what changed since the screen you last received, one line per node: \"added\" (a new node), \"changed\" (with what it read before), or \"removed\" (the node is gone; never target it again). A node not listed as removed is still there under the id you have. When most of the screen changed, the result sends the whole new screen instead, headed \"Current screen\", and it replaces what you had: target only the ids it lists. Every action result already waited for the effect and contains the changes, so do not call observe after an action. Call observe only after waiting for something the last result showed in progress. Conclude passed only when the latest result shows the outcome the step asked for. A result that reports no change, or only the keyboard closing, means the decisive action did not land: act again or conclude failed. A result that says only that no listed node changed has not ruled out a drawn change: take a screenshot, when offered, before deciding. Never describe an unchanged screen as the outcome. Never type a secret value. Call type_secret with the secret name. A secret looks like . If what you need is not on the screen, scroll to it. screenshot attaches the viewport's pixels when the tree lacks what you need (a shape on a canvas, a pin on a map, a region of an image, a control inside a system sheet) or contradicts what you expect; once you have one, every action result carries a fresh screenshot so you can see what the action did. tap_at, hover_at, drag_at, type_at, press_at, and select_at act at a point in the latest screenshot's pixel coordinates. tap_at, hover_at, and drag_at act on the bare points where the engine can, and on the listed controls under them by id where it cannot; type_at, press_at, and select_at act on a listed control under the point by its id. Prefer ids: tap, type, press, and select by id whenever the screen lists the target, use the point tools only for a target the screen does not list, and take a screenshot before guessing a point rather than aiming from memory or from what you expect to be drawn." ``` diff --git a/docs-site/src/content/docs/api/E2E/ExploreOptions.md b/docs-site/src/content/docs/api/E2E/ExploreOptions.md index 451794c0a..392628c67 100644 --- a/docs-site/src/content/docs/api/E2E/ExploreOptions.md +++ b/docs-site/src/content/docs/api/E2E/ExploreOptions.md @@ -118,7 +118,7 @@ public IReadOnlyDictionary? Credentials { get; init; } The accounts the explorer can sign in with, by name. Each travels into every step as the username and a password secret -named `.password`, which the explorer fills with `fill_secret`; the password never reaches a model. +named `.password`, which the explorer fills with `type_secret`; the password never reaches a model. The config's `credentials` are always accounts, as upstream carries them; this adds accounts the config does not declare (.NET-only: `e2e explore` passes the `E2E_USER_*` pairs of the environment). diff --git a/docs-site/src/content/docs/api/E2E/PreparedMessages.md b/docs-site/src/content/docs/api/E2E/PreparedMessages.md index fcba453c4..382b6f705 100644 --- a/docs-site/src/content/docs/api/E2E/PreparedMessages.md +++ b/docs-site/src/content/docs/api/E2E/PreparedMessages.md @@ -20,7 +20,7 @@ public sealed record PreparedMessages : IEquatable What [PrepareMessages](/api/E2E/ToolLoopExecutorOptions/#E2E_ToolLoopExecutorOptions_PrepareMessages) hands back: the history for the next turn, and optionally a reason to -stop. The executor has evidence the loop guards cannot see that the step is not progressing. A stop leaves the model only `done`, like a loop guard. +stop. The executor has evidence the loop guards cannot see that the step is not progressing. A stop leaves the model only `complete_step`, like a loop guard. **Inheritance:** [object](https://learn.microsoft.com/dotnet/api/system.object) @@ -42,7 +42,7 @@ public PreparedMessages(IReadOnlyList Messages, string? Stop = nul What [PrepareMessages](/api/E2E/ToolLoopExecutorOptions/#E2E_ToolLoopExecutorOptions_PrepareMessages) hands back: the history for the next turn, and optionally a reason to -stop. The executor has evidence the loop guards cannot see that the step is not progressing. A stop leaves the model only `done`, like a loop guard. +stop. The executor has evidence the loop guards cannot see that the step is not progressing. A stop leaves the model only `complete_step`, like a loop guard. **Parameters** diff --git a/docs-site/src/content/docs/api/E2E/ToolLoop.md b/docs-site/src/content/docs/api/E2E/ToolLoop.md index e2ec400b5..d8f97f5aa 100644 --- a/docs-site/src/content/docs/api/E2E/ToolLoop.md +++ b/docs-site/src/content/docs/api/E2E/ToolLoop.md @@ -20,7 +20,7 @@ public static class ToolLoop Upstream's `createToolLoopExecutor` (`agent/tool-loop.ts`): the chassis of an executor that thinks in a loop of model -turns and tool calls. It owns everything but the tool vocabulary: the `done` verdict tool and its closed code policy, the +turns and tool calls. It owns everything but the tool vocabulary: the `complete_step` verdict tool and its closed code policy, the hard stops (a spent budget, a timeout, or a cancel ends the loop, never the model), the loop guards, the shrink of a history that overflowed the model, and the count of model calls. The built-in agent is this chassis plus the grammar tools ([CreateBuiltInAgent(BuiltInAgentOptions)](/api/E2E/DefaultAgent/#E2E_DefaultAgent_CreateBuiltInAgent_E2E_BuiltInAgentOptions_)). An executor for a device or an API brings other tools and inherits the same discipline. diff --git a/docs-site/src/content/docs/api/E2E/ToolLoopExecutorOptions.md b/docs-site/src/content/docs/api/E2E/ToolLoopExecutorOptions.md index 620a59cd3..4a4d4abe9 100644 --- a/docs-site/src/content/docs/api/E2E/ToolLoopExecutorOptions.md +++ b/docs-site/src/content/docs/api/E2E/ToolLoopExecutorOptions.md @@ -162,7 +162,7 @@ public required Func DefaultAgent.BaseRules + "\n\nCheck the inbox with read_inbox when An executor that drives a model outside the loop gets the built-in verdict from `AgentTools.CreateVerdictTool(context.Step.Kind)`: give the model its `Tool` and -`AgentTools.VerdictRules`, run each `done` call with `Execute(call)`, and return -`Verdict` once `Concluded`. The tool is `done`, and it takes `code`. +`AgentTools.VerdictRules`, run each `complete_step` call with `Execute(call)`, and return +`Verdict` once `Concluded`. The tool is `complete_step`, and it takes `code`. The observation's `KeyboardVisible` says whether an on-screen keyboard is showing, and is `null` when the engine can't tell. diff --git a/skills/e2e-dotnet/references/bug-bash.md b/skills/e2e-dotnet/references/bug-bash.md index c8b91d099..0addfdfd2 100644 --- a/skills/e2e-dotnet/references/bug-bash.md +++ b/skills/e2e-dotnet/references/bug-bash.md @@ -53,7 +53,7 @@ prove each bug. Report confirmed bugs only, each with its failing test. replay for any confirmed bug. - The port has no saved sessions: `e2e explore --session` exits 2, so an explorer starts signed out. A charter that needs an account names its - credential and the explorer signs in with `fill_secret` (topic `explore`). + credential and the explorer signs in with `type_secret` (topic `explore`). A bug-bash config, left untracked, is a copy of the project's `e2e.config.json` with what a bug bash needs changed. JSON has no spread, so diff --git a/skills/e2e-dotnet/references/debugging.md b/skills/e2e-dotnet/references/debugging.md index 2d4053099..c920c85d3 100644 --- a/skills/e2e-dotnet/references/debugging.md +++ b/skills/e2e-dotnet/references/debugging.md @@ -160,7 +160,7 @@ characters) and what came back (cut to 600), with the loop's own notes after them (`[loop] turn 2 made no tool call: asking for one`). The loop-guard notes are among them: `[loop] loop guard warned before turn 5: ...` when the loop warned the model it was going in circles, and `[loop] the step was stopped -before turn 7: ...` when it left the model only `done`. The built-in agent and +before turn 7: ...` when it left the model only `complete_step`. The built-in agent and any `ToolLoop.CreateToolLoopExecutor` executor record them. An executor of your own records them with `AttachTurns`. @@ -262,7 +262,7 @@ failed before it read a screen has `observationRevision` `unrecorded`. See | `EVALUATE_FAILED` | `Browser.EvaluateAsync` threw inside the page, or the page deleted or replaced a JavaScript built-in (such as `Map`) that the engine needs, which fails later evaluates and locators | Fix the page code; do not replace built-ins the page shares with automation | | `ACTION_MAY_HAVE_COMMITTED` | An action, scroll, key press, typing, or mouse call timed out, or hit a strict mode violation, after its input may have reached the page. On a device, a tap, key, swipe, or `Back` the device did not answer within `actionTimeout`. It is never repeated | Check the page state before you retry by hand | | `STEP_TIMEOUT`, `STEP_BUDGET_EXHAUSTED` | Goal too big or ambiguous, or a slow provider | Split the goal, use on-screen wording, add vocabulary in `context`; raise `maxSteps` or `maxModelCalls`, `StepTimeout`, or `judgmentTimeout` | -| `STEP_NO_CONCLUSION` | The agent used every model turn without calling `done` (it kept acting, or answered in prose). A failed verdict, not a budget stop | Split the goal or raise `maxModelCalls`; read the step's turns to see what it kept doing | +| `STEP_NO_CONCLUSION` | The agent used every model turn without calling `complete_step` (it kept acting, or answered in prose). A failed verdict, not a budget stop | Split the goal or raise `maxModelCalls`; read the step's turns to see what it kept doing | | `AUTHENTICATION_FAILED` | The agent concluded that signing in failed inside the app: a product verdict (test, exit 1). `AUTH_CREDENTIAL_INVALID` is the blocked counterpart, for a configured credential the app rejects (configuration, exit 2) | Sign in by hand with the same account and fix the app or the account's state. A rejected credential is `AUTH_CREDENTIAL_INVALID` instead | | `SESSION_UNAVAILABLE` (configuration, exit 2), `SESSION_CONTRACT` (test, exit 1) | A test restores a session no selected setup test saved. Or a setup test saved a session twice, saved an undeclared name, or finished without saving a declared one | Add or fix the `[E2ESetup]` test; run the setup test with the tests that restore its sessions | | `SESSION_MISMATCH` | A stored session does not match the current run, target, engine, platform, or app identity (configuration, exit 2) | Run the setup test again in this run, and do not change the target, engine, or app between the setup test and the tests that restore it; a session lasts one run | diff --git a/src/E2E.Acp/AcpExecutor.cs b/src/E2E.Acp/AcpExecutor.cs index 10397d8ea..a35d315dc 100644 --- a/src/E2E.Acp/AcpExecutor.cs +++ b/src/E2E.Acp/AcpExecutor.cs @@ -352,7 +352,7 @@ private static async Task StepMessageAsync(ActiveStep active, IReadOnlyL if (step.Secrets.Count > 0) { - parts.Add($"Secrets to fill with fill_secret, by name: {string.Join(", ", step.Secrets.Select(secret => secret.Purpose is null ? secret.Name : $"{secret.Name} ({secret.Purpose})"))}"); + parts.Add($"Secrets to fill with type_secret, by name: {string.Join(", ", step.Secrets.Select(secret => secret.Purpose is null ? secret.Name : $"{secret.Name} ({secret.Purpose})"))}"); } var missing = StepTools.MissingTools(active, tools); diff --git a/src/E2E.Acp/StepTools.cs b/src/E2E.Acp/StepTools.cs index 2d45e7e7f..b7b14a3cb 100644 --- a/src/E2E.Acp/StepTools.cs +++ b/src/E2E.Acp/StepTools.cs @@ -9,7 +9,7 @@ namespace E2E.Acp; /// /// The tools a session offers (upstream's tools.ts): the built-in agent's action tools () -/// and its verdict tool (), which is done where upstream's is complete_step. An ACP +/// and its verdict tool (). An ACP /// session lists its tools once, while e2e builds them per step, so the session lists them from its first step and each call runs the /// active step's own tool of that name. Every action goes through the step's actions, where the runner authorizes, counts, and records it, /// so steps replay from the cache as the built-in agent's do. diff --git a/src/E2E.Cli/Mcp/SessionCatalog.cs b/src/E2E.Cli/Mcp/SessionCatalog.cs index cd480c517..45e30d948 100644 --- a/src/E2E.Cli/Mcp/SessionCatalog.cs +++ b/src/E2E.Cli/Mcp/SessionCatalog.cs @@ -137,11 +137,11 @@ public static SessionCatalog Create(ISessionContext context, EngineCapabilities if (typeText) { var typeTarget = verbs.Contains(ExecutorVerb.Type) ? OptionalTarget : []; - Add("type", Verb(context, screen, "type", "fill", "Replace the text in a field with text. Without a target, the text is typed into whatever has focus (a field the screen does not list, after tap_at gave it focus): it is inserted at the caret, and replace clears the field first.", Schema(typeTarget, ("text", Property("string", "The text the field should hold"), true), ("replace", new JsonObject { ["type"] = "boolean", ["description"] = AgentTools.ReplaceDescription }, false)), ("text", "value"))); + Add("type", Verb(context, screen, "type", "type", "Replace the text in a field with text. Without a target, the text is typed into whatever has focus (a field the screen does not list, after tap_at gave it focus): it is inserted at the caret, and replace clears the field first.", Schema(typeTarget, ("text", Property("string", "The text the field should hold"), true), ("replace", new JsonObject { ["type"] = "boolean", ["description"] = AgentTools.ReplaceDescription }, false)), ("text", "value"))); } else { - Add("type", Verb(context, screen, "type", "fill", "Replace the text in a field with text.", Schema(Target, ("text", Property("string", "The text the field should hold"), true)), ("text", "value"))); + Add("type", Verb(context, screen, "type", "type", "Replace the text in a field with text.", Schema(Target, ("text", Property("string", "The text the field should hold"), true)), ("text", "value"))); } } @@ -159,12 +159,12 @@ public static SessionCatalog Create(ISessionContext context, EngineCapabilities if (verbs.Contains(ExecutorVerb.Check)) { Add("check", Verb(context, screen, "check", "check", "Check a checkbox or radio.", Schema(Target))); - Add("uncheck", Verb(context, screen, "uncheck", "uncheck", "Uncheck a checkbox.", Schema(Target))); + Add("uncheck", Verb(context, screen, "uncheck", "check", "Uncheck a checkbox.", Schema(Target), extra: new JsonObject { ["checked"] = false })); } - if (verbs.Contains(ExecutorVerb.Clear)) + if (verbs.Contains(ExecutorVerb.Clear) && verbs.Contains(ExecutorVerb.Type)) { - Add("clear", Verb(context, screen, "clear", "clear", "Clear a field.", Schema(Target))); + Add("clear", Verb(context, screen, "clear", "type", "Clear a field.", Schema(Target), extra: new JsonObject { ["value"] = "" })); } // Upstream's grammarTools serves upload with the agent's description; the session takes the agent tool's text and names its node as target. @@ -459,7 +459,7 @@ private static CatalogTool Observe(ISessionContext context, ScreenPresenter scre } /// A grammar verb: the arguments are renamed to the agent tool's (target is its ref), and the tool runs as in an act. - private static CatalogTool Verb(ISessionContext context, ScreenPresenter screen, string name, string agentTool, string description, JsonElement schema, (string From, string To)? rename = null) + private static CatalogTool Verb(ISessionContext context, ScreenPresenter screen, string name, string agentTool, string description, JsonElement schema, (string From, string To)? rename = null, JsonObject? extra = null) { return new CatalogTool { @@ -481,6 +481,11 @@ private static CatalogTool Verb(ISessionContext context, ScreenPresenter screen, mapped[key] = JsonNode.Parse(property.Value.GetRawText()); } + foreach (var (key, value) in extra ?? []) + { + mapped[key] = value?.DeepClone(); + } + var outcome = await RunAsync(context, screen, name, agentTool, args, token, mapped).ConfigureAwait(false); return outcome.Content; }, @@ -506,7 +511,7 @@ private static CatalogTool TypeSecret(ISessionContext context, ScreenPresenter s var mapped = new JsonObject { ["ref"] = args.GetProperty("target").GetString(), ["secret"] = args.GetProperty("secret").GetString() }; // Marked before the fill runs: a fill that fails after typing still leaves the secret on screen. context.SecretFilled(); - var outcome = await RunAsync(context, screen, "type_secret", "fill_secret", args, token, mapped).ConfigureAwait(false); + var outcome = await RunAsync(context, screen, "type_secret", "type_secret", args, token, mapped).ConfigureAwait(false); return outcome.Content; }, }; diff --git a/src/E2E/Agent/ActContext.cs b/src/E2E/Agent/ActContext.cs index 7a3c1da20..a700367df 100644 --- a/src/E2E/Agent/ActContext.cs +++ b/src/E2E/Agent/ActContext.cs @@ -184,7 +184,7 @@ public void WriteTranscript() // Upstream's KINDS_BY_VERB (engine/session.ts): the action kinds an engine must declare for each kind-derived grammar verb. `check` sets a // state rather than flipping one, so it needs both directions; `fill` unlocks the plain and the secret fill alike. The port's `clear` verb - // follows its own kind: upstream has no clear verb. + // follows its own kind and serves the MCP `clear` tool through `type`: upstream has no clear verb. private static readonly (ExecutorVerb Verb, string[] Kinds)[] KindsByVerb = [ (ExecutorVerb.Tap, ["tap"]), diff --git a/src/E2E/Agent/ActTools.cs b/src/E2E/Agent/ActTools.cs index 1f8931e64..c5854612d 100644 --- a/src/E2E/Agent/ActTools.cs +++ b/src/E2E/Agent/ActTools.cs @@ -24,7 +24,7 @@ namespace E2E; internal sealed partial class ActTools(AttemptScope scope, ObservationFeed feed, ScreenPresenter screen, bool session = false) { private static readonly HashSet ActionTools = - new(["navigate", "tap", "double_tap", "long_press", "right_click", "hover", "drag", "fill", "fill_secret", "press", "select", "check", "uncheck", "clear", "upload", "back", "resize_viewport", "switch_tab", "close_tab", "dismiss_keyboard", "scroll", "scroll_to", "tap_at", "hover_at", "drag_at", "type_at", "press_at", "select_at"], StringComparer.Ordinal); + new(["navigate", "tap", "double_tap", "long_press", "right_click", "hover", "drag", "type", "type_secret", "press", "select", "check", "upload", "back", "resize_viewport", "switch_tab", "close_tab", "dismiss_keyboard", "scroll", "scroll_to", "tap_at", "hover_at", "drag_at", "type_at", "press_at", "select_at"], StringComparer.Ordinal); private readonly AttemptScope _scope = scope; private readonly ObservationFeed _feed = feed; @@ -150,12 +150,12 @@ private async Task RunActionAsync(ModelToolCall call, IExecutorActi return await PointToolAsync(call.Name, arguments, actions, budget, token).ConfigureAwait(false); case "press" when !HasTarget(arguments): return await PressAsync(arguments, null, "the focused field", actions, token).ConfigureAwait(false); - case "fill" when !HasTarget(arguments) && Keyboard: + case "type" when !HasTarget(arguments) && Keyboard: // Keyboard input to whatever holds focus: how a field the tree does not list gets its text after a tap_at gave it focus. var typed = Args.String(arguments, "value") ?? ""; var replace = Args.Bool(arguments, "replace") == true; return await ActingAsync( - new ActionLabel("Typed into the focused field.", "fill into the focused field"), + new ActionLabel("Typed into the focused field.", "type into the focused field"), () => actions.TypeTextAsync(typed, new TypeTextOptions { Replace = replace }, token), token).ConfigureAwait(false); } @@ -171,7 +171,7 @@ private async Task RunActionAsync(ModelToolCall call, IExecutorActi } var target = new ExecutorTarget(node); - if (call.Name == "fill" && Keyboard) + if (call.Name == "type" && Keyboard) { var value = Args.String(arguments, "value") ?? ""; return await ActingAsync(NodeLabel(call.Name, arguments, "#" + node.Ref), () => TypeIntoNodeAsync(actions, target, value, "#" + node.Ref, token), token).ConfigureAwait(false); @@ -191,12 +191,10 @@ private async Task RunActionAsync(ModelToolCall call, IExecutorActi "long_press" => actions.LongPressAsync(target, token), "right_click" => actions.SecondaryTapAsync(target, ClickOptionsOf(arguments), token), "hover" => actions.HoverAsync(target, token), - "fill" => actions.TypeAsync(target, Args.String(arguments, "value") ?? "", token), - "fill_secret" => actions.TypeSecretAsync(target, Args.String(arguments, "secret") ?? "", token), + "type" => actions.TypeAsync(target, Args.String(arguments, "value") ?? "", token), + "type_secret" => actions.TypeSecretAsync(target, Args.String(arguments, "secret") ?? "", token), "select" => actions.SelectAsync(target, Args.String(arguments, "value") ?? "", token), - "check" => actions.CheckAsync(target, true, token), - "uncheck" => actions.CheckAsync(target, false, token), - "clear" => actions.ClearAsync(target, token), + "check" => actions.CheckAsync(target, Args.Bool(arguments, "checked") != false, token), _ => throw new TestException("INVALID_ARGUMENT", "Unknown tool " + call.Name + "."), }, token).ConfigureAwait(false); @@ -483,8 +481,7 @@ private async Task FailedAsync(ActionLabel label, E2EException erro // Upstream's failureLead: the attempt, then the error's code, then its message. `navigate file:///etc/passwd failed: POLICY_DENIED: forbidden URL scheme: file:`. private static string FailureLead(ActionLabel label, E2EException error) => label.Attempt + " failed: " + error.Code + ": " + error.Message; - // The done and attempt leads of a node tool, upstream's wording; `reference` stands where upstream's `#id` does. fill and fill_secret - // keep the port's tool names in the attempt, where upstream's are type and type_secret. clear has no upstream tool. + // The done and attempt leads of a node tool, upstream's wording; `reference` stands where upstream's `#id` does. private static ActionLabel NodeLabel(string tool, JsonElement arguments, string reference) { var key = Args.String(arguments, "key") ?? "Enter"; @@ -497,16 +494,14 @@ private static ActionLabel NodeLabel(string tool, JsonElement arguments, string "long_press" => new("Long-pressed " + reference + ".", "long_press " + reference), "right_click" => new("Right-clicked " + reference + ".", "right_click " + reference), "hover" => new("Hovered over " + reference + ".", "hover " + reference), - "fill" => new("Typed into " + reference + ".", "fill " + reference), - "fill_secret" => new( + "type" => new("Typed into " + reference + ".", "type " + reference), + "type_secret" => new( "Filled secret \"" + secret + "\" into " + reference + "; its value is masked in every observation.", - "fill_secret \"" + secret + "\" into " + reference, + "type_secret \"" + secret + "\" into " + reference, false), "press" => new("Pressed " + key + " on " + reference + ".", "press " + key + " on " + reference, !MovesFocusOnly(key)), "select" => new("Selected \"" + value + "\" in " + reference + ".", "select \"" + value + "\" in " + reference), - "check" => new("Checked " + reference + ".", "check " + reference), - "uncheck" => new("Unchecked " + reference + ".", "check " + reference), - "clear" => new("Cleared " + reference + ".", "clear " + reference), + "check" => new((Args.Bool(arguments, "checked") == false ? "Unchecked " : "Checked ") + reference + ".", "check " + reference), _ => new(tool + " " + reference, tool + " " + reference), }; } diff --git a/src/E2E/Agent/ActionNames.cs b/src/E2E/Agent/ActionNames.cs index 010cb10b6..7ea6d827a 100644 --- a/src/E2E/Agent/ActionNames.cs +++ b/src/E2E/Agent/ActionNames.cs @@ -21,13 +21,11 @@ internal static class ActionNames /// /// Every name the grammar may hand out (upstream's GRAMMAR_TOOL_NAMES). A project tool cannot take one. The port's grammar also - /// offers fill, fill_secret, uncheck, and clear under its own names (upstream's are type and - /// type_secret, and upstream has no uncheck or clear), so a failure of those leads with the port's name and is read here too. /// public static readonly HashSet GrammarToolNames = new( [ "observe", "tap", "double_tap", "long_press", "right_click", "hover", "type", "type_secret", "press", "select", "check", "scroll", "scroll_to", "drag", "upload", "navigate", "back", "resize_viewport", "switch_tab", "close_tab", "screenshot", "tap_at", "hover_at", "drag_at", "type_at", - "press_at", "select_at", "dismiss_keyboard", "fill", "fill_secret", "uncheck", "clear", + "press_at", "select_at", "dismiss_keyboard", ], StringComparer.Ordinal); } diff --git a/src/E2E/Agent/Agent.cs b/src/E2E/Agent/Agent.cs index c6e0b125e..aee1c10d3 100644 --- a/src/E2E/Agent/Agent.cs +++ b/src/E2E/Agent/Agent.cs @@ -257,13 +257,13 @@ private async Task ExtractCoreAsync(string instruction, ExtractOptions? op IReadOnlyList fields = []; if (extract is null) { - var done = response.ToolCalls.FirstOrDefault(item => string.Equals(item.Name, "done", StringComparison.Ordinal)); + var done = response.ToolCalls.FirstOrDefault(item => string.Equals(item.Name, VerdictTool.Name, StringComparison.Ordinal)); if (done is not null) { throw new AgentException("ASSERTION_INCONCLUSIVE", ExplainInconclusive("nothing to extract: " + (Args.String(done.Arguments, "summary") ?? "the screen does not show the requested data"), vision)); } - issue = "call extract with the data, or done when the screen does not show it"; + issue = "call extract with the data, or complete_step when the screen does not show it"; previous = response.Content; } else if (!TryProperty(extract.Arguments, "data", out var data)) diff --git a/src/E2E/Agent/AgentTools.cs b/src/E2E/Agent/AgentTools.cs index 465a6ba36..8fd4c835a 100644 --- a/src/E2E/Agent/AgentTools.cs +++ b/src/E2E/Agent/AgentTools.cs @@ -243,8 +243,8 @@ private static readonly (string Tool, ExecutorVerb Bare, ExecutorVerb Node, stri [ Tool("tap", "Activate a control.", Target), Tool("double_tap", "Double-click a control. Use it only when a tap does nothing, such as a label that opens an editor on a double-click.", Target), - Tool("fill", "Replace the text in a field.", WithValue(Target, "value")), - Tool("fill_secret", "Type a declared secret into an editable field. A password secret needs a password field. Pass the secret name, never the value.", WithValue(Target, "secret")), + Tool("type", "Replace the text in a field.", WithValue(Target, "value")), + Tool("type_secret", "Type a declared secret into an editable field. A password secret needs a password field. Pass the secret name, never the value.", WithValue(Target, "secret")), Tool( "press", "Press a key such as Enter, Escape, or Tab. Omit role and name to press the focused control. " + @@ -254,9 +254,10 @@ private static readonly (string Tool, ExecutorVerb Bare, ExecutorVerb Node, stri $$"""{ "times": { "type": "integer", "minimum": 1, "maximum": {{MaxPressTimes}}, "description": "Press the key this many times in this one call, 1 to {{MaxPressTimes}}; default 1. Use more to extend a selection or move a caret several steps at once." } }""", "key")), Tool("select", "Choose an option in a combobox.", WithValue(Target, "value")), - Tool("check", "Check a checkbox.", Target), - Tool("uncheck", "Uncheck a checkbox.", Target), - Tool("clear", "Clear a field.", Target), + Tool( + "check", + "Set a checkbox, switch, or radio to a state: checked true (the default) or false. Unlike tap it never flips a control already in that state, so use it whenever the step names the state to end in.", + Extend(Target, """{ "checked": { "type": "boolean", "description": "The state to end in; default true" } }""")), ]; private static readonly ModelTool LongPress = Tool( @@ -306,8 +307,7 @@ private static readonly (string Tool, ExecutorVerb Bare, ExecutorVerb Node, stri /// /// The verbs of that offer each tool, any one of them (the verbs.has(...) at each tool's site in /// upstream's createGrammarTools and createPointTools): a tool whose verbs the engine's declarations do not unlock is absent - /// from the toolset. fill and press are also offered by the keyboard alone, without a target. The port's clear and - /// uncheck have no upstream tool; they follow their own action kind and the kinds of check. + /// from the toolset. type and press are also offered by the keyboard alone, without a target. /// internal static readonly IReadOnlyDictionary ToolVerbs = new Dictionary(StringComparer.Ordinal) { @@ -317,13 +317,11 @@ private static readonly (string Tool, ExecutorVerb Bare, ExecutorVerb Node, stri ["right_click"] = [ExecutorVerb.SecondaryTap], ["hover"] = [ExecutorVerb.Hover], ["drag"] = [ExecutorVerb.Drag], - ["fill"] = [ExecutorVerb.Type, ExecutorVerb.TypeText], - ["fill_secret"] = [ExecutorVerb.TypeSecret], + ["type"] = [ExecutorVerb.Type, ExecutorVerb.TypeText], + ["type_secret"] = [ExecutorVerb.TypeSecret], ["press"] = [ExecutorVerb.Press, ExecutorVerb.PressKey], ["select"] = [ExecutorVerb.Select], ["check"] = [ExecutorVerb.Check], - ["uncheck"] = [ExecutorVerb.Check], - ["clear"] = [ExecutorVerb.Clear], ["tap_at"] = [ExecutorVerb.TapAt, ExecutorVerb.Tap], ["hover_at"] = [ExecutorVerb.HoverAt, ExecutorVerb.Hover], ["drag_at"] = [ExecutorVerb.DragAt, ExecutorVerb.Drag], @@ -343,10 +341,10 @@ private static ModelTool NodeTool(ModelTool verb, IReadOnlySet ver var typeText = verbs.Contains(ExecutorVerb.TypeText); return verb.Name switch { - "fill" when typeText && !verbs.Contains(ExecutorVerb.Type) => WithoutTarget( + "type" when typeText && !verbs.Contains(ExecutorVerb.Type) => WithoutTarget( FillFocused, - "Type a plain-text value into whatever has focus. After tap_at on a field the screen does not list, call fill: the value is inserted at the caret of the focused field; set replace to clear it first. Several fields can be typed in one turn."), - "fill" when typeText => FillFocused, + "Type a plain-text value into whatever has focus. After tap_at on a field the screen does not list, call type: the value is inserted at the caret of the focused field; set replace to clear it first. Several fields can be typed in one turn."), + "type" when typeText => FillFocused, "press" when verbs.Contains(ExecutorVerb.PressKey) && !verbs.Contains(ExecutorVerb.Press) => WithoutTarget( verb, "Press a key such as Enter, Escape, or Tab on whatever has focus. Shift with an arrow key extends the selection in a field one step per press, and the screen then shows the selected text as selection=\"...\"."), @@ -379,9 +377,9 @@ private static ModelTool WithoutTarget(ModelTool tool, string description) // With a keyboard, a fill with no target types into whatever has focus (upstream's type with typeText): how a field the tree does not // list gets its text after a tap_at gave it focus. private static readonly ModelTool FillFocused = Tool( - "fill", + "type", "Type a plain-text value into one input node, or into whatever has focus when target is omitted. With target the value replaces the node's value. " + - "After tap_at on a field the screen does not list, call fill with NO target (not the document or a container id): the value is inserted at the caret of the focused field; set replace to clear it first. " + + "After tap_at on a field the screen does not list, call type with NO target (not the document or a container id): the value is inserted at the caret of the focused field; set replace to clear it first. " + "Several fields can be typed in one turn.", Extend(Target, $$"""{ "value": { "type": "string" }, "replace": { "type": "boolean", "description": "Without target: {{ReplaceDescription}}" } }""", "value")); @@ -400,16 +398,15 @@ verb.Name is "tap" or "double_tap" or "right_click" /// /// Upstream's VERDICT_RULES: the guidance for . The tool loop appends it to the instructions of every - /// executor it builds, so an executor outside the loop gives it to its own model beside the tool. The port's tool is done - /// where upstream's is complete_step, and takes code where upstream's takes errorCode. + /// executor it builds, so an executor outside the loop gives it to its own model beside the tool. The tool takes code where upstream's takes errorCode. /// public const string VerdictRules = - "Finish with done when the step's goal is achieved, or you are certain it cannot be. " + - "done status is passed, failed, or blocked. " + + "Finish with complete_step when the step's goal is achieved, or you are certain it cannot be. " + + "complete_step status is passed, failed, or blocked. " + "blocked means credentials, the environment, or test setup prevented a verdict, and requires a code."; /// - /// Upstream's createVerdictTool: the done tool with the closed blocked-code policy, and the verdict it collected. Make a + /// Upstream's createVerdictTool: the complete_step tool with the closed blocked-code policy, and the verdict it collected. Make a /// fresh one for each step: the first accepted verdict is final. /// /// @@ -444,6 +441,8 @@ internal static IReadOnlyList GrammarTools(StepExecutorContext con var guard = options?.Guard; var act = new ActTools(host.Scope, host.Feed, DefaultAgent.PresenterFor(context)); return GrammarToolDefinitions(host.Scope.EngineCapabilities, offered, host.Scope.ActionKinds, host.Scope.Session.PointerActionKinds, host.Scope.Session.CanDismissKeyboard) + // Offered only when the step declared secrets (upstream: an empty vocabulary is better than a tool the model can only be rejected on). + .Where(definition => definition.Name != "type_secret" || context.Step.Secrets.Count > 0) .Where(definition => !asserting || AssertionOffers(definition.Name, project, context.Step.Vision)) .Select(definition => new ToolLoopTool( definition, @@ -671,7 +670,7 @@ private static List CreatePointTools(IReadOnlySet verbs /// The tools the harness adds beside the grammar, each where it adds them: report_finding under e2e explore, and /// locate, test_lines, start_recording, and stop_recording under an e2e mcp session (the session /// catalog, SessionCatalog.Create; run_code is reserved, and the port has no tool of that name). A project tool named like one would be - /// replaced there, so the name is refused. Upstream's HARNESS_TOOL_NAMES; the port's other harness tool, done, + /// replaced there, so the name is refused. Upstream's HARNESS_TOOL_NAMES; the port's other harness tool, complete_step, /// is reserved with the grammar's names. /// internal static readonly IReadOnlyDictionary HarnessToolNames = new Dictionary(StringComparer.Ordinal) @@ -687,7 +686,6 @@ private static List CreatePointTools(IReadOnlySet verbs /// True when is a tool the agent offers itself, whatever the engine. internal static bool IsReserved(string name) => - string.Equals(name, VerdictTool.Name, StringComparison.Ordinal) || GrammarToolDefinitions(EngineCapabilities.Scroll | EngineCapabilities.History | EngineCapabilities.Resize | EngineCapabilities.SwitchTab | EngineCapabilities.CloseTab | EngineCapabilities.Upload | EngineCapabilities.Keyboard, null, EngineActionKinds.Locator, ["tap", "hover", "dragTo"], dismissKeyboard: true).Any(tool => string.Equals(tool.Name, name, StringComparison.Ordinal)); /// The model's view of a project tool: its name, description, and input schema. @@ -706,7 +704,7 @@ internal static IReadOnlyList ExtractFor(JsonNode schema) return [ Open("extract", "Return the data read from the screen.", JsonSerializer.SerializeToElement(parameters)), - Open("done", "Use this only when the data is not on screen.", JudgeDone), + Open("complete_step", "Use this only when the data is not on screen.", JudgeDone), ]; } @@ -834,9 +832,9 @@ internal VerdictTool(ModelTool tool, ExecutorStepKind? kind = null) } /// The name the model calls to end a step. - public const string Name = "done"; + public const string Name = "complete_step"; - /// The done tool; offer it beside the grammar tools. + /// The complete_step tool; offer it beside the grammar tools. public ModelTool Tool { get; } /// The verdict once the model called the tool with an accepted one. Upstream's verdict(). @@ -891,7 +889,7 @@ internal ToolOutcome Decide(ModelToolCall call) var status = Args.String(call.Arguments, "status"); if (status is not ("passed" or "failed" or "blocked")) { - return ToolOutcome.Reject("done status must be passed, failed, or blocked."); + return ToolOutcome.Reject("complete_step status must be passed, failed, or blocked."); } // Upstream's schema (summary: z.string().min(1).max(2000)) turns an empty or oversized summary back to the model. @@ -899,7 +897,7 @@ internal ToolOutcome Decide(ModelToolCall call) if (string.IsNullOrWhiteSpace(summary) || summary.Length > AgentTools.MaxVerdictSummaryChars) { return ToolOutcome.Reject( - "done summary must be a non-empty string of at most " + AgentTools.MaxVerdictSummaryChars.ToString(System.Globalization.CultureInfo.InvariantCulture) + + "complete_step summary must be a non-empty string of at most " + AgentTools.MaxVerdictSummaryChars.ToString(System.Globalization.CultureInfo.InvariantCulture) + " characters: what you did and what the screen shows now."); } diff --git a/src/E2E/Agent/DefaultAgent.cs b/src/E2E/Agent/DefaultAgent.cs index 8544412cc..96c70e090 100644 --- a/src/E2E/Agent/DefaultAgent.cs +++ b/src/E2E/Agent/DefaultAgent.cs @@ -39,7 +39,7 @@ public static class DefaultAgent "Every action result already waited for the effect and contains the changes, so do not call observe after an action. Call observe only after waiting for something the last result showed in progress. " + "Conclude passed only when the latest result shows the outcome the step asked for. A result that reports no change, or only the keyboard closing, means the decisive action did not land: act again or conclude failed. " + "A result that says only that no listed node changed has not ruled out a drawn change: take a screenshot, when offered, before deciding. Never describe an unchanged screen as the outcome. " + - "Never type a secret value. Call fill_secret with the secret name. " + + "Never type a secret value. Call type_secret with the secret name. " + "A secret looks like . " + "If what you need is not on the screen, scroll to it. " + "screenshot attaches the viewport's pixels when the tree lacks what you need (a shape on a canvas, a pin on a map, a region of an image, a control inside a system sheet) or contradicts what you expect; " + @@ -47,13 +47,13 @@ public static class DefaultAgent "tap_at, hover_at, drag_at, type_at, press_at, and select_at act at a point in the latest screenshot's pixel coordinates. " + "tap_at, hover_at, and drag_at act on the bare points where the engine can, and on the listed controls under them by id where it cannot; " + "type_at, press_at, and select_at act on a listed control under the point by its id. " + - "Prefer ids: tap, fill, press, and select by id whenever the screen lists the target, " + + "Prefer ids: tap, type, press, and select by id whenever the screen lists the target, " + "use the point tools only for a target the screen does not list, " + "and take a screenshot before guessing a point rather than aiming from memory or from what you expect to be drawn."; /// - /// Assertion-specific rules (upstream's ASSERT_RULES): they allow evidence gathered across several viewports. The port's verdict tool is done, - /// which takes code. + /// Assertion-specific rules (upstream's ASSERT_RULES): they allow evidence gathered across several viewports. The verdict tool takes code + /// where upstream's takes errorCode. /// public const string AssertRules = "You are judging exactly one assertion against the current application screen.\n" + diff --git a/src/E2E/Agent/LoopGuards.cs b/src/E2E/Agent/LoopGuards.cs index 74f395501..e3f8ea967 100644 --- a/src/E2E/Agent/LoopGuards.cs +++ b/src/E2E/Agent/LoopGuards.cs @@ -39,7 +39,7 @@ internal static partial class LoopGuards messages .Where(message => message.ToolCalls is not null) .SelectMany(message => message.ToolCalls!) - .Where(call => !string.Equals(call.Name, "done", StringComparison.Ordinal)) + .Where(call => !string.Equals(call.Name, VerdictTool.Name, StringComparison.Ordinal)) .Select(call => (call.Name, call.Arguments.GetRawText())) .ToList(); diff --git a/src/E2E/Agent/Model.cs b/src/E2E/Agent/Model.cs index 1c463bc8b..e30a50263 100644 --- a/src/E2E/Agent/Model.cs +++ b/src/E2E/Agent/Model.cs @@ -222,7 +222,7 @@ public static class ModelResponses { public static ModelResponse Done(string status, string summary, string? code = null) { - return Call("done", code is null + return Call(VerdictTool.Name, code is null ? new { status, summary } : (object)new { status, summary, code }); } @@ -246,7 +246,7 @@ public static ModelResponse Tap(string role, string name) public static ModelResponse Fill(string role, string name, string value) { - return Call("fill", new { role, name, value }); + return Call("type", new { role, name, value }); } public static ModelResponse Call(string name, object arguments) @@ -301,7 +301,7 @@ public Task CompleteAsync(ModelRequest request, CancellationToken private static ModelResponse AsJudgment(ModelResponse response) { - var done = response.ToolCalls.FirstOrDefault(call => string.Equals(call.Name, "done", StringComparison.Ordinal)); + var done = response.ToolCalls.FirstOrDefault(call => string.Equals(call.Name, VerdictTool.Name, StringComparison.Ordinal)); if (done is null || response.Content is not null) { return response; diff --git a/src/E2E/Agent/Models/ToolChoice.cs b/src/E2E/Agent/Models/ToolChoice.cs index 299b81289..b45d098d3 100644 --- a/src/E2E/Agent/Models/ToolChoice.cs +++ b/src/E2E/Agent/Models/ToolChoice.cs @@ -8,7 +8,7 @@ namespace E2E.Internal; /// /// Upstream's agent/model/tool-choice.ts: recognizes a provider refusing a forced tool choice. The act loop asks for a tool call -/// on every turn (required) and names done on the final ones; some models reject that request shape outright (Anthropic's +/// on every turn (required) and names complete_step on the final ones; some models reject that request shape outright (Anthropic's /// Claude Fable 5.1 answers HTTP 400 with tool_choice: type "tool" and "any" are not supported for this model.; DeepSeek's V4 /// models think by default and answer a forced choice with Thinking mode does not support this tool_choice; Meta's Muse Spark /// models answer only "auto" is supported for tool_choice), and a gateway forwards the provider's words. The loop reacts to this diff --git a/src/E2E/Agent/ToolLoop.cs b/src/E2E/Agent/ToolLoop.cs index f86893065..6e4e87f2f 100644 --- a/src/E2E/Agent/ToolLoop.cs +++ b/src/E2E/Agent/ToolLoop.cs @@ -10,7 +10,7 @@ namespace E2E; /// /// Upstream's createToolLoopExecutor (agent/tool-loop.ts): the chassis of an executor that thinks in a loop of model -/// turns and tool calls. It owns everything but the tool vocabulary: the done verdict tool and its closed code policy, the +/// turns and tool calls. It owns everything but the tool vocabulary: the complete_step verdict tool and its closed code policy, the /// hard stops (a spent budget, a timeout, or a cancel ends the loop, never the model), the loop guards, the shrink of a history /// that overflowed the model, and the count of model calls. The built-in agent is this chassis plus the grammar tools /// (). An executor for a device or an API brings other tools and inherits the same discipline. @@ -65,7 +65,7 @@ public sealed class ToolLoopExecutorOptions /// public Func? System { get; init; } - /// Builds the step's tool vocabulary. The chassis adds done. + /// Builds the step's tool vocabulary. The chassis adds complete_step. public required Func> Tools { get; init; } /// Builds the step's opening prompt: the first user message. Set this or . @@ -167,7 +167,7 @@ public sealed record PreparedTurn(int StepNumber, IReadOnlyList Previous /// /// What hands back: the history for the next turn, and optionally a reason to -/// stop. The executor has evidence the loop guards cannot see that the step is not progressing. A stop leaves the model only done, like a loop guard. +/// stop. The executor has evidence the loop guards cannot see that the step is not progressing. A stop leaves the model only complete_step, like a loop guard. /// public sealed record PreparedMessages(IReadOnlyList Messages, string? Stop = null) { @@ -189,13 +189,13 @@ internal sealed class ToolLoopExecutor(ToolLoopExecutorOptions options) : IStepE // Upstream's FORCED_CONCLUSION_TURNS: turns from the budget ceiling at which only the conclusion tool is offered. private const int ForcedConclusionTurns = 2; - // Upstream's TOOL_CALLS_ONLY_RULE, with the port's done: part of every step's instructions from the first turn. + // Upstream's TOOL_CALLS_ONLY_RULE: part of every step's instructions from the first turn. internal const string ToolCallsOnlyRule = - "Reply with tool calls only. A reply without a tool call does nothing and spends a turn; the step ends only through done."; + "Reply with tool calls only. A reply without a tool call does nothing and spends a turn; the step ends only through complete_step."; - // Upstream's TEXT_REPLY_NOTICE, with the port's done: sent when a turn came back as text without a tool call. + // Upstream's TEXT_REPLY_NOTICE: sent when a turn came back as text without a tool call. private const string TextReplyNotice = - "[SYSTEM] Your last reply had no tool call, so nothing happened. Continue with a tool call, or call done with your verdict."; + "[SYSTEM] Your last reply had no tool call, so nothing happened. Continue with a tool call, or call complete_step with your verdict."; // Upstream's FREE_TOOL_CHOICE_MODELS: models that refused a forced tool choice once. A model that answers HTTP 400 to that request // shape gets auto for the rest of the process, so the refusal costs one round trip per model, not one per step. Upstream's WeakSet @@ -309,11 +309,11 @@ public async Task RunStepAsync(StepExecutorContext context, Cancell messages.Add(new ModelMessage { Role = "user", - Content = "[SYSTEM] The step is nearly out of time. Call done now with your best verdict: passed only if you verified the goal, failed if the application misbehaved, blocked (with code) if something outside the application stopped you.", + Content = "[SYSTEM] The step is nearly out of time. Call complete_step now with your best verdict: passed only if you verified the goal, failed if the application misbehaved, blocked (with code) if something outside the application stopped you.", }); } - // A repeat warns first. A stop leaves the model only done, so it still writes its own verdict. + // A repeat warns first. A stop leaves the model only complete_step, so it still writes its own verdict. var (stop, repeat) = preparedStop is not null ? (true, preparedStop) : stopped ? (true, null) : LoopGuards.CheckLoopGuards(LoopGuards.ExtractGuardCalls(messages)); @@ -326,8 +326,8 @@ public async Task RunStepAsync(StepExecutorContext context, Cancell { Role = "user", Content = stop - ? "[SYSTEM] Loop guard: " + repeat + ". Repeating it further will not make progress. Call done now with your best verdict: failed if the application misbehaved, blocked (with code) if something outside the application stopped you." - : "[SYSTEM NOTICE] You appear to be going in circles: " + repeat + ". Change approach, or call done with your best verdict.", + ? "[SYSTEM] Loop guard: " + repeat + ". Repeating it further will not make progress. Call complete_step now with your best verdict: failed if the application misbehaved, blocked (with code) if something outside the application stopped you." + : "[SYSTEM NOTICE] You appear to be going in circles: " + repeat + ". Change approach, or call complete_step with your best verdict.", }); } @@ -344,8 +344,8 @@ public async Task RunStepAsync(StepExecutorContext context, Cancell { Role = "user", Content = streakStop - ? "[SYSTEM] Loop guard: " + streak + ". Repeating it further will not make progress. Call done now with your best verdict: failed if the application misbehaved, blocked (with code) if something outside the application stopped you." - : "[SYSTEM NOTICE] " + streak + ". Read the failure text and the current screen and change approach; if the step cannot be done, call done with your best verdict.", + ? "[SYSTEM] Loop guard: " + streak + ". Repeating it further will not make progress. Call complete_step now with your best verdict: failed if the application misbehaved, blocked (with code) if something outside the application stopped you." + : "[SYSTEM NOTICE] " + streak + ". Read the failure text and the current screen and change approach; if the step cannot be done, call complete_step with your best verdict.", }); } } @@ -356,17 +356,17 @@ public async Task RunStepAsync(StepExecutorContext context, Cancell messages.Add(new ModelMessage { Role = "user", - Content = "[SYSTEM NOTICE] Only " + turnsLeft.ToString(CultureInfo.InvariantCulture) + " turns remain for this step. Finish the remaining work now, or call done with your best verdict: failed if the application misbehaved, blocked (with code) if something outside the application stopped you.", + Content = "[SYSTEM NOTICE] Only " + turnsLeft.ToString(CultureInfo.InvariantCulture) + " turns remain for this step. Finish the remaining work now, or call complete_step with your best verdict: failed if the application misbehaved, blocked (with code) if something outside the application stopped you.", }); } // Near the budget, after a loop guard stop, or on a low clock only the conclusion tool is offered. Two forced turns, not one, // so a rejected verdict (blocked without a code) can be repaired. var forced = turnsLeft <= ForcedConclusionTurns || stopped || lowClock; - var offered = forced ? definitions.Where(tool => string.Equals(tool.Name, "done", StringComparison.Ordinal)).ToList() : definitions; + var offered = forced ? definitions.Where(tool => string.Equals(tool.Name, VerdictTool.Name, StringComparison.Ordinal)).ToList() : definitions; // Upstream's prepareStep: every turn asks for a tool call, and a forced turn names the conclusion tool. - var turnToolChoice = forced && toolChoice.Type == "required" ? ModelToolChoice.Tool("done") : toolChoice; + var turnToolChoice = forced && toolChoice.Type == "required" ? ModelToolChoice.Tool(VerdictTool.Name) : toolChoice; AiTraceTurn traced; try { @@ -452,8 +452,8 @@ public async Task RunStepAsync(StepExecutorContext context, Cancell var screenshots = new List<(string Tool, ExecutorPixels Pixels)>(); foreach (var toolCall in response.ToolCalls) { - var outcome = ToolOutcome.Fail(toolCall.Name + " did not run: only done is available now. Call done with your verdict."); - if (!forced || string.Equals(toolCall.Name, "done", StringComparison.Ordinal)) + var outcome = ToolOutcome.Fail(toolCall.Name + " did not run: only complete_step is available now. Call complete_step with your verdict."); + if (!forced || string.Equals(toolCall.Name, VerdictTool.Name, StringComparison.Ordinal)) { await traced.ExecuteToolAsync(async () => outcome = await RunToolAsync(toolCall, tools, verdictTool, token).ConfigureAwait(false)).ConfigureAwait(false); } @@ -515,7 +515,7 @@ public async Task RunStepAsync(StepExecutorContext context, Cancell // Upstream's end of LoopRun.run: the turns ran out without a verdict, so the step fails, as any step that never concluded. return new StepVerdict( StepVerdictStatus.Failed, - "the agent used " + turns.Used.ToString(CultureInfo.InvariantCulture) + " of " + maxCalls.ToString(CultureInfo.InvariantCulture) + " turn(s) without calling done", + "the agent used " + turns.Used.ToString(CultureInfo.InvariantCulture) + " of " + maxCalls.ToString(CultureInfo.InvariantCulture) + " turn(s) without calling complete_step", "STEP_NO_CONCLUSION"); } diff --git a/src/E2E/Explore/Body.cs b/src/E2E/Explore/Body.cs index 21e980a44..dc3f8f501 100644 --- a/src/E2E/Explore/Body.cs +++ b/src/E2E/Explore/Body.cs @@ -32,7 +32,7 @@ internal sealed record ExploreFixtures(IExploreAgent Agent, Func /// /// The configured accounts, by name. Each travels with every charter as a step secret, so the explorer can sign in with -/// fill_secret by name. The password itself never reaches the model. Upstream resolves them from its credential registry +/// type_secret by name. The password itself never reaches the model. Upstream resolves them from its credential registry /// when the body runs. /// /// The agent the steps and the planner run as. Null is default. diff --git a/src/E2E/Explore/Executor.cs b/src/E2E/Explore/Executor.cs index c9012b3f8..b9a4cb141 100644 --- a/src/E2E/Explore/Executor.cs +++ b/src/E2E/Explore/Executor.cs @@ -26,9 +26,7 @@ public static class ExploreExecutor public const string FindingToolName = "report_finding"; /// - /// Appended to the project's own guidance; the goal travels as the test title (Test: <goal>). Upstream's text, with the port's tool - /// names: done ends a step where upstream calls complete_step, and fill_secret fills a password where - /// upstream calls type_secret. + /// Appended to the project's own guidance; the goal travels as the test title (Test: <goal>). Upstream's text. /// private const string ExploreRules = "Exploration mode: this run has no scripted test. Each step is an exploration charter the planner wrote toward the goal on the Test: line.\n" + @@ -36,10 +34,10 @@ public static class ExploreExecutor "- On every screen, sanity-check beyond \"it renders\": totals equal the sum of their parts and quantities multiply; counts match the items listed; nothing is negative that cannot be; dates are plausible and in order, not an epoch default; copy has no template tokens, placeholders, or misspellings; every control does what its name says; what a screen claims happened (saved, added, removed) is true on the next screen; a password is never shown as you type it.\n" + "- Bugs hide at the edges, so push on them: whitespace-only, duplicate, very long unbroken, accented, emoji, and special-character input (%, _, quotes, ); boundary numbers and dates (0, negative, huge, far past, far future); submitting twice quickly; deleting or renaming a record you created this run that other screens refer to. Combine controls: apply a filter, then sort, search, page, or switch a tab, and check the earlier choices are kept; check a sort really orders the list; check a count of one reads in the singular and a count matches the items it links to. After a save, reload or revisit and check what was kept. Read each screen as text too: a raw error, stack trace, or JSON dump is a defect. An edge input is a defect when the app mangles, loses, or contradicts it (stored differently than shown, a crash, a raw error, a broken layout), or when a required field accepts blank or whitespace-only input.\n" + "- Report every defect with " + FindingToolName + " the moment the evidence is on screen, one call per distinct defect: what you expected, what the screen shows, and the actions that reach it. Do not save findings for the conclusion: a defect that appears only in a summary is lost, since the report reads the tool, not the prose. Findings listed under \"reportedFindings\" in the step parameters are already recorded: never report them again and spend no actions re-confirming them. The same defect on another screen is a new finding: when one screen gets something wrong, try the same control or data on the other screens that offer it (another list, scope, or page showing the same records), and report each kind of screen where it reproduces, naming that screen in the title. The same defect on another record of the same screen (another user, item, or row) is not new.\n" + - "- Accounts listed under \"credentials\" in the step parameters are yours to sign in with: type the username as text and fill the password with fill_secret by the name of its password secret. Never type a password as text, and never ask for one. If no fill_secret tool is offered, this engine cannot fill a password securely: skip signing in, note it in the step summary, and explore what is reachable without it.\n" + + "- Accounts listed under \"credentials\" in the step parameters are yours to sign in with: type the username as text and fill the password with type_secret by the name of its password secret. Never type a password as text, and never ask for one. If no type_secret tool is offered, this engine cannot fill a password securely: skip signing in, note it in the step summary, and explore what is reachable without it.\n" + "- Report defects, not wishes. A defect is something the screen breaks or contradicts: a wrong value, a dead control, a navigation that lands wrong, a claim that turns out false, leaked template text, a misspelling. A missing feature, a design choice, a form the browser refuses to submit while a required field is empty, or something you merely expected is not a defect unless the screen or the goal promised it. A failed tool call, a refused action, or a limit of this harness is not a product defect either.\n" + "- Stay inside the application under test. Do not destroy data that existed before this run (deleting accounts, wiping lists) unless the goal asks for it; creating and editing your own data is fine.\n" + - "- When the charter is covered, or nothing new is reachable within it, call done. \"passed\" means you carried the charter out, however many defects you reported on the way: reported findings never make a step fail. \"failed\" is only for a charter the application would not let you carry out at all: a flow that cannot be completed, a page that cannot be reached. Keep to the charter: a step is one flow, not the whole app. The summary is read by the planner of the next step: say what was covered, what was found, and what looks worth exploring next."; + "- When the charter is covered, or nothing new is reachable within it, call complete_step. \"passed\" means you carried the charter out, however many defects you reported on the way: reported findings never make a step fail. \"failed\" is only for a charter the application would not let you carry out at all: a flow that cannot be completed, a page that cannot be reached. Keep to the charter: a step is one flow, not the whole app. The summary is read by the planner of the next step: say what was covered, what was found, and what looks worth exploring next."; private const int MinTitleChars = 4; private const int MaxTitleChars = 200; diff --git a/src/E2E/Explore/Explore.cs b/src/E2E/Explore/Explore.cs index 9282ce91f..472836ff8 100644 --- a/src/E2E/Explore/Explore.cs +++ b/src/E2E/Explore/Explore.cs @@ -87,7 +87,7 @@ public sealed class ExploreOptions /// /// The accounts the explorer can sign in with, by name. Each travels into every step as the username and a password secret - /// named <name>.password, which the explorer fills with fill_secret; the password never reaches a model. + /// named <name>.password, which the explorer fills with type_secret; the password never reaches a model. /// The config's credentials are always accounts, as upstream carries them; this adds accounts the config does not declare /// (.NET-only: e2e explore passes the E2E_USER_* pairs of the environment). /// diff --git a/tests/E2E.Acp.Tests/Executor/AcpTests.cs b/tests/E2E.Acp.Tests/Executor/AcpTests.cs index d68f27ee5..b604b62f8 100644 --- a/tests/E2E.Acp.Tests/Executor/AcpTests.cs +++ b/tests/E2E.Acp.Tests/Executor/AcpTests.cs @@ -11,12 +11,12 @@ namespace E2E.Acp.Tests.Executor; /// /// Ports describe('acp') of tester-army/e2e packages/acp/tests/executor.test.ts (0410fc9c). The scripted agent is the C# program /// E2E.Acp.Tests.Agent, and the steps run through a real session over the in-memory document engine, since the grammar tools drive the -/// context of a step the runtime opened. The port's verdict tool is done (upstream's complete_step) and takes code. +/// context of a step the runtime opened. The verdict tool takes code. /// [Collection(WorkingDirectoryCollection.Name)] public sealed class AcpTests : IDisposable { - private static readonly object Pass = Call("done", new { status = "passed", summary = "added" }); + private static readonly object Pass = Call("complete_step", new { status = "passed", summary = "added" }); private readonly string _directory = Directory.CreateTempSubdirectory("e2e-acp-test-").FullName; @@ -60,11 +60,11 @@ public async Task Acts_through_ctx_actions_and_passes_on_done() Assert.Contains(entries, entry => entry["config"]?["model"]?.GetValue() == "slow"); var offered = Assert.Single(agent.Sessions())["session"]!; Assert.Equal("e2e_step", offered["server"]!.GetValue()); - // The built-in agent's tools for the target's verbs, plus done. + // The built-in agent's tools for the target's verbs, plus complete_step. Assert.Equal( - ["back", "check", "clear", "done", "double_tap", "fill", "fill_secret", "navigate", "observe", "press", "press_at", "screenshot", "scroll", "scroll_to", "select", "select_at", "tap", "tap_at", "type_at", "uncheck"], + ["back", "check", "complete_step", "double_tap", "navigate", "observe", "press", "press_at", "screenshot", "scroll", "scroll_to", "select", "select_at", "tap", "tap_at", "type", "type_at", "type_secret"], offered["tools"]!.AsArray().Select(tool => tool!.GetValue()).Order(StringComparer.Ordinal)); - Assert.Contains("Typed into", agent.ResultOf("fill"), StringComparison.Ordinal); + Assert.Contains("Typed into", agent.ResultOf("type"), StringComparison.Ordinal); Assert.Contains("Tapped", agent.ResultOf("tap"), StringComparison.Ordinal); // The step's first screen is whole, in the message. Assert.Contains("button \"Add\"", agent.Prompts()[0], StringComparison.Ordinal); @@ -185,7 +185,7 @@ public async Task Partial_Refuses_actions_that_change_the_screen_on_an_assertion // Partial: upstream also has the assertion call hover and sees it refused; the document engine has no pointer, so there is no hover to // refuse, and the read-only tool here is scroll. var page = new TodoPage(); - var agent = Scripted(_directory, Turns([Tap("Add"), Call("scroll", new { direction = "down" }), Call("done", new { status = "failed", summary = "no todo" })])); + var agent = Scripted(_directory, Turns([Tap("Add"), Call("scroll", new { direction = "down" }), Call("complete_step", new { status = "failed", summary = "no todo" })])); await using var session = await StartAsync(agent.Executor, page.World()); var error = await Assert.ThrowsAsync(() => session.Agent.AssertAsync("the list shows a todo")); @@ -212,7 +212,7 @@ public async Task Partial_Refuses_screen_changes_on_an_assertion_that_follows_an Assert.Equal(0, page.Adds); Assert.Contains("this step is an assertion", agent.ResultOf("tap"), StringComparison.Ordinal); var served = Assert.Single(agent.Sessions())["session"]!; - Assert.Equal(["done", "observe", "screenshot", "scroll", "scroll_to"], served["readOnly"]!.AsArray().Select(name => name!.GetValue()).Order(StringComparer.Ordinal)); + Assert.Equal(["complete_step", "observe", "screenshot", "scroll", "scroll_to"], served["readOnly"]!.AsArray().Select(name => name!.GetValue()).Order(StringComparer.Ordinal)); } [Fact] @@ -233,7 +233,7 @@ public async Task Advertises_action_tools_when_the_attempt_begins_with_an_assert [Fact] public async Task Drops_an_inconclusive_code_from_an_act_verdict_and_keeps_it_on_an_assertion_in_one_attempt() { - var inconclusive = Call("done", new { status = "failed", summary = "not enough evidence", code = "ASSERTION_INCONCLUSIVE" }); + var inconclusive = Call("complete_step", new { status = "failed", summary = "not enough evidence", code = "ASSERTION_INCONCLUSIVE" }); var agent = Scripted(_directory, Turns([inconclusive], [inconclusive])); await using var session = await StartAsync(agent.Executor, new TodoPage().World()); @@ -251,8 +251,8 @@ public async Task Concludes_through_the_built_in_agents_done_blocked_needs_a_cod { var page = new TodoPage(); var agent = Scripted(_directory, Turns([ - Call("done", new { status = "blocked", summary = "the app is down" }), - Call("done", new { status = "blocked", summary = "the app is down", code = "APP_UNREACHABLE" }), + Call("complete_step", new { status = "blocked", summary = "the app is down" }), + Call("complete_step", new { status = "blocked", summary = "the app is down", code = "APP_UNREACHABLE" }), Tap("Add"), ])); await using var session = await StartAsync(agent.Executor, page.World()); @@ -261,13 +261,13 @@ public async Task Concludes_through_the_built_in_agents_done_blocked_needs_a_cod Assert.Equal("APP_UNREACHABLE", error.Code); Assert.Contains("the app is down", error.Message, StringComparison.Ordinal); - var answers = agent.Log().Where(entry => entry["call"]?.GetValue() == "done").Select(entry => entry["result"]!.ToJsonString()).ToList(); + var answers = agent.Log().Where(entry => entry["call"]?.GetValue() == "complete_step").Select(entry => entry["result"]!.ToJsonString()).ToList(); Assert.Contains("blocked", answers[0], StringComparison.Ordinal); Assert.Contains("isError", answers[0], StringComparison.Ordinal); Assert.Contains("Step concluded. Wait for the next instruction.", answers[1], StringComparison.Ordinal); // Nothing runs after the verdict. Assert.Equal(0, page.Adds); - Assert.Contains("Finish with done when", agent.Prompts()[0], StringComparison.Ordinal); + Assert.Contains("Finish with complete_step when", agent.Prompts()[0], StringComparison.Ordinal); } [Fact] @@ -596,17 +596,17 @@ public async Task Reports_a_model_the_agent_does_not_offer() public async Task Partial_Lists_the_tools_once_and_tells_the_agent_which_ones_a_step_does_not_offer() { // Partial: the document engine offers every step the same tools, so no step lacks a tool the session listed. - // fill_secret is listed, and a call of it for a secret the step does not declare says so. - var agent = Scripted(_directory, Turns([Call("fill_secret", new { role = "textbox", name = "Title", secret = "admin.password" }), Pass], [Pass])); + // type_secret is listed, and a call of it for a secret the step does not declare says so. + var agent = Scripted(_directory, Turns([Call("type_secret", new { role = "textbox", name = "Title", secret = "admin.password" }), Pass], [Pass])); await using var session = await StartAsync(agent.Executor, new TodoPage().World()); await session.Agent.ActAsync("add a todo"); await session.Agent.ActAsync("add a todo"); var offered = Assert.Single(agent.Sessions())["session"]!["tools"]!.AsArray().Select(tool => tool!.GetValue()).ToList(); - Assert.Contains("fill_secret", offered); + Assert.Contains("type_secret", offered); Assert.DoesNotContain("Not offered in this step", agent.Prompts()[0], StringComparison.Ordinal); - Assert.Contains("isError", agent.ResultOf("fill_secret"), StringComparison.Ordinal); + Assert.Contains("isError", agent.ResultOf("type_secret"), StringComparison.Ordinal); } [Fact] @@ -680,7 +680,7 @@ public async Task Reports_an_agent_that_does_not_start() private static object Tap(string name) => Call("tap", new { role = "button", name }); - private static object Fill(string name, string value) => Call("fill", new { role = "textbox", name, value }); + private static object Fill(string name, string value) => Call("type", new { role = "textbox", name, value }); /// One list of moves per prompt turn. private static object[] Turns(params object[][] turns) => turns; diff --git a/tests/E2E.Acp.Tests/Presets/AgentPresetsTests.cs b/tests/E2E.Acp.Tests/Presets/AgentPresetsTests.cs index b7e0f56d9..01b9abb7c 100644 --- a/tests/E2E.Acp.Tests/Presets/AgentPresetsTests.cs +++ b/tests/E2E.Acp.Tests/Presets/AgentPresetsTests.cs @@ -17,7 +17,7 @@ namespace E2E.Acp.Tests.Presets; [Collection(WorkingDirectoryCollection.Name)] public sealed class AgentPresetsTests : IDisposable { - private static readonly object[] Pass = [new { call = "done", args = new { status = "passed", summary = "added" } }]; + private static readonly object[] Pass = [new { call = "complete_step", args = new { status = "passed", summary = "added" } }]; private readonly List _directories = []; diff --git a/tests/E2E.Tests/Agent/AnInconclusiveJudgmentAfterASecretFillTests.cs b/tests/E2E.Tests/Agent/AnInconclusiveJudgmentAfterASecretFillTests.cs index 86eaa0429..697cc112c 100644 --- a/tests/E2E.Tests/Agent/AnInconclusiveJudgmentAfterASecretFillTests.cs +++ b/tests/E2E.Tests/Agent/AnInconclusiveJudgmentAfterASecretFillTests.cs @@ -19,11 +19,11 @@ public async Task Carries_the_vision_hint_a_secret_fill_does_not_withhold_pixels { page.Roots.Add(new DocumentElement { Role = "textbox", Name = "Password", Value = "", Secure = true, InputPurpose = "password", OnFill = typed.Add }); }); - // The act model fills the password through fill_secret; a screenshot can still reach a judge + // The act model fills the password through type_secret; a screenshot can still reach a judge // afterwards. var calls = 0; var actor = new ScriptedModel(_ => calls++ == 0 - ? ModelResponses.Call("fill_secret", new { role = "textbox", name = "Password", secret = "member-password" }) + ? ModelResponses.Call("type_secret", new { role = "textbox", name = "Password", secret = "member-password" }) : ModelResponses.Done("passed", "filled the password")); var judge = new ScriptedModel(_ => ModelResponses.Done("failed", "no order total or line items are on this screen", "ASSERTION_INCONCLUSIVE")); await using var session = await E2ESession.StartAsync(new E2ESessionOptions diff --git a/tests/E2E.Tests/AgentAct/AgentActVerdictMappingTests.cs b/tests/E2E.Tests/AgentAct/AgentActVerdictMappingTests.cs index 8af125a6f..19687efca 100644 --- a/tests/E2E.Tests/AgentAct/AgentActVerdictMappingTests.cs +++ b/tests/E2E.Tests/AgentAct/AgentActVerdictMappingTests.cs @@ -105,7 +105,7 @@ public async Task Stamps_the_budget_code_onto_a_code_less_failed_verdict_after_a // Upstream's generic secret in a plain textbox and its provider-backed secret are not ported: the port has no // secret purpose check for a plain field and no credential provider functions. [Fact] - public async Task Fills_a_declared_secret_through_fill_secret_and_refuses_an_undeclared_one() + public async Task Fills_a_declared_secret_through_type_secret_and_refuses_an_undeclared_one() { string? typed = null; var world = new DocumentWorld().Map("/", page => page.Roots.Add(new DocumentElement @@ -118,9 +118,9 @@ public async Task Fills_a_declared_secret_through_fill_secret_and_refuses_an_und })); var model = new ScriptedModel(request => request.Messages.Count(message => message.Role == "tool") switch { - 0 => ModelResponses.Call("fill_secret", new { role = "textbox", name = "Password", secret = "admin.password" }), + 0 => ModelResponses.Call("type_secret", new { role = "textbox", name = "Password", secret = "admin.password" }), // A configured secret the step never declared is refused. - 1 => ModelResponses.Call("fill_secret", new { role = "textbox", name = "Password", secret = "stripe-key" }), + 1 => ModelResponses.Call("type_secret", new { role = "textbox", name = "Password", secret = "stripe-key" }), _ => ModelResponses.Done("passed", "filled the declared secret only"), }); var result = await RunAsync( diff --git a/tests/E2E.Tests/AgentAct/AgentActWithTheDefaultToolLoopAgentExecutorTests.cs b/tests/E2E.Tests/AgentAct/AgentActWithTheDefaultToolLoopAgentExecutorTests.cs index cc2433b62..7698b195c 100644 --- a/tests/E2E.Tests/AgentAct/AgentActWithTheDefaultToolLoopAgentExecutorTests.cs +++ b/tests/E2E.Tests/AgentAct/AgentActWithTheDefaultToolLoopAgentExecutorTests.cs @@ -2,12 +2,14 @@ // Modified by Dario Kondratiuk. // SPDX-License-Identifier: Apache-2.0 +using E2E.Tests.AgentActVerbs; using E2E.Tests.CacheModelProvenance; using E2E.Tests.ReportJson; using static E2E.Tests.CoreTests; namespace E2E.Tests.AgentAct; +[Collection(BrowserCollection.Name)] public sealed class AgentActWithTheDefaultToolLoopAgentExecutorTests { private const string Instruction = "increment the counter once and verify it shows 1"; @@ -28,35 +30,30 @@ public async Task Drives_the_step_through_the_tool_loop_to_a_passed_verdict() Assert.Null(result.Error); } - // Partial: upstream lists tools the port does not offer here (upload, drag, drag_at, hover, hover_at, long_press, right_click, resize_viewport, type), - // and its verdict tool is complete_step where the port's is done. Upstream's tool results report changes since a revision, which this does not assert. - // Upstream lists a toolset the port does not have here (upload, and the point tools). The document engine declares no pointer action, - // so it offers no long_press, right_click, hover, or drag. Upstream's tool - // results report changes since a revision. The port's results carry the whole screen instead. [Fact] - public async Task Partial_Offers_the_default_toolset_and_concludes_through_complete_step() + public async Task Offers_the_default_toolset_and_concludes_through_complete_step() { + using var site = await TinySite.StartAsync(context => TinySite.RespondAsync(context, CounterSitePage)); var model = IncrementThenConclude(); - var result = await RunAsync( - async ctx => - { - await ctx.App.OpenAsync("/"); - await ctx.Agent.ActAsync(Instruction); - }, - CounterPage.World(), - model, - wrapEngine: engine => new TabsEngine(engine)); + await using var session = await GesturesSession.StartAsync(site, model, new FileStepCache(TempCache()), "default toolset"); + await GesturesSession.RunAsync(session, async () => + { + await session.App.OpenAsync("/"); + await session.Agent.ActAsync(Instruction); + }); - Assert.Null(result.Error); Assert.True(model.CallCount >= 2); Assert.Equal( - ["back", "check", "clear", "close_tab", "done", "double_tap", "fill", "fill_secret", "navigate", "observe", "press", "press_at", "screenshot", "scroll", "scroll_to", "select", "select_at", "switch_tab", "tap", "tap_at", "type_at", "uncheck"], + ["back", "check", "close_tab", "complete_step", "double_tap", "drag", "drag_at", "hover", "hover_at", "long_press", "navigate", "observe", "press", "press_at", "resize_viewport", "right_click", "screenshot", "scroll", "scroll_to", "select", "select_at", "switch_tab", "tap", "tap_at", "type", "type_at", "upload"], model.Requests[0].Tools.Select(tool => tool.Name).Order(StringComparer.Ordinal)); var opening = model.Requests[0].Messages[0].Content!; - Assert.Contains("Goal: " + Instruction, opening, StringComparison.Ordinal); - Assert.Contains("button \"Increment\"", opening, StringComparison.Ordinal); - // The tap result shows the screen it produced, not the tree the model already had. - Assert.Contains("status \"Counter\" text=\"1\"", CounterPage.ToolResults(model)[0], StringComparison.Ordinal); + Assert.Contains("increment the counter once", opening, StringComparison.Ordinal); + Assert.Matches(@"Current screen \(revision b\d+, path /, \d+ nodes\):", opening); + // The result reports the change since the opening screen, not the tree again. + var result = CounterPage.ToolResults(model)[0]; + Assert.Contains("Screen changes since revision", result, StringComparison.Ordinal); + Assert.Matches(@"changed #\S+ status ""Counter"" text=""1"" \(was: #\S+ status ""Counter"" text=""0""\)", result); + Assert.DoesNotContain("button \"Increment\"", result, StringComparison.Ordinal); } [Fact] @@ -112,6 +109,17 @@ public async Task Accounts_executor_model_calls_in_the_step_metrics_and_provenan Assert.Equal(calls, step["events"]!.AsArray().Count(each => (string)each!["kind"]! == "model")); } + private const string CounterSitePage = """ + + + Counter + + +

0

+ + + """; + private static ScriptedModel IncrementThenConclude() => new(IncrementThenConcludeTurn); private static ModelResponse IncrementThenConcludeTurn(ModelRequest request) diff --git a/tests/E2E.Tests/AgentAct/LoopGuardsAndTranscriptsTests.cs b/tests/E2E.Tests/AgentAct/LoopGuardsAndTranscriptsTests.cs index eccadc39d..6948e1d08 100644 --- a/tests/E2E.Tests/AgentAct/LoopGuardsAndTranscriptsTests.cs +++ b/tests/E2E.Tests/AgentAct/LoopGuardsAndTranscriptsTests.cs @@ -16,7 +16,7 @@ public async Task Forces_a_verdict_when_the_model_repeats_itself() { var model = new ScriptedModel(request => { - if (request.Tools.Count == 1 && request.Tools[0].Name == "done") + if (request.Tools.Count == 1 && request.Tools[0].Name == "complete_step") { return ModelResponses.Done("failed", "stuck repeating the same tap"); } diff --git a/tests/E2E.Tests/AgentAct/TabsEngine.cs b/tests/E2E.Tests/AgentAct/TabsEngine.cs deleted file mode 100644 index 248babf4f..000000000 --- a/tests/E2E.Tests/AgentAct/TabsEngine.cs +++ /dev/null @@ -1,54 +0,0 @@ -// Copyright 2026 TesterArmy. -// Modified by Dario Kondratiuk. -// SPDX-License-Identifier: Apache-2.0 - -using System.Reflection; -using E2E.Engine; - -namespace E2E.Tests.AgentAct; - -/// -/// Wraps an engine and declares tabs (.NET-only): upstream's toolset test runs the web engine, which has tabs, so the agent is offered -/// switch_tab and close_tab. The document engine has no tabs, so this adds the two capabilities and accepts the two verbs. -/// -internal sealed class TabsEngine(IEngine inner) : IEngine -{ - public string Name => inner.Name; - - public string Platform => inner.Platform; - - public string Version => inner.Version; - - public EngineCapabilities Capabilities => inner.Capabilities | EngineCapabilities.SwitchTab | EngineCapabilities.CloseTab; - - public async Task StartAsync(EngineStartOptions options, CancellationToken cancellationToken) - { - var session = await inner.StartAsync(options, cancellationToken).ConfigureAwait(false); - var proxy = DispatchProxy.Create(); - ((TabsSession)(object)proxy).Inner = session; - return proxy; - } - - public class TabsSession : DispatchProxy - { - public IEngineSession Inner { get; set; } = null!; - - protected override object? Invoke(MethodInfo? targetMethod, object?[]? args) - { - if (targetMethod!.Name is nameof(IEngineSession.SwitchTabAsync) or nameof(IEngineSession.CloseTabAsync)) - { - return Task.CompletedTask; - } - - try - { - return targetMethod.Invoke(Inner, args); - } - catch (TargetInvocationException ex) when (ex.InnerException is not null) - { - System.Runtime.ExceptionServices.ExceptionDispatchInfo.Capture(ex.InnerException).Throw(); - throw; - } - } - } -} diff --git a/tests/E2E.Tests/AgentActPixelTools/AgentActPixelVerbsTests.Points.cs b/tests/E2E.Tests/AgentActPixelTools/AgentActPixelVerbsTests.Points.cs index 150d17ec1..073b355c7 100644 --- a/tests/E2E.Tests/AgentActPixelTools/AgentActPixelVerbsTests.Points.cs +++ b/tests/E2E.Tests/AgentActPixelTools/AgentActPixelVerbsTests.Points.cs @@ -332,14 +332,14 @@ public async Task Offers_fill_and_press_without_a_target_typing_into_the_focused { 0 => ModelResponses.Call("screenshot", new { }), 1 => ModelResponses.Call("tap_at", new { x = 120, y = 60 }), - 2 => ModelResponses.Call("fill", new { value = "ok" }), + 2 => ModelResponses.Call("type", new { value = "ok" }), 3 => ModelResponses.Call("press", new { key = "Enter" }), _ => Done(), }, screen => HasStatus(screen, "drawn: ok")); Assert.Null(run.Error); Assert.Equal(["tapAt", "typeText", "pressKey"], EngineEvents(run).Select(item => item.Name)); - Assert.Contains(run.Requests[0].Tools, tool => tool.Name == "fill"); + Assert.Contains(run.Requests[0].Tools, tool => tool.Name == "type"); Assert.DoesNotContain(run.Requests[0].Tools, tool => tool.Name == "dismiss_keyboard"); Assert.Contains("Typed into the focused field.", Result(run, 3), StringComparison.Ordinal); } @@ -352,7 +352,7 @@ public async Task Falls_back_to_focusing_an_unfillable_listed_node_and_typing_th "type yo into the pad", calls => calls switch { - 0 => ModelResponses.Call("fill", new { role = "application", name = "Pad", value = "yo" }), + 0 => ModelResponses.Call("type", new { role = "application", name = "Pad", value = "yo" }), 1 => ModelResponses.Call("press", new { role = "application", name = "Pad", key = "Enter" }), _ => Done(), }, @@ -373,7 +373,7 @@ public async Task Refuses_focused_typing_into_nothing_editable_as_an_action_fail var run = await ActOnAsync( "/", "type blind", - calls => calls == 0 ? ModelResponses.Call("fill", new { value = "nope" }) : Done(passed: false)); + calls => calls == 0 ? ModelResponses.Call("type", new { value = "nope" }) : Done(passed: false)); Assert.NotNull(run.Error); // The engine refused before any keystroke: the action was reserved, not committed. Assert.DoesNotContain(EngineEvents(run), item => item.Status == StepEventStatus.Passed); @@ -401,7 +401,7 @@ public async Task Counts_failing_point_verbs_toward_the_failure_streak_a_warning Assert.Equal(1, run.Requests[2].Messages[^1].Images?.Count); Assert.Contains(run.Requests, request => request.Messages.Any(message => message.Role == "user" && message.Content?.Contains("[SYSTEM NOTICE] the last 3 actions failed in a row", StringComparison.Ordinal) == true)); var last = run.Requests[^1]; - Assert.Equal(["done"], last.Tools.Select(tool => tool.Name)); + Assert.Equal(["complete_step"], last.Tools.Select(tool => tool.Name)); Assert.Contains("[SYSTEM] Loop guard: the last 5 actions failed in a row", last.Messages[^1].Content, StringComparison.Ordinal); // The loop notes the warning and the stop on the turn they followed, as upstream's note() does. var turns = Assert.IsAssignableFrom>(run.Step.Turns); @@ -419,7 +419,7 @@ public async Task Keeps_every_pixel_verb_in_the_vocabulary_after_a_secret_fill() before: screen => screen.GetByLabel("Password").FillAsync(Secret.Create("memberPassword", "hunter2-secret"))); Assert.Null(run.Error); var first = run.Requests[0]; - foreach (var verb in new[] { "screenshot", "tap_at", "drag_at", "type_at", "press_at", "select_at", "tap", "fill" }) + foreach (var verb in new[] { "screenshot", "tap_at", "drag_at", "type_at", "press_at", "select_at", "tap", "type" }) { Assert.Contains(first.Tools, tool => tool.Name == verb); } diff --git a/tests/E2E.Tests/AgentActStress/SecretFillPolicyUnderAHostileModelTests.cs b/tests/E2E.Tests/AgentActStress/SecretFillPolicyUnderAHostileModelTests.cs index df1aedc52..ca4a50ae3 100644 --- a/tests/E2E.Tests/AgentActStress/SecretFillPolicyUnderAHostileModelTests.cs +++ b/tests/E2E.Tests/AgentActStress/SecretFillPolicyUnderAHostileModelTests.cs @@ -31,8 +31,8 @@ public async Task Fills_a_generic_secret_into_a_contenteditable_host_and_refuses toolResults.Add(LastToolResult(request)); return calls++ switch { - 0 => ModelResponses.Call("fill_secret", new { role = "textbox", name = "Notes", secret = "admin" }), - 1 => ModelResponses.Call("fill_secret", new { role = "textbox", name = "Notes", secret = "stripe-key" }), + 0 => ModelResponses.Call("type_secret", new { role = "textbox", name = "Notes", secret = "admin" }), + 1 => ModelResponses.Call("type_secret", new { role = "textbox", name = "Notes", secret = "stripe-key" }), _ => ModelResponses.Done("passed", "filled the API key"), }; }); @@ -62,7 +62,7 @@ public async Task Never_leaks_the_plaintext_into_prompts_or_tool_results() { using var site = await WebFixtureApp.StartAsync(); var model = new ScriptedModel(request => LastToolResult(request) == "" - ? ModelResponses.Call("fill_secret", new { role = "textbox", name = "Password", secret = "admin" }) + ? ModelResponses.Call("type_secret", new { role = "textbox", name = "Password", secret = "admin" }) : ModelResponses.Done("passed", "filled the secret")); await using var session = await StartAsync(site.Url, model); await WebSemanticsTests.RunAsync(session, async () => diff --git a/tests/E2E.Tests/AgentAssertExplore/AgentAssertExploreTests.cs b/tests/E2E.Tests/AgentAssertExplore/AgentAssertExploreTests.cs index ae09c419c..6fde69e0e 100644 --- a/tests/E2E.Tests/AgentAssertExplore/AgentAssertExploreTests.cs +++ b/tests/E2E.Tests/AgentAssertExplore/AgentAssertExploreTests.cs @@ -81,7 +81,7 @@ public async Task Routes_built_in_exploring_asserts_through_a_scroll_only_loop_w Assert.Null(run.Error); Assert.Equal(2, model.CallCount); - Assert.Equal(["done", "observe", "screenshot", "scroll", "scroll_to"], model.Requests[0].Tools.Select(tool => tool.Name).Order(StringComparer.Ordinal)); + Assert.Equal(["complete_step", "observe", "screenshot", "scroll", "scroll_to"], model.Requests[0].Tools.Select(tool => tool.Name).Order(StringComparer.Ordinal)); Assert.True(Images(model.Requests[0]) > 0); Assert.DoesNotContain("Increment", model.Requests[0].Messages[0].Content, StringComparison.Ordinal); Assert.DoesNotContain("Completed steps", model.Requests[0].Messages[0].Content, StringComparison.Ordinal); diff --git a/tests/E2E.Tests/AgentConfig/TheBuiltInAgentOptionsTests.cs b/tests/E2E.Tests/AgentConfig/TheBuiltInAgentOptionsTests.cs index 27066a0f8..827545384 100644 --- a/tests/E2E.Tests/AgentConfig/TheBuiltInAgentOptionsTests.cs +++ b/tests/E2E.Tests/AgentConfig/TheBuiltInAgentOptionsTests.cs @@ -6,9 +6,8 @@ namespace E2E.Tests.AgentConfig; -// Upstream's "rejects the names of the tools the harness adds in an agent step, an e2e mcp session, and explore" is ported as -// Partial_ (the port's verdict tool is done, not complete_step; the reserved names are a hand-kept list). Neither has a model -// instance as the entry. +// Upstream's "rejects the names of the tools the harness adds in an agent step, an e2e mcp session, and explore" is ported with a +// hand-kept list of the reserved names. Neither has a model instance as the entry. public sealed class TheBuiltInAgentOptionsTests { private static readonly string Root = Path.Combine(Path.GetTempPath(), "e2e-agent-config-project"); @@ -66,8 +65,7 @@ private sealed class CircularStore : IStepCache /// /// Port of "rejects the names of the tools the harness adds in an agent step, an e2e mcp session, and explore": every name in - /// upstream's HARNESS_TOOL_NAMES is refused with its surface. The port's verdict tool is done, which is refused - /// below with the agent's own names. The list is kept by hand, so a new harness tool needs a new entry here. + /// upstream's HARNESS_TOOL_NAMES is refused with its surface. The list is kept by hand, so a new harness tool needs a new entry here. /// [Fact] public void Rejects_the_names_of_the_tools_the_harness_adds_in_an_agent_step_an_e2e_mcp_session_and_explore() @@ -89,9 +87,6 @@ public void Rejects_the_names_of_the_tools_the_harness_adds_in_an_agent_step_an_ Assert.Equal("INVALID_CONFIG", error.Code); Assert.Equal($"agents.ux.tools.{name}: the {name} tool name is reserved for the tool the harness adds in {surface}; rename it", error.Message); } - - var agentStep = Assert.Throws(() => ResolvedAgent.From("ux", new AgentOptions { Tools = new Dictionary { ["done"] = ReadOnlyTool() } })); - Assert.Equal("agents.ux.tools.done: the done tool name is reserved for the agent's own tools", agentStep.Message); } /// @@ -102,7 +97,7 @@ public void Rejects_the_names_of_the_tools_the_harness_adds_in_an_agent_step_an_ [Fact] public void Partial_Rejects_a_reserved_tool_name() { - foreach (var name in new[] { "tap", "observe", "done" }) + foreach (var name in new[] { "tap", "observe" }) { var error = Assert.Throws(() => ResolvedAgent.From("ux", new AgentOptions { Tools = new Dictionary { [name] = ReadOnlyTool() } })); diff --git a/tests/E2E.Tests/AgentExecutorContext/AttachedTurnsTests.cs b/tests/E2E.Tests/AgentExecutorContext/AttachedTurnsTests.cs index 3c45c28ac..a2cb3fc9f 100644 --- a/tests/E2E.Tests/AgentExecutorContext/AttachedTurnsTests.cs +++ b/tests/E2E.Tests/AgentExecutorContext/AttachedTurnsTests.cs @@ -66,8 +66,8 @@ public async Task The_tool_loop_records_each_turn_with_its_calls_and_results_and Assert.Equal("assistant: I will look around first.\n[loop] turn 1 made no tool call: asking for one", turns[0].Outcome); Assert.StartsWith("tap(", Assert.Single(turns[1].Calls), StringComparison.Ordinal); Assert.StartsWith("[tap] ", turns[1].Outcome, StringComparison.Ordinal); - Assert.StartsWith("done(", Assert.Single(turns[2].Calls), StringComparison.Ordinal); - Assert.Equal("[done] Step concluded.", turns[2].Outcome); + Assert.StartsWith("complete_step(", Assert.Single(turns[2].Calls), StringComparison.Ordinal); + Assert.Equal("[complete_step] Step concluded.", turns[2].Outcome); } [Fact] diff --git a/tests/E2E.Tests/AgentGrammarTools/CreateGrammarToolsTests.cs b/tests/E2E.Tests/AgentGrammarTools/CreateGrammarToolsTests.cs index 2648a02b8..4daf3d9ca 100644 --- a/tests/E2E.Tests/AgentGrammarTools/CreateGrammarToolsTests.cs +++ b/tests/E2E.Tests/AgentGrammarTools/CreateGrammarToolsTests.cs @@ -62,7 +62,7 @@ public async Task Keeps_the_built_in_actions_beside_a_tool_of_its_own() Assert.Contains("observe", offered); Assert.Contains("tap", offered); Assert.Contains("read_inbox", offered); - Assert.Equal("done", offered[^1]); + Assert.Equal("complete_step", offered[^1]); Assert.StartsWith(DefaultAgent.BaseRules, model.Requests[0].System, StringComparison.Ordinal); Assert.EndsWith(AgentTools.VerdictRules + "\n- " + global::E2E.ToolLoopExecutor.ToolCallsOnlyRule, model.Requests[0].System, StringComparison.Ordinal); Assert.Equal(1, act!.Actions); diff --git a/tests/E2E.Tests/AgentGrammarTools/TheGrammarToolsHaveClosedSchemasTests.cs b/tests/E2E.Tests/AgentGrammarTools/TheGrammarToolsHaveClosedSchemasTests.cs index 7548f1850..71579185b 100644 --- a/tests/E2E.Tests/AgentGrammarTools/TheGrammarToolsHaveClosedSchemasTests.cs +++ b/tests/E2E.Tests/AgentGrammarTools/TheGrammarToolsHaveClosedSchemasTests.cs @@ -10,8 +10,7 @@ namespace E2E.Tests.AgentGrammarTools; // Upstream validates a call against the zod schema of a tool (safeParse). The port's schema is the JSON Schema a provider receives, so these // tests validate against it (JsonSchemaCheck) and read the refusal text of the closed schema through AgentTools.SchemaTool. The port's -// vocabulary differs from upstream's in names only (type is fill, type_secret is fill_secret, target is role, name, or ref, complete_step -// is done, errorCode is code, and check takes no checked flag but has uncheck). The tests keep upstream's `it` text. +// vocabulary differs from upstream's in names only (target is role, name, or ref, and errorCode is code). The tests keep upstream's `it` text. public sealed class TheGrammarToolsHaveClosedSchemasTests { /// An input each tool accepts; a tool added to the grammar without one fails the vocabulary check. @@ -23,13 +22,11 @@ public sealed class TheGrammarToolsHaveClosedSchemasTests ["long_press"] = """{ "ref": "n1" }""", ["right_click"] = """{ "ref": "n1" }""", ["hover"] = """{ "ref": "n1" }""", - ["fill"] = """{ "ref": "n1", "value": "ada" }""", - ["fill_secret"] = """{ "ref": "n1", "secret": "admin" }""", + ["type"] = """{ "ref": "n1", "value": "ada" }""", + ["type_secret"] = """{ "ref": "n1", "secret": "admin" }""", ["press"] = """{ "ref": "n1", "key": "Shift+ArrowLeft", "times": 8 }""", ["select"] = """{ "ref": "n1", "value": "Pro" }""", - ["check"] = """{ "ref": "n1" }""", - ["uncheck"] = """{ "ref": "n1" }""", - ["clear"] = """{ "ref": "n1" }""", + ["check"] = """{ "ref": "n1", "checked": false }""", ["scroll"] = """{ "direction": "down", "times": 2 }""", ["scroll_to"] = """{ "ref": "n1" }""", ["drag"] = """{ "ref": "n1", "to": { "ref": "n2" } }""", @@ -122,9 +119,9 @@ public void Closes_the_keyboard_only_shapes_too_type_and_press_without_a_target_ { var tools = OverVerbs(ExecutorVerb.TypeText, ExecutorVerb.PressKey, ExecutorVerb.Scroll); Assert.Equal( - ["fill", "observe", "press", "press_at", "screenshot", "scroll", "type_at"], + ["observe", "press", "press_at", "screenshot", "scroll", "type", "type_at"], tools.Select(tool => tool.Name).Order(StringComparer.Ordinal)); - var fill = tools.Single(tool => tool.Name == "fill"); + var fill = tools.Single(tool => tool.Name == "type"); ExpectClosed(fill, """{ "value": "ada", "replace": true }"""); ExpectClosed(tools.Single(tool => tool.Name == "press"), """{ "key": "Enter" }"""); ExpectClosed(tools.Single(tool => tool.Name == "scroll"), """{ "direction": "down" }"""); @@ -183,10 +180,10 @@ public void Accepts_a_passed_verdict_beside_an_error_code_the_body_records_the_p Assert.True(schema.IsValid(Input("""{ "status": "passed", "summary": "done", "code": "ACTION_FAILED" }"""))); Assert.Equal( "Step concluded as passed; the code ACTION_FAILED you sent does not apply to a passed verdict and was ignored.", - passed.Execute(Call("done", Input("""{ "status": "passed", "summary": "done", "code": "ACTION_FAILED" }"""))).Content); + passed.Execute(Call("complete_step", Input("""{ "status": "passed", "summary": "done", "code": "ACTION_FAILED" }"""))).Content); Assert.Equal(new StepVerdict(StepVerdictStatus.Passed, "done"), passed.Verdict); var failed = AgentTools.CreateVerdictTool(); - Assert.Equal("Step concluded.", failed.Execute(Call("done", Input("""{ "status": "failed", "summary": "done", "code": "ACTION_FAILED" }"""))).Content); + Assert.Equal("Step concluded.", failed.Execute(Call("complete_step", Input("""{ "status": "failed", "summary": "done", "code": "ACTION_FAILED" }"""))).Content); Assert.Equal(new StepVerdict(StepVerdictStatus.Failed, "done", "ACTION_FAILED"), failed.Verdict); Assert.True(schema.IsValid(Input("""{ "status": "blocked", "summary": "done", "code": "ENVIRONMENT_UNAVAILABLE" }"""))); } @@ -195,8 +192,8 @@ public void Accepts_a_passed_verdict_beside_an_error_code_the_body_records_the_p public void Keeps_the_first_verdict_a_later_complete_step_in_the_same_turn_cannot_turn_a_failure_into_a_pass() { var conclusion = AgentTools.CreateVerdictTool(); - Assert.Equal("Step concluded.", conclusion.Execute(Call("done", Input("""{ "status": "failed", "summary": "total is wrong", "code": "ASSERTION_FAILED" }"""))).Content); - Assert.Equal("The step already concluded.", conclusion.Execute(Call("done", Input("""{ "status": "passed", "summary": "looks fine" }"""))).Content); + Assert.Equal("Step concluded.", conclusion.Execute(Call("complete_step", Input("""{ "status": "failed", "summary": "total is wrong", "code": "ASSERTION_FAILED" }"""))).Content); + Assert.Equal("The step already concluded.", conclusion.Execute(Call("complete_step", Input("""{ "status": "passed", "summary": "looks fine" }"""))).Content); Assert.Equal(new StepVerdict(StepVerdictStatus.Failed, "total is wrong", "ASSERTION_FAILED"), conclusion.Verdict); } @@ -206,10 +203,10 @@ public void Keeps_the_first_verdict_a_later_complete_step_in_the_same_turn_canno public void Refuses_a_blocked_verdict_without_a_blocking_error_code_and_leaves_the_step_open_for_a_real_one() { var conclusion = AgentTools.CreateVerdictTool(); - Assert.StartsWith("failed: a blocked verdict requires a code", conclusion.Execute(Call("done", Input("""{ "status": "blocked", "summary": "could not tell" }"""))).Content, StringComparison.Ordinal); - Assert.StartsWith("failed: ", conclusion.Execute(Call("done", Input("""{ "status": "blocked", "summary": "the total is wrong", "code": "ASSERTION_FAILED" }"""))).Content, StringComparison.Ordinal); + Assert.StartsWith("failed: a blocked verdict requires a code", conclusion.Execute(Call("complete_step", Input("""{ "status": "blocked", "summary": "could not tell" }"""))).Content, StringComparison.Ordinal); + Assert.StartsWith("failed: ", conclusion.Execute(Call("complete_step", Input("""{ "status": "blocked", "summary": "the total is wrong", "code": "ASSERTION_FAILED" }"""))).Content, StringComparison.Ordinal); Assert.False(conclusion.Concluded); - Assert.Equal("Step concluded.", conclusion.Execute(Call("done", Input("""{ "status": "blocked", "summary": "no login", "code": "AUTH_CREDENTIAL_UNAVAILABLE" }"""))).Content); + Assert.Equal("Step concluded.", conclusion.Execute(Call("complete_step", Input("""{ "status": "blocked", "summary": "no login", "code": "AUTH_CREDENTIAL_UNAVAILABLE" }"""))).Content); Assert.Equal(new StepVerdict(StepVerdictStatus.Blocked, "no login", "AUTH_CREDENTIAL_UNAVAILABLE"), conclusion.Verdict); } @@ -220,7 +217,7 @@ public void Offers_the_node_verbs_and_the_scroll_and_history_tools_their_capabil { var full = AgentTools.GrammarToolDefinitions(EngineCapabilities.Scroll | EngineCapabilities.History, null, EngineActionKinds.Locator); Assert.Equal( - ["back", "check", "clear", "double_tap", "drag", "drag_at", "fill", "fill_secret", "hover", "hover_at", "long_press", "navigate", "observe", "press", "press_at", "right_click", "screenshot", "scroll", "scroll_to", "select", "select_at", "tap", "tap_at", "type_at", "uncheck"], + ["back", "check", "double_tap", "drag", "drag_at", "hover", "hover_at", "long_press", "navigate", "observe", "press", "press_at", "right_click", "screenshot", "scroll", "scroll_to", "select", "select_at", "tap", "tap_at", "type", "type_at", "type_secret"], full.Select(tool => tool.Name).Order(StringComparer.Ordinal)); // An engine that declares no action kind has no node verb: only what its capabilities unlock is left. @@ -237,7 +234,7 @@ public void Offers_the_node_verbs_and_the_scroll_and_history_tools_their_capabil // .NET-only: the node verbs follow the action kinds the engine declares (upstream's KINDS_BY_VERB), and the point verbs that act on a // listed control follow the node verb they fall back to. Upstream gates tap on tap, double_tap on doubleTap, type and type_secret on fill, - // press on press, select on selectOption, and check on both check and uncheck; the port's clear follows clear and uncheck follows check. + // press on press, select on selectOption, and check on both check and uncheck. [Fact] public void Offers_each_node_verb_only_for_the_action_kinds_the_engine_declares() { @@ -246,30 +243,30 @@ static string[] Offered(EngineCapabilities capabilities, params string[] kinds) Assert.Equal(["tap", "tap_at"], Offered(EngineCapabilities.None, "tap")); Assert.Equal(["double_tap"], Offered(EngineCapabilities.None, "doubleTap")); - Assert.Equal(["fill", "fill_secret", "type_at"], Offered(EngineCapabilities.None, "fill")); + Assert.Equal(["type", "type_at", "type_secret"], Offered(EngineCapabilities.None, "fill")); Assert.Equal(["press", "press_at"], Offered(EngineCapabilities.None, "press")); Assert.Equal(["select", "select_at"], Offered(EngineCapabilities.None, "selectOption")); - Assert.Equal(["clear"], Offered(EngineCapabilities.None, "clear")); + Assert.Empty(Offered(EngineCapabilities.None, "clear")); // A check sets a state rather than flipping one, so it needs both directions. Assert.Empty(Offered(EngineCapabilities.None, "check")); Assert.Empty(Offered(EngineCapabilities.None, "uncheck")); - Assert.Equal(["check", "uncheck"], Offered(EngineCapabilities.None, "check", "uncheck")); + Assert.Equal(["check"], Offered(EngineCapabilities.None, "check", "uncheck")); // Kinds that unlock no node verb unlock none (a focus is no verb). Assert.Empty(Offered(EngineCapabilities.None, "focus", "scrollIntoView", "setInputFiles")); Assert.Empty(Offered(EngineCapabilities.None)); // A keyboard takes fill and press without a target, with or without the kinds, and the point tools that type and press. - Assert.Equal(["fill", "press", "press_at", "type_at"], Offered(EngineCapabilities.Keyboard)); + Assert.Equal(["press", "press_at", "type", "type_at"], Offered(EngineCapabilities.Keyboard)); // With both kinds and the keyboard, the target is optional and a call without one acts on the focused field; with fill alone // the schema carries a target too; with the keyboard alone it carries none. static bool HasTarget(ModelTool tool) => tool.Parameters.GetProperty("properties").TryGetProperty("ref", out _); - Assert.True(HasTarget(AgentTools.GrammarToolDefinitions(EngineCapabilities.Keyboard, null, ["fill", "press"]).Single(tool => tool.Name == "fill"))); + Assert.True(HasTarget(AgentTools.GrammarToolDefinitions(EngineCapabilities.Keyboard, null, ["fill", "press"]).Single(tool => tool.Name == "type"))); Assert.True(HasTarget(AgentTools.GrammarToolDefinitions(EngineCapabilities.Keyboard, null, ["fill", "press"]).Single(tool => tool.Name == "press"))); - Assert.True(HasTarget(AgentTools.GrammarToolDefinitions(EngineCapabilities.None, null, ["fill"]).Single(tool => tool.Name == "fill"))); - Assert.False(HasTarget(AgentTools.GrammarToolDefinitions(EngineCapabilities.Keyboard, null, ["tap"]).Single(tool => tool.Name == "fill"))); + Assert.True(HasTarget(AgentTools.GrammarToolDefinitions(EngineCapabilities.None, null, ["fill"]).Single(tool => tool.Name == "type"))); + Assert.False(HasTarget(AgentTools.GrammarToolDefinitions(EngineCapabilities.Keyboard, null, ["tap"]).Single(tool => tool.Name == "type"))); // The same set decides the verbs of the executor's target, and a call to a verb the engine did not unlock is refused as unknown. Assert.Equal( diff --git a/tests/E2E.Tests/AgentPolicy/ToolCallsOutsideTheVocabularyTests.cs b/tests/E2E.Tests/AgentPolicy/ToolCallsOutsideTheVocabularyTests.cs index dd55f8493..8e5511781 100644 --- a/tests/E2E.Tests/AgentPolicy/ToolCallsOutsideTheVocabularyTests.cs +++ b/tests/E2E.Tests/AgentPolicy/ToolCallsOutsideTheVocabularyTests.cs @@ -102,8 +102,8 @@ public async Task Drops_the_error_code_a_passed_verdict_carries_notes_it_in_the_ Assert.Equal(("the counter shows 1", 2, 1), (act.Summary, act.ModelCalls, act.Actions)); await Expect.That(session.Screen.GetByRole("status")).ToHaveTextAsync("1"); var verdictTurn = Assert.IsAssignableFrom>(AgentStep(session).Turns)[^1]; - Assert.StartsWith("done(", Assert.Single(verdictTurn.Calls), StringComparison.Ordinal); - Assert.Equal("[done] Step concluded as passed; the code ACTION_FAILED you sent does not apply to a passed verdict and was ignored.", verdictTurn.Outcome); + Assert.StartsWith("complete_step(", Assert.Single(verdictTurn.Calls), StringComparison.Ordinal); + Assert.Equal("[complete_step] Step concluded as passed; the code ACTION_FAILED you sent does not apply to a passed verdict and was ignored.", verdictTurn.Outcome); } [Fact] @@ -142,7 +142,7 @@ public async Task Counts_refusals_toward_the_failure_streak_so_a_model_that_neve var turn = 0; var model = new ScriptedModel(request => { - if (request.Tools.Count == 1 && request.Tools[0].Name == "done") + if (request.Tools.Count == 1 && request.Tools[0].Name == "complete_step") { return ModelResponses.Done("failed", "could not get a call past the schema"); } @@ -159,7 +159,7 @@ public async Task Counts_refusals_toward_the_failure_streak_so_a_model_that_neve // Three refusals warn, five force the verdict; a model this stuck never reaches the turn budget. Assert.True(model.CallCount <= 8, "model calls: " + model.CallCount); Assert.Contains(model.Requests, request => request.Messages.Any(message => message.Content?.Contains("failed in a row", StringComparison.Ordinal) == true)); - Assert.Equal(["done"], model.Requests[^1].Tools.Select(tool => tool.Name)); + Assert.Equal(["complete_step"], model.Requests[^1].Tools.Select(tool => tool.Name)); } private static StepRecord AgentStep(E2ESession session) => session.StepRecords.Single(step => step.Api == "agent.act"); diff --git a/tests/E2E.Tests/AgentTapModifiers/TapModifiersInTheGrammarToolsTests.cs b/tests/E2E.Tests/AgentTapModifiers/TapModifiersInTheGrammarToolsTests.cs index 195d98da9..8873690b2 100644 --- a/tests/E2E.Tests/AgentTapModifiers/TapModifiersInTheGrammarToolsTests.cs +++ b/tests/E2E.Tests/AgentTapModifiers/TapModifiersInTheGrammarToolsTests.cs @@ -45,7 +45,7 @@ private static DocumentWorld World() var model = new ScriptedModel(_ => next < calls.Length ? calls[next++] : ModelResponses.Done("passed", "Done.")); var step = await StepCachePipelineTests.StepAsync(World(), new FileStepCache(CoreTests.TempCache()), "/", null, model, wrap: inner => declared ? engine = new TapModifiersEngine(inner) : inner); Assert.Null(step.Error); - var results = model.Requests[^1].Messages.Where(message => message is { Role: "tool" } && message.Name != "done").Select(message => message.Content ?? "").ToList(); + var results = model.Requests[^1].Messages.Where(message => message is { Role: "tool" } && message.Name != "complete_step").Select(message => message.Content ?? "").ToList(); return (results, engine); } diff --git a/tests/E2E.Tests/Explore/E2eExploreTests.cs b/tests/E2E.Tests/Explore/E2eExploreTests.cs index 4db64c9d6..80814e232 100644 --- a/tests/E2E.Tests/Explore/E2eExploreTests.cs +++ b/tests/E2E.Tests/Explore/E2eExploreTests.cs @@ -497,7 +497,7 @@ public async Task Replaces_a_custom_executor_with_the_built_in_agent_on_the_mode } [Fact] - public async Task Lets_the_explorer_sign_in_with_a_configured_credential_through_fill_secret_never_seeing_the_password() + public async Task Lets_the_explorer_sign_in_with_a_configured_credential_through_type_secret_never_seeing_the_password() { var planPrompts = new List(); string? fillResult = null; @@ -514,7 +514,7 @@ public async Task Lets_the_explorer_sign_in_with_a_configured_credential_through { if (call.Turn == 1) { - return ModelResponses.Call("fill_secret", new { role = "textbox", name = "Password", secret = "ada.password" }); + return ModelResponses.Call("type_secret", new { role = "textbox", name = "Password", secret = "ada.password" }); } fillResult = call.LastToolResult; @@ -533,7 +533,7 @@ public async Task Lets_the_explorer_sign_in_with_a_configured_credential_through Assert.Null(outcome.Error); Assert.Equal(RunStatus.Passed, outcome.Status); Assert.Contains("- ada (username: ada@example.test)", planPrompts[0], StringComparison.Ordinal); - Assert.Contains("fill_secret", model.LoopCalls.First().Tools.Select(tool => tool.Name)); + Assert.Contains("type_secret", model.LoopCalls.First().Tools.Select(tool => tool.Name)); Assert.Contains("\"password\":\" (password)\"", model.LoopCalls.First().Messages[0].Content, StringComparison.Ordinal); Assert.Contains("ada.password", fillResult, StringComparison.Ordinal); Assert.Equal("bookworm", typed); diff --git a/tests/E2E.Tests/LoopGuards/FailureStreakTests.cs b/tests/E2E.Tests/LoopGuards/FailureStreakTests.cs index de5e6c1fd..e2044697f 100644 --- a/tests/E2E.Tests/LoopGuards/FailureStreakTests.cs +++ b/tests/E2E.Tests/LoopGuards/FailureStreakTests.cs @@ -12,7 +12,7 @@ public sealed class FailureStreakTests { private const string Failed = "tap #n6 failed: LOCATOR_NOT_FOUND: node #n6 is not on the current screen (observation b3); it was removed or never existed\n\nCurrent screen (revision b3, 2 nodes):\n#n1 document\n #n2 heading \"X\""; private const string Tapped = "Tapped #n6.\n\nScreen unchanged since revision b3 (2 nodes)."; - private const string Conclude = "done"; + private const string Conclude = "complete_step"; private static readonly E2E.GuardToolResult Failure = new(Failed, Failed: true); private static readonly E2E.GuardToolResult Success = new(Tapped, Failed: false); @@ -76,7 +76,7 @@ public void Partial_Reads_text_results_for_the_failure_shape_counts_a_call_the_S { // What the SDK hands back for an input outside the closed schema and for a tool the step does not offer. const string undeclaredField = "failed: Invalid input for tool tap: Type validation failed: Value: {\"target\":\"n6\",\"force\":true}.\nError message: [{\"code\":\"unrecognized_keys\",\"keys\":[\"force\"],\"path\":[],\"message\":\"Unrecognized key: \\\"force\\\"\"}]"; - const string unknownTool = "failed: Model tried to call unavailable tool 'click'. Available tools: observe, tap, type, done."; + const string unknownTool = "failed: Model tried to call unavailable tool 'click'. Available tools: observe, tap, type, complete_step."; var messages = new List { @@ -92,20 +92,6 @@ public void Partial_Reads_text_results_for_the_failure_shape_counts_a_call_the_S E2E.LoopGuards.ExtractToolResults(messages, Conclude)); } - /// - /// .NET-only: the port's own tool names (fill, uncheck, clear) lead their failures with the name they were called by, - /// so the streak reads them as failures. Upstream has no such names; its failure leads with type and no uncheck or clear. - /// - [Fact] - public void Counts_a_failed_port_only_tool_toward_the_streak() - { - var results = E2E.LoopGuards.ExtractToolResults( - [Result("fill", "fill #n3 failed: LOCATOR_NOT_FOUND: node #n3 is not on the current screen"), Result("clear", "clear #n4 failed: LOCATOR_NOT_FOUND: node #n4 is gone"), Result("uncheck", "uncheck #n5 failed: LOCATOR_NOT_FOUND: node #n5 is gone")], - Conclude); - Assert.All(results, result => Assert.True(result.Failed)); - Assert.Equal((false, "the last 3 actions failed in a row"), E2E.LoopGuards.CheckFailureStreak(results)); - } - [Fact] public void Reads_the_text_beside_a_screenshot_so_the_streak_warns_and_stops_in_pixel_mode_too() { @@ -126,7 +112,7 @@ public void Reads_the_text_beside_a_screenshot_so_the_streak_warns_and_stops_in_ public void Counts_a_refused_conclusion_toward_the_streak_three_warn_five_force_the_verdict_and_the_conclusion_tools_own_text_neither_counts_nor_resets() { // The repeat guard never sees conclusion calls, so a model resending a done the schema turns away has only this bound short of the turn budget. - const string refusedVerdict = "failed: Invalid input for tool done: Type validation failed: Value: {\"status\":\"passed\",\"summary\":\"done\",\"force\":true}.\nError message: [{\"code\":\"unrecognized_keys\",\"keys\":[\"force\"],\"path\":[],\"message\":\"Unrecognized key: \\\"force\\\"\"}]"; + const string refusedVerdict = "failed: Invalid input for tool complete_step: Type validation failed: Value: {\"status\":\"passed\",\"summary\":\"done\",\"force\":true}.\nError message: [{\"code\":\"unrecognized_keys\",\"keys\":[\"force\"],\"path\":[],\"message\":\"Unrecognized key: \\\"force\\\"\"}]"; var refused = (int count) => Enumerable.Range(0, count).Select(_ => Result(Conclude, refusedVerdict, errorText: true)).ToList(); Assert.Equal(new E2E.GuardToolResult[] { new(refusedVerdict, Failed: true) }, E2E.LoopGuards.ExtractToolResults(refused(1), Conclude)); diff --git a/tests/E2E.Tests/ModelRequestHeaders/ModelRequestHeadersTests.cs b/tests/E2E.Tests/ModelRequestHeaders/ModelRequestHeadersTests.cs index 06165cb70..03b9e2ddf 100644 --- a/tests/E2E.Tests/ModelRequestHeaders/ModelRequestHeadersTests.cs +++ b/tests/E2E.Tests/ModelRequestHeaders/ModelRequestHeadersTests.cs @@ -63,7 +63,7 @@ public async Task Identifies_e2e_on_every_turn_of_an_act_loop() { var handler = new RecordingHandler( Completion("tap", """{"role":"button","name":"Upgrade to Pro"}"""), - Completion("done", """{"status":"passed","summary":"upgraded"}""")); + Completion("complete_step", """{"status":"passed","summary":"upgraded"}""")); ActResult? result = null; await RunAsync(handler, async ctx => result = await ctx.Agent.ActAsync("upgrade the workspace to the Pro plan")); diff --git a/tests/E2E.Tests/OpenaiSecondTurn/SecondTurnOfAnOpenAiShapedStepTests.cs b/tests/E2E.Tests/OpenaiSecondTurn/SecondTurnOfAnOpenAiShapedStepTests.cs index 40b8a2576..cd277078f 100644 --- a/tests/E2E.Tests/OpenaiSecondTurn/SecondTurnOfAnOpenAiShapedStepTests.cs +++ b/tests/E2E.Tests/OpenaiSecondTurn/SecondTurnOfAnOpenAiShapedStepTests.cs @@ -22,7 +22,7 @@ public sealed class SecondTurnOfAnOpenAiShapedStepTests private const string SecondTurn = """ { "output": [ - { "type": "function_call", "id": "fc_2", "call_id": "call_2", "name": "done", "arguments": "{\"status\":\"passed\",\"summary\":\"upgraded\"}", "status": "completed" } + { "type": "function_call", "id": "fc_2", "call_id": "call_2", "name": "complete_step", "arguments": "{\"status\":\"passed\",\"summary\":\"upgraded\"}", "status": "completed" } ], "usage": { "input_tokens": 3, "output_tokens": 1 } } diff --git a/tests/E2E.Tests/ToolLoopExecutor/CreateToolLoopExecutorTests.cs b/tests/E2E.Tests/ToolLoopExecutor/CreateToolLoopExecutorTests.cs index c692e3c24..3fca40ba2 100644 --- a/tests/E2E.Tests/ToolLoopExecutor/CreateToolLoopExecutorTests.cs +++ b/tests/E2E.Tests/ToolLoopExecutor/CreateToolLoopExecutorTests.cs @@ -70,7 +70,7 @@ public async Task Runs_a_vocabulary_of_its_own_and_adds_the_done_tool() executor: TapOnly()); Assert.Null(result.Error); - Assert.Equal(["read_screen", "tap", "done"], model.Requests[0].Tools.Select(tool => tool.Name)); + Assert.Equal(["read_screen", "tap", "complete_step"], model.Requests[0].Tools.Select(tool => tool.Name)); Assert.Equal(Instruction, model.Requests[0].Messages[0].Content); Assert.Equal("Read the screen before acting.\n\n" + AgentTools.VerdictRules + "\n- " + global::E2E.ToolLoopExecutor.ToolCallsOnlyRule, model.Requests[0].System); Assert.Equal(3, act!.ModelCalls); @@ -177,7 +177,7 @@ public async Task Offers_only_done_once_the_message_preparation_says_to_stop() executor: TapOnly(prepare: (messages, _) => new PreparedMessages(messages, "the step makes no progress"))); Assert.Equal("ACTION_FAILED", Assert.IsType(result.Error).Code); - Assert.Equal(["done"], model.Requests[0].Tools.Select(tool => tool.Name)); + Assert.Equal(["complete_step"], model.Requests[0].Tools.Select(tool => tool.Name)); Assert.Contains(model.Requests[0].Messages, message => message.Content?.Contains("the step makes no progress", StringComparison.Ordinal) == true); } diff --git a/tests/E2E.Tests/ToolLoopPrefix/ToolLoopPrefixHelpers.cs b/tests/E2E.Tests/ToolLoopPrefix/ToolLoopPrefixHelpers.cs index d86a517b4..dfdfa3011 100644 --- a/tests/E2E.Tests/ToolLoopPrefix/ToolLoopPrefixHelpers.cs +++ b/tests/E2E.Tests/ToolLoopPrefix/ToolLoopPrefixHelpers.cs @@ -28,7 +28,7 @@ internal static string Answer(int n, string tool, object input) => internal static string Look(int n) => Answer(n, "look", new { }); - internal static string Done(int n) => Answer(n, "done", new { status = "passed", summary = "done" }); + internal static string Done(int n) => Answer(n, "complete_step", new { status = "passed", summary = "done" }); /// messages.i.content.j=signature for every thinking block the request replays. internal static List ReplayedThinking(RecordedRequest request) => diff --git a/tests/E2E.Tests/ToolLoopToolChoice/ToolLoopForcedConclusionAnsweredWithAnotherToolTests.cs b/tests/E2E.Tests/ToolLoopToolChoice/ToolLoopForcedConclusionAnsweredWithAnotherToolTests.cs index 346c7e681..5872aa687 100644 --- a/tests/E2E.Tests/ToolLoopToolChoice/ToolLoopForcedConclusionAnsweredWithAnotherToolTests.cs +++ b/tests/E2E.Tests/ToolLoopToolChoice/ToolLoopForcedConclusionAnsweredWithAnotherToolTests.cs @@ -49,11 +49,11 @@ public async Task Sends_the_call_back_as_an_error_and_asks_again_on_the_same_tur agentTools: ToolLoopForcedToolChoiceTests.Peek()); Assert.Null(attempt.Error); - Assert.Equal(["required", "tool:done", "tool:done"], model.Requests.Select(request => request.ToolChoice!.Type == "tool" ? "tool:" + request.ToolChoice.ToolName : request.ToolChoice.Type)); + Assert.Equal(["required", "tool:complete_step", "tool:complete_step"], model.Requests.Select(request => request.ToolChoice!.Type == "tool" ? "tool:" + request.ToolChoice.ToolName : request.ToolChoice.Type)); // The first turn offers every tool; the last two are the forced-conclusion turns, which offer only done. Assert.Contains(model.Requests[0].Tools, tool => tool.Name == "peek"); - Assert.Equal(["done"], model.Requests[1].Tools.Select(tool => tool.Name)); - Assert.Equal(["done"], model.Requests[2].Tools.Select(tool => tool.Name)); + Assert.Equal(["complete_step"], model.Requests[1].Tools.Select(tool => tool.Name)); + Assert.Equal(["complete_step"], model.Requests[2].Tools.Select(tool => tool.Name)); Assert.Equal(2, results.Count); Assert.Equal("looked", results[0]); Assert.Contains("peek did not run", results[1], StringComparison.Ordinal); diff --git a/tests/E2E.Tests/ToolLoopToolChoice/ToolLoopForcedToolChoiceTests.cs b/tests/E2E.Tests/ToolLoopToolChoice/ToolLoopForcedToolChoiceTests.cs index 54809389f..4a27cf98d 100644 --- a/tests/E2E.Tests/ToolLoopToolChoice/ToolLoopForcedToolChoiceTests.cs +++ b/tests/E2E.Tests/ToolLoopToolChoice/ToolLoopForcedToolChoiceTests.cs @@ -69,7 +69,7 @@ public async Task Remembers_the_refusal_for_later_steps_on_the_same_model() public async Task Offers_only_complete_step_under_auto_on_the_forced_conclusion_turns() { var loop = new FakeLoopModel(RefusingForcedChoice(call => - call.ToolNames is ["done"] ? Done() : ModelResponses.Call("peek", new { }))); + call.ToolNames is ["complete_step"] ? Done() : ModelResponses.Call("peek", new { }))); var attempt = await RunAsync( async ctx => @@ -82,7 +82,7 @@ public async Task Offers_only_complete_step_under_auto_on_the_forced_conclusion_ agentTools: Peek()); Assert.Null(attempt.Error); - Assert.Equal(["done"], loop.Calls[^1].ToolNames); + Assert.Equal(["complete_step"], loop.Calls[^1].ToolNames); Assert.Equal("auto", loop.Calls[^1].ToolChoice); } diff --git a/tests/E2E.Tests/TraceCacheReplay/TraceCacheFlows.cs b/tests/E2E.Tests/TraceCacheReplay/TraceCacheFlows.cs index 5ad490a77..df7894a26 100644 --- a/tests/E2E.Tests/TraceCacheReplay/TraceCacheFlows.cs +++ b/tests/E2E.Tests/TraceCacheReplay/TraceCacheFlows.cs @@ -50,7 +50,7 @@ public sealed record FlowOutcome(string Title, Exception? Error, IReadOnlyList ("tap", new { @ref = reference }); - private static (string, object) Type(string reference, string value) => ("fill", new { @ref = reference, value }); + private static (string, object) Type(string reference, string value) => ("type", new { @ref = reference, value }); /// The reference of the last listed node whose line matches . private static string NodeIdFor(string prompt, string pattern) => NodeIdsFor(prompt, pattern).LastOrDefault() ?? throw new InvalidOperationException("No line " + pattern + " in the screen:\n" + prompt); diff --git a/tests/E2E.Tests/TraceReplay/ReplayThroughTheActTests.cs b/tests/E2E.Tests/TraceReplay/ReplayThroughTheActTests.cs index 6341b6ffd..f6d2f87e0 100644 --- a/tests/E2E.Tests/TraceReplay/ReplayThroughTheActTests.cs +++ b/tests/E2E.Tests/TraceReplay/ReplayThroughTheActTests.cs @@ -31,7 +31,7 @@ public async Task Partial_Settles_each_look_as_far_as_the_previous_actions_polic tap, ModelResponses.Fill("textbox", "Email", "a@b.c"), tap, - ModelResponses.Call("fill_secret", new { role = "textbox", name = "Password", secret = "password" }), + ModelResponses.Call("type_secret", new { role = "textbox", name = "Password", secret = "password" }), tap, ModelResponses.Call("select", new { role = "combobox", name = "Plan", value = "Pro" }), tap); diff --git a/tests/E2E.Tests/TruncatedSecrets/SecretsCutShortByObservationLimitsTests.cs b/tests/E2E.Tests/TruncatedSecrets/SecretsCutShortByObservationLimitsTests.cs index e0ae86229..9fd50d93e 100644 --- a/tests/E2E.Tests/TruncatedSecrets/SecretsCutShortByObservationLimitsTests.cs +++ b/tests/E2E.Tests/TruncatedSecrets/SecretsCutShortByObservationLimitsTests.cs @@ -110,7 +110,7 @@ public async Task Keeps_the_cut_fragment_out_of_the_failure_screen_the_report_an return last.Role == "tool" ? ModelResponses.Done("passed", "filled") - : ModelResponses.Call("fill_secret", new { role = "textbox", name = "Source", secret = "member" }); + : ModelResponses.Call("type_secret", new { role = "textbox", name = "Source", secret = "member" }); }); await using var session = await E2ESession.StartAsync(new E2ESessionOptions { From e0244c1286e036d18dee5fa16e2d1e8e1e2b5b8f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 20:22:18 +0000 Subject: [PATCH 02/13] List type_secret once for the ACP session and gate it per step; restore MCP clear on its own kind (#1630) The ACP session lists the widest toolset from its first step, as upstream's listing does, so type_secret stays listed and a step without secrets is told "this step declares no secrets". The MCP clear tool is offered on the clear kind alone again. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1xnBhxbpAgMwCuPBUA --- src/E2E.Acp/StepTools.cs | 9 ++++++--- src/E2E.Cli/Mcp/SessionCatalog.cs | 2 +- src/E2E/Agent/AgentTools.cs | 8 +++++++- tests/E2E.Acp.Tests/Executor/AcpTests.cs | 9 ++++----- 4 files changed, 18 insertions(+), 10 deletions(-) diff --git a/src/E2E.Acp/StepTools.cs b/src/E2E.Acp/StepTools.cs index b7b14a3cb..7ebb698af 100644 --- a/src/E2E.Acp/StepTools.cs +++ b/src/E2E.Acp/StepTools.cs @@ -16,6 +16,9 @@ namespace E2E.Acp; /// internal static class StepTools { + // Upstream's TYPE_SECRET: the session lists the tool once, for every step, so the description sends the agent to the step message. + private const string TypeSecret = "Fill one declared secret into an input by its name; the plaintext never passes through you. A password fills only a password field; a generic-secret fills any editable input. The step message lists the secrets it declares."; + private const string StepEnded = "The step has ended. Wait for the next instruction."; /// The session's tools, listed from its first step. Upstream's stepTools. @@ -24,13 +27,13 @@ public static IReadOnlyList Create(StepSlot slot, StepExecutorContex var tools = new List(); // The tools an assertion offers change nothing in the app (upstream's `name in readOnlyTools`); the listing is of an act's, the widest. var readOnlyTools = AgentTools.CreateGrammarTools(first, new GrammarToolsOptions { Kind = ExecutorStepKind.Assert }).Select(tool => tool.Definition.Name).ToHashSet(StringComparer.Ordinal); - foreach (var listed in AgentTools.CreateGrammarTools(first, new GrammarToolsOptions { Kind = ExecutorStepKind.Act })) + foreach (var listed in AgentTools.CreateGrammarTools(first, new GrammarToolsOptions { Kind = ExecutorStepKind.Act, OfferTypeSecret = true })) { var name = listed.Definition.Name; var readOnly = readOnlyTools.Contains(name); tools.Add(new ServedTool( name, - listed.Definition.Description, + name == "type_secret" ? TypeSecret : listed.Definition.Description, listed.Definition.Parameters, readOnly, (arguments, token) => GuardedAsync(slot, name, async active => @@ -136,7 +139,7 @@ private static ActiveStep Current(StepSlot slot, string name) return active; } - private static string Unavailable(string name) => $"{name} is not available in this step"; + private static string Unavailable(string name) => name == "type_secret" ? "this step declares no secrets" : $"{name} is not available in this step"; private static ModelToolCall CallOf(string name, JsonElement arguments) => new() { diff --git a/src/E2E.Cli/Mcp/SessionCatalog.cs b/src/E2E.Cli/Mcp/SessionCatalog.cs index 45e30d948..a58da6a3e 100644 --- a/src/E2E.Cli/Mcp/SessionCatalog.cs +++ b/src/E2E.Cli/Mcp/SessionCatalog.cs @@ -162,7 +162,7 @@ public static SessionCatalog Create(ISessionContext context, EngineCapabilities Add("uncheck", Verb(context, screen, "uncheck", "check", "Uncheck a checkbox.", Schema(Target), extra: new JsonObject { ["checked"] = false })); } - if (verbs.Contains(ExecutorVerb.Clear) && verbs.Contains(ExecutorVerb.Type)) + if (verbs.Contains(ExecutorVerb.Clear)) { Add("clear", Verb(context, screen, "clear", "type", "Clear a field.", Schema(Target), extra: new JsonObject { ["value"] = "" })); } diff --git a/src/E2E/Agent/AgentTools.cs b/src/E2E/Agent/AgentTools.cs index 8fd4c835a..5b5be9dca 100644 --- a/src/E2E/Agent/AgentTools.cs +++ b/src/E2E/Agent/AgentTools.cs @@ -442,7 +442,7 @@ internal static IReadOnlyList GrammarTools(StepExecutorContext con var act = new ActTools(host.Scope, host.Feed, DefaultAgent.PresenterFor(context)); return GrammarToolDefinitions(host.Scope.EngineCapabilities, offered, host.Scope.ActionKinds, host.Scope.Session.PointerActionKinds, host.Scope.Session.CanDismissKeyboard) // Offered only when the step declared secrets (upstream: an empty vocabulary is better than a tool the model can only be rejected on). - .Where(definition => definition.Name != "type_secret" || context.Step.Secrets.Count > 0) + .Where(definition => definition.Name != "type_secret" || context.Step.Secrets.Count > 0 || options?.OfferTypeSecret == true) .Where(definition => !asserting || AssertionOffers(definition.Name, project, context.Step.Vision)) .Select(definition => new ToolLoopTool( definition, @@ -811,6 +811,12 @@ public sealed class GrammarToolsOptions /// Object.create, which the port has no counterpart of). .NET-only. ///
internal ExecutorStepKind? Kind { get; init; } + + /// + /// Offer type_secret although the step declares no secrets: the session of an ACP agent lists the widest toolset once (upstream's + /// listing builds a context with a secret by Object.create, which the port has no counterpart of). .NET-only. + /// + internal bool OfferTypeSecret { get; init; } } /// diff --git a/tests/E2E.Acp.Tests/Executor/AcpTests.cs b/tests/E2E.Acp.Tests/Executor/AcpTests.cs index b604b62f8..a48b88c30 100644 --- a/tests/E2E.Acp.Tests/Executor/AcpTests.cs +++ b/tests/E2E.Acp.Tests/Executor/AcpTests.cs @@ -593,10 +593,9 @@ public async Task Reports_a_model_the_agent_does_not_offer() } [Fact] - public async Task Partial_Lists_the_tools_once_and_tells_the_agent_which_ones_a_step_does_not_offer() + public async Task Lists_the_tools_once_and_tells_the_agent_which_ones_a_step_does_not_offer() { - // Partial: the document engine offers every step the same tools, so no step lacks a tool the session listed. - // type_secret is listed, and a call of it for a secret the step does not declare says so. + // type_secret is listed for every step, and a step that declares no secrets is told so. var agent = Scripted(_directory, Turns([Call("type_secret", new { role = "textbox", name = "Title", secret = "admin.password" }), Pass], [Pass])); await using var session = await StartAsync(agent.Executor, new TodoPage().World()); @@ -605,8 +604,8 @@ public async Task Partial_Lists_the_tools_once_and_tells_the_agent_which_ones_a_ var offered = Assert.Single(agent.Sessions())["session"]!["tools"]!.AsArray().Select(tool => tool!.GetValue()).ToList(); Assert.Contains("type_secret", offered); - Assert.DoesNotContain("Not offered in this step", agent.Prompts()[0], StringComparison.Ordinal); - Assert.Contains("isError", agent.ResultOf("type_secret"), StringComparison.Ordinal); + Assert.Contains("Not offered in this step: type_secret (this step declares no secrets).", agent.Prompts()[0], StringComparison.Ordinal); + Assert.Contains("this step declares no secrets", agent.ResultOf("type_secret"), StringComparison.Ordinal); } [Fact] From a07026187ccc69d2cb20b85670eec19f45dca1ff Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 20:44:50 +0000 Subject: [PATCH 03/13] Address review: complete_step in the remaining prompts, session-only clear, COMPATIBILITY (#1630) The extract prompts, the action-budget refusals, and the ACP session rules name complete_step. The MCP clear tool runs the session-only clear of ActTools on the clear kind, so an engine with no fill still clears. COMPATIBILITY.md and the guides state the new tool names and the open differences. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1xnBhxbpAgMwCuPBUA --- COMPATIBILITY.md | 34 +++++++++---------- skills/e2e-dotnet/references/agent.md | 4 +-- skills/e2e-dotnet/references/explore.md | 2 +- src/E2E.Acp/AcpExecutor.cs | 4 +-- src/E2E.Cli/Mcp/SessionCatalog.cs | 2 +- src/E2E/Agent/ActTools.cs | 10 ++++-- src/E2E/Agent/Agent.cs | 8 ++--- .../AgentAct/LoopGuardsAndTranscriptsTests.cs | 2 +- 8 files changed, 35 insertions(+), 31 deletions(-) diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index d9d9115c5..923dbea3c 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -56,8 +56,8 @@ Names are C# versions of the JavaScript API: `agent.act` is `ActAsync`, `screen. - The `browser` fixture subset: `goto`, `reload`, `back`, `forward`, `url`, `title`, `waitForURL`, `evaluate`, `waitForResponse`, `route`, `unroute`, `unrouteAll`, `cookies`, `setCookies`, `clearCookies`, `setViewport`, `emulateMedia`, `addInitScript`, `keyboard.press`, `keyboard.type`, `keyboard.down`, `keyboard.up`, and `mouse.move`, `wheel`, `down`, `up` - `press` keys are checked in core, as upstream's `requireKey` (`internal/keys.ts`, `Keys.RequireKey`) does: `Locator.PressAsync` checks the key before the node is resolved (in `LocatorEngine.CheckAction`, upstream's `checkAction`), and the agent's `press` and `pressKey` check it in the dispatcher before the action is committed. A key outside the grammar (`KeyGrammar.Parse`, upstream's `parseKey`: zero or more modifiers and one key joined by `+`) is `INVALID_ARGUMENT` on every engine, where the mobile and Windows engines alone refused it with `UNSUPPORTED_CAPABILITY` (those checks stay, as upstream's engines keep theirs). `Locator.PressAsync` no longer throws an `ArgumentException` for an empty or blank key: `""` is `INVALID_ARGUMENT` like any key outside the grammar, and `" "` is a space character key. `Browser.Keyboard.PressAsync` takes any Playwright key (`Alt+KeyA`), as upstream's `browser.keyboard` does, and is not checked. The agent's `press` still refuses an empty or blank key first with `press requires a key` (a .NET-only pre-check). - Agent budgets: `ActOptions.MaxSteps`, `ActOptions.MaxModelCalls`, and `WaitForOptions.MaxModelCalls`. A per-call budget can only lower the agent's configured one. A higher or non-positive value throws `INVALID_ARGUMENT` -- An `act` past its action budget ends `STEP_BUDGET_EXHAUSTED`, blocked. The model is told and may still conclude: a passing verdict, or a failure without a code, becomes the budget error. Replayed actions draw on the same budget. `navigate`, `back`, `scroll`, and `scroll_to` take a slot; `observe` does not. An `act` whose tool loop uses every model turn without a `done` call ends `STEP_NO_CONCLUSION`, a failed verdict, as upstream's does (see the tool loop bullet); an executor that records one model call more than the budget allows still ends `STEP_BUDGET_EXHAUSTED`, blocked (`agent.act exhausted its model-call budget of N`) -- As upstream, an `act` guards against a model that repeats itself. The same call with the same input 3 times in a row, or a sequence of 2 to 4 calls twice, adds a warning. 5 identical calls, or a sequence 3 times, offers the model only `done`, so it still writes its own verdict. The `done` call never counts. The failure streak is upstream's `checkFailureStreak` (`LoopGuards.CheckFailureStreak`, tester-army/e2e `loop-guards.ts` at `f4be1c4`): 3 failed calls in a row add a notice, 5 offer the model only `done`, and a call that worked starts the count over. The loop reads each tool result's text as upstream does: `LoopGuards.ExtractToolResults` (upstream's `extractToolResults`) over the tool messages, and `LoopGuards.IsFailedResult` (upstream's `isFailedResult`) for the failure shape. A call the SDK refused or a tool that threw is an `error-text` result (`ModelMessage.ErrorText`, `ToolOutcome.ErrorText`), so it counts as failed whatever its text says. Differences: a project tool's structured result has no port counterpart (the port's tool results are text, so the `it` that reads a structured result skips it), and the grammar's failure leads are the port's own text: `ActionNames.GrammarToolNames` also holds the port's `fill`, `fill_secret`, `uncheck`, and `clear`, which upstream does not have, so their failures count too. The partial read of a structured result is `Partial_Reads_text_results_...` in `LoopGuards/FailureStreakTests`. The thresholds are upstream's `LoopGuardThresholds` and `DEFAULT_LOOP_GUARD_THRESHOLDS` (`LoopGuards.DefaultLoopGuardThresholds`), and `CheckLoopGuards` and `CheckFailureStreak` take an optional `thresholds` argument; the loop passes none, and the type is internal, so no public option exposes it. A turn with no tool call counts as a failure. An executor's transcript is kept only under `E2E_DEBUG=1`, as upstream keeps it under `--debug`. +- An `act` past its action budget ends `STEP_BUDGET_EXHAUSTED`, blocked. The model is told and may still conclude: a passing verdict, or a failure without a code, becomes the budget error. Replayed actions draw on the same budget. `navigate`, `back`, `scroll`, and `scroll_to` take a slot; `observe` does not. An `act` whose tool loop uses every model turn without a `complete_step` call ends `STEP_NO_CONCLUSION`, a failed verdict, as upstream's does (see the tool loop bullet); an executor that records one model call more than the budget allows still ends `STEP_BUDGET_EXHAUSTED`, blocked (`agent.act exhausted its model-call budget of N`) +- As upstream, an `act` guards against a model that repeats itself. The same call with the same input 3 times in a row, or a sequence of 2 to 4 calls twice, adds a warning. 5 identical calls, or a sequence 3 times, offers the model only `complete_step`, so it still writes its own verdict. The `complete_step` call never counts. The failure streak is upstream's `checkFailureStreak` (`LoopGuards.CheckFailureStreak`, tester-army/e2e `loop-guards.ts` at `f4be1c4`): 3 failed calls in a row add a notice, 5 offer the model only `complete_step`, and a call that worked starts the count over. The loop reads each tool result's text as upstream does: `LoopGuards.ExtractToolResults` (upstream's `extractToolResults`) over the tool messages, and `LoopGuards.IsFailedResult` (upstream's `isFailedResult`) for the failure shape. A call the SDK refused or a tool that threw is an `error-text` result (`ModelMessage.ErrorText`, `ToolOutcome.ErrorText`), so it counts as failed whatever its text says. Differences: a project tool's structured result has no port counterpart (the port's tool results are text, so the `it` that reads a structured result skips it), and the grammar's failure leads are the port's own text: `ActionNames.GrammarToolNames` also holds the port's `fill`, `fill_secret`, `uncheck`, and `clear`, which upstream does not have, so their failures count too. The partial read of a structured result is `Partial_Reads_text_results_...` in `LoopGuards/FailureStreakTests`. The thresholds are upstream's `LoopGuardThresholds` and `DEFAULT_LOOP_GUARD_THRESHOLDS` (`LoopGuards.DefaultLoopGuardThresholds`), and `CheckLoopGuards` and `CheckFailureStreak` take an optional `thresholds` argument; the loop passes none, and the type is internal, so no public option exposes it. A turn with no tool call counts as a failure. An executor's transcript is kept only under `E2E_DEBUG=1`, as upstream keeps it under `--debug`. - `ActResult.ModelCalls` (0 for a full replay) and `ActResult.Actions` (replayed and live actions, counting failed attempts) - `AgentException.Blocked` and `AgentException.Explanation` (the same text as `Message`) - OpenAI-compatible tool calling @@ -191,8 +191,8 @@ The tests of upstream commits `adcfc2ba`, `c9445ce6`, `a13e119c`, `1731e0d8`, `f Ported in part (the test name carries `Partial_` or a `PARTIAL` suffix): -- Agent verbs follow the engine's declared action kinds (#1278, tester-army/e2e `engine/session.ts`, `declaredVerbs` and `KINDS_BY_VERB`): `ActContext.VerbsOf` adds `tap`, `double_tap`, `type` and `type_secret` (`fill`), `press`, `select` (`selectOption`), and `check` (both `check` and `uncheck`) only for the kinds the engine declares in `IEngineSession.ActionKinds` (`ActContext.KindsByVerb`), and `AgentTools.GrammarToolDefinitions` and `SessionCatalog.Create` offer a tool only when `AgentTools.OffersVerb` finds a verb that unlocks it, as `verbs.has(...)` does at each tool's site in `createGrammarTools` and `createPointTools` (`agent/primitives.ts`). `type` and `press` are also offered with the keyboard alone, with no target field when the engine declares no `fill` or `press`. `ActTools` refuses a call to a tool the engine did not unlock as `Unknown tool`, and `ActionDispatcher.TakesNodeVerb` reads `tap` from the kinds, so `tap_at` on an engine that declares no bare tap and no `tap` kind is not offered. `type_at`, `press_at`, and `select_at` were already offered; they now follow `fill` or the keyboard, `press` or the keyboard, and `selectOption`. Differences: the port's `clear` verb (upstream has none) follows the `clear` kind, and `uncheck` follows `check`. An engine that declares no kind gets none of these tools (a null kind list is none, as upstream's `engine?.actions ?? []`). Not ported: the scroll, `scroll_to`, `upload`, and `navigate` verbs still follow `EngineCapabilities` (the tools are shaped by the verbs they give) where upstream derives them from `swipe`, `scrollIntoView`, `setInputFiles`, and the `open` hook. Tests: `Offers_each_node_verb_only_for_the_action_kinds_the_engine_declares` (.NET-only, `AgentGrammarTools/`), `Catalogs_a_node_verb_only_for_the_action_kinds_the_engine_declares` (.NET-only, MCP catalog), and the kind lines in `Engine/CreateEngineSessionTests` (`Partial_Derives_the_keyboard_and_pointer_verbs_from_the_declared_hooks`: the lines for `tap`, `doubleTap`, `fill`, `press`, `selectOption`, `check`, `uncheck`, and `clear` of upstream's `derives the grammar verbs from the declared action kinds and hooks`) -- The ported `describe('the grammar tools have closed schemas')` and `describe('the schema a provider receives')` run in the port's names (`fill` for `type`, `done` for `complete_step`, `code` for `errorCode`) and validate against the JSON Schema a provider receives, with `JsonSchemaCheck`, where upstream parses with zod. The rejection of a blocked verdict without a blocking code reads `failed: a blocked verdict requires a code ...` where upstream's reads `Rejected: a blocked verdict requires errorCode ...`. The port's tool schemas carry no `$schema` keyword (zod's JSON Schema output does), so the plain-object check lists `additionalProperties`, `properties`, `required`, and `type` +- Agent verbs follow the engine's declared action kinds (#1278, tester-army/e2e `engine/session.ts`, `declaredVerbs` and `KINDS_BY_VERB`): `ActContext.VerbsOf` adds `tap`, `double_tap`, `type` and `type_secret` (`fill`), `press`, `select` (`selectOption`), and `check` (both `check` and `uncheck`) only for the kinds the engine declares in `IEngineSession.ActionKinds` (`ActContext.KindsByVerb`), and `AgentTools.GrammarToolDefinitions` and `SessionCatalog.Create` offer a tool only when `AgentTools.OffersVerb` finds a verb that unlocks it, as `verbs.has(...)` does at each tool's site in `createGrammarTools` and `createPointTools` (`agent/primitives.ts`). `type` and `press` are also offered with the keyboard alone, with no target field when the engine declares no `fill` or `press`. `ActTools` refuses a call to a tool the engine did not unlock as `Unknown tool`, and `ActionDispatcher.TakesNodeVerb` reads `tap` from the kinds, so `tap_at` on an engine that declares no bare tap and no `tap` kind is not offered. `type_at`, `press_at`, and `select_at` were already offered; they now follow `fill` or the keyboard, `press` or the keyboard, and `selectOption`. Differences: the agent has no `clear` or `uncheck` tool, as upstream; an MCP session still serves them (`clear` on the `clear` kind, run as `ActTools`' session-only `clear`; `uncheck` as `check` with `checked: false`). An engine that declares no kind gets none of these tools (a null kind list is none, as upstream's `engine?.actions ?? []`). Not ported: the scroll, `scroll_to`, `upload`, and `navigate` verbs still follow `EngineCapabilities` (the tools are shaped by the verbs they give) where upstream derives them from `swipe`, `scrollIntoView`, `setInputFiles`, and the `open` hook. Tests: `Offers_each_node_verb_only_for_the_action_kinds_the_engine_declares` (.NET-only, `AgentGrammarTools/`), `Catalogs_a_node_verb_only_for_the_action_kinds_the_engine_declares` (.NET-only, MCP catalog), and the kind lines in `Engine/CreateEngineSessionTests` (`Partial_Derives_the_keyboard_and_pointer_verbs_from_the_declared_hooks`: the lines for `tap`, `doubleTap`, `fill`, `press`, `selectOption`, `check`, `uncheck`, and `clear` of upstream's `derives the grammar verbs from the declared action kinds and hooks`) +- The ported `describe('the grammar tools have closed schemas')` and `describe('the schema a provider receives')` run in the port's names (`code` for `errorCode`, #1643) and validate against the JSON Schema a provider receives, with `JsonSchemaCheck`, where upstream parses with zod. The rejection of a blocked verdict without a blocking code reads `failed: a blocked verdict requires a code ...` where upstream's reads `Rejected: a blocked verdict requires errorCode ...`. The port's tool schemas carry no `$schema` keyword (zod's JSON Schema output does), so the plain-object check lists `additionalProperties`, `properties`, `required`, and `type` - `replays the typing, hands off at the renamed button, and tells the model what already ran` (`Partial_Hands_off_at_the_renamed_button_after_the_typing_and_tells_the_model_what_already_ran`): the port does not check that the entry the lenient run leaves keeps the renamed button's old name, because it rewrites the entry after a hand-off - `refuses $sink before reading the plaintext`: the port has no plaintext resolver to count reads and records no policy decisions - `never sends a secret value to the model`: runs on a scripted model, not a real one @@ -503,15 +503,15 @@ Not ported: upstream's `describe('explicit fixture operations')` in `packages/e2 `AcpExecutor` (the `E2E.Acp` package, namespace `E2E.Acp`) ports `@e2e-dev/acp` (tester-army/e2e#960, `0410fc9`): an `IStepExecutor` that runs `agent.act` and `agent.assert` steps on a coding agent that speaks the Agent Client Protocol on stdio. It starts the agent, opens one session per attempt, and serves the built-in action tools and the verdict tool to it as an MCP server on `127.0.0.1`. Every action goes through `ctx.Actions`, so the runner authorizes, counts, and records it, and cached steps replay as the built-in agent's do. -- Upstream's `acp({ command, args, ... })` is `AcpExecutor.Create(command, args, options)`; `claudeCode(options)` and `codex(options)` are `AcpExecutor.ClaudeCode(options)` and `AcpExecutor.Codex(options)` (`presets.ts` is the internal `Presets`). `AcpAgentOptions` (`Env`, `Model`, `System`, `Name`) and `AcpOptions` (adds `Mode` and `SessionMeta`) are records. The step is `done` with a `code`, as the port's verdict tool, not upstream's `complete_step` with `errorCode`. The tools are the port's `AgentTools.CreateGrammarTools`, so the agent gets `fill` and `fill_secret`, not `type` and `type_secret`. -- Upstream uses the ACP and MCP TypeScript SDKs. There is no .NET ACP SDK, so `E2E.Acp` has an internal JSON-RPC peer over newline-delimited JSON (`JsonRpcConnection`), an internal ACP session (`AcpSession`, `AgentLaunch`), and an internal MCP server over `HttpListener` (`McpToolServer`, stateless Streamable HTTP, a random path segment, a request with an `Origin` header is refused). `StepTools` builds the served tools from upstream's `tools.ts`; its per-step gate runs batched action calls one at a time, so a call queued behind `done` is skipped. These are .NET-only. +- Upstream's `acp({ command, args, ... })` is `AcpExecutor.Create(command, args, options)`; `claudeCode(options)` and `codex(options)` are `AcpExecutor.ClaudeCode(options)` and `AcpExecutor.Codex(options)` (`presets.ts` is the internal `Presets`). `AcpAgentOptions` (`Env`, `Model`, `System`, `Name`) and `AcpOptions` (adds `Mode` and `SessionMeta`) are records. The step is `complete_step` with a `code`, where upstream's takes `errorCode`. The tools are the port's `AgentTools.CreateGrammarTools`, so the agent gets `fill` and `fill_secret`, not `type` and `type_secret`. +- Upstream uses the ACP and MCP TypeScript SDKs. There is no .NET ACP SDK, so `E2E.Acp` has an internal JSON-RPC peer over newline-delimited JSON (`JsonRpcConnection`), an internal ACP session (`AcpSession`, `AgentLaunch`), and an internal MCP server over `HttpListener` (`McpToolServer`, stateless Streamable HTTP, a random path segment, a request with an `Origin` header is refused). `StepTools` builds the served tools from upstream's `tools.ts`; its per-step gate runs batched action calls one at a time, so a call queued behind `complete_step` is skipped. These are .NET-only. - Permission requests fail closed: an unknown request is answered `cancelled`. A tool of the agent's own (read from the adapter's marks and the tool kind, never from the agent's text) ends the step with `POLICY_DENIED`. - `InternalsVisibleTo E2E.Acp` on `E2E` reaches `ctx.Host.Scope.Redactor` and `ReplayHandOffReasons`, which upstream's package imports from `e2e`. .NET-only. - Node is required for the presets. Like upstream, they run the adapter from the install they find in `node_modules`, walking up from the working directory (the `bin` of its `package.json`, with a link resolved at the package folder), never `npx `. Upstream starts it with the running Node (`process.execPath`); the port runs `node` from the `PATH`, since a .NET process has no Node of its own. The adapters are npm packages, so a .NET project needs a `package.json` with `npm install -D @agentclientprotocol/claude-agent-acp` or `@agentclientprotocol/codex-acp`. The Codex preset lists your MCP servers with `codex mcp list --json` (30 s limit, process tree killed on a timeout or cancel) from the `@openai/codex` the adapter depends on, or from `CODEX_PATH`. A pnpm-style link in a folder above the package, not at the package, is not resolved. - .NET-only in `Presets.cs`: `CodexListTimeout` (upstream's inline `30_000`), `ResolveCodexBin` (upstream's inline `createRequire(...).resolve`), `RunAsync` (upstream's `execFileAsync` with `timeout`; it also kills the process tree on a timeout or cancel, as the repo rules require), and the local `StartAsync` (upstream's `.then` chain). The Codex list is shared by every caller, and each caller waits on it with its own cancellation token (upstream has no token). The error for a failed list adds the process's stderr. - The agent's environment has no variable named `E2E_SECRET_*` or `E2E_USER_*`, in any casing, from this process or from `Env`; the `codex mcp list` call gets the same environment. Secrets in any other variable reach the agent, as upstream warns. - On Windows, a command for `AcpExecutor.Create` must be an executable, not a `.cmd` shim: nothing goes through a shell. -- The tool names the agent sees differ from upstream: `fill` and `fill_secret` for `type` and `type_secret`, `done` for `complete_step`, with the port's `code` for `errorCode`. +- The agent's tool names are upstream's (`type`, `type_secret`, `check` with `checked`, `complete_step`; #1630). Differences: the verdict tool takes `code` where upstream's takes `errorCode` (#1643), `type_secret` takes `secret` where upstream's takes `name` and does not list the available secrets (#1642), and tools address a node by `ref` or `role` and `name` where upstream's take a `target` id (#1644). The ACP session lists `type_secret` for every step, as upstream's `listing` does, through the .NET-only `GrammarToolsOptions.OfferTypeSecret` (the port has no `Object.create` of a context); a step that declares no secrets is told so. - Upstream moved the ACP guide into `subscriptions.mdx` (#1488), a page the port has no counterpart for. The guide is `docs-site/src/content/docs/features/acp.mdx`. - `tests/E2E.Acp.Tests` ports `packages/acp/tests/executor.test.ts` into `Executor/`, all 31 of its `it`s (the secret-stripping `it` included), against `tests/E2E.Acp.Tests.Agent`, a scripted fake agent (a console app, so no Node). Four tests are `Partial_`: the serial-group test (the port's attempt memory is per attempt, not shared by serial members), the read-only assertion test (the document engine has no `hover`, so it uses `scroll`), the tool-list-once test (no step lacks a tool), and the attempt-ends-while-starting test. The runtime-error test is `Partial_` too: the port's action budget answers the agent in text, so the error comes from an `OnAction` hook, and the runner does not count the cancelled turn as a model call. - `tests/E2E.Acp.Tests/Presets/AgentPresetsTests.cs` ports the three `it`s of `presets.test.ts`. The adapter in its tests is a Node script that starts the scripted agent, so the tests need Node on the `PATH`, as the presets do. The tests change the working directory and the environment, so they run in one serial collection (`WorkingDirectoryCollection`, .NET-only). @@ -621,7 +621,7 @@ The catalog is what the testing agent's tools do, so a coding agent acts as the Differences: - Node ids and diff results (tester-army/e2e#231, `b9f032b`, added `ScreenPresenter`; the session has shared it since #181, `afde198`): `observe` lists `#id role "name" ...` lines under `Current screen (revision b3, path /, 12 nodes):`, and an action answers with its lead, the test line, and what changed since the screen the agent last received (`Screen changes since revision b3 (now revision b4, 12 nodes): 1 added, 1 changed.`, then `added`, `changed … (was: …)`, `removed` lines), as upstream's `ScreenPresenter` (`agent/screen-update.ts`) does. `ScreenPresenter` is `Agent/ScreenUpdate.cs`: the first screen whole, a later one as the difference keyed by node id, focus left out of the comparison, the whole screen again past 60 changed lines or half of it, no removals against a truncated listing, and the note when an action closed the on-screen keyboard. An unchanged screen reads `Screen unchanged since revision …`, or `No listed node changed within the wait after this action …` after an action that should have changed it (`expectChange` is false for `scroll`, `scroll_to`, `type_secret`, and a `press` that only moves focus, as upstream). A failed action reports the changes after its `tap n9 failed: LOCATOR_NOT_FOUND: node n9 is not on the current screen; re-observe for the newest ids` lead. The built-in agent uses it too (#1022): `DefaultAgent.PresenterFor` makes one presenter per step, shared by the opening prompt (`Open`) and the tools (`Present`), with the pixel half (`Attach`, `PixelEvidence`, the screenshot the model holds). Transcript compaction and the overflow shrink find a whole screen by `Current screen (revision `, as upstream's `FULL_SCREEN_PATTERN`. Differences: the session has no screenshots on `act` of its own, so its presenter has no pixel half (the "No screenshot can be taken" text, the unchanged-screen cases that compare images); the lines are `FailureEvidence.Listing`'s (`SnapshotText.FormatNode`: states each in its own bracket, a secure field as `secure`), and the listing is cut at the default observation limit (262144 bytes), where upstream cuts at the agent's `maxObservationBytes`. A custom executor's `ExecutorObservation.Text` is the same lines (see Custom executors (#1022) below). Node ids are the engine's refs (`SemanticNode.Ref`), unchanged. Custom executors (#1022): `ExecutorObservation.Text`, which an executor reads from `ObserveAsync` and `Start`, is the `#id role "name" ...` listing, one node per line, cut at the observation byte budget (see [Agents](#agents)), as upstream's `text`; `ExecutorTarget` takes the id with or without the `#`. A custom executor that parsed the earlier `Screen (route):` text with `[ref=…]` lines breaks and reads the id after `#` now. The listing has no `Screen (` heading: the route is `ExecutorObservation.Path`. The presenter (the `Current screen` and diff results) stays the built-in agent's and the session's, as upstream's `screen-update.ts` does; a custom executor gets no diff results. .NET-only: the judgment path (`assert`, `waitFor`, `extract`) prints a `Screen (route):` line before the same listing, where upstream wraps its text in ``; `OverflowShrink.FullScreenPattern` still matches that line. -- The grammar is the port's: `type` and `type_secret` take `text` and `secret`; `uncheck` and `clear` are tools of their own. Upstream's point tools (`tap_at`, `hover_at`, `drag_at`, `type_at`, `press_at`, `select_at`) are served as upstream's grammar offers them, from `AgentTools.CreateGrammarTools`: the session's `screenshot` notes the shot and the viewport on the session's `ScreenPresenter` (`Attach`), which the point tools read their points against (before a screenshot they answer with the line that says to take one). On an engine with a keyboard `type` takes an optional target and `replace`, and a `type` with no target has no test line (`the value went to the focused field`). `long_press`, `right_click`, `hover`, and `drag` are, and a coding agent over MCP gets each one only where the engine declares its action kind (`longPress`, `secondaryTap`, `hover`, `dragTo`), as `act` does. `dismiss_keyboard` is a tool only where the engine can hide its keyboard (`IEngineSession.CanDismissKeyboard`: the mobile engine). +- The grammar is the port's: `type` takes `text`, and `type_secret` takes `secret`; `uncheck` and `clear` are tools of the MCP catalog only (the agent has none, as upstream). Upstream's point tools (`tap_at`, `hover_at`, `drag_at`, `type_at`, `press_at`, `select_at`) are served as upstream's grammar offers them, from `AgentTools.CreateGrammarTools`: the session's `screenshot` notes the shot and the viewport on the session's `ScreenPresenter` (`Attach`), which the point tools read their points against (before a screenshot they answer with the line that says to take one). On an engine with a keyboard `type` takes an optional target and `replace`, and a `type` with no target has no test line (`the value went to the focused field`). `long_press`, `right_click`, `hover`, and `drag` are, and a coding agent over MCP gets each one only where the engine declares its action kind (`longPress`, `secondaryTap`, `hover`, `dragTo`), as `act` does. `dismiss_keyboard` is a tool only where the engine can hide its keyboard (`IEngineSession.CanDismissKeyboard`: the mobile engine). - `locate` writes the `Screen` call a test uses (`Screen.GetByRole("button", "Save")`) and upstream's codes `LOCATOR_NOT_FOUND` and `LOCATOR_AMBIGUOUS`. - As upstream, a `locate` with a `status`, `alert`, or `listitem` name that matches nothing points at `Screen.GetByRole(role).Filter(text)` and `Screen.GetByText` (spelled as `GetByRole(role).filter({ hasText })` upstream), and an `exact: false` query is not told to loosen the match again (upstream #1147, `5e9597c`): the advice says "an" before a role that starts with a vowel (`an alert`, `a status`), and for an `exact: false` query it keeps the option in the text call, `Screen.GetByText("Saved.", exact: false)` (`getByText(q, { exact: false })` upstream). The web engine names those three roles from their content, so there the name also matches; the advice still holds, since the text matches either way. - Project tools (#1272): `agents..tools` declares command tools (`CommandTool`, `E2E/Agent/CommandTool.cs`, .NET-only: upstream's config holds `defineTool` calls, and a JSON file has no code). An entry has `description`, `mutates`, `command` (the program and its arguments, started with `ArgumentList`), and optionally `inputSchema` and `platforms`. The command runs in the project root with the input as JSON on stdin; stdout is the answer, stderr is printed line by line (on the server it reaches the operator's stderr redacted), a non-zero exit fails the call (`TOOL_FAILED`), and a command still running after two minutes is killed with its children. The session catalog serves each tool after the built-in ones (`SessionCatalog.Create` takes the agent's `Tools`, the target's platform, and a warning sink, as upstream's `createSessionCatalog` takes `tools`); a read-only one joins the read-only set, a tool whose platforms leave the target out is skipped, and a name a built-in tool holds is not served and logs `project tool "x" is not served over MCP: the name belongs to a built-in session tool`. The same tools reach `AgentConfig.CreateOptions().Tools`, so an act offers them. Still not ported: a config that runs code when it loads, so `mcp-user-output.test.ts` stays partial: `E2EMcpRedactsWhatUserCodePrintsTests` has `Partial_Withholds_what_a_config_printed_before_it_failed_to_load` (the output layer in process) and `Partial_Keeps_the_protocol_working_and_redacts_the_config_and_a_project_tool_on_stdout_and_stderr` (spawns the server; the tool prints on stderr, and the config's top-level print, the tool's stdout and the stray rejection are not checked). The in-process test `Redacts_what_a_config_and_a_tool_print_and_logs_an_uncaught_error_through_the_output_layer` is .NET-only and keeps those three checks on `McpOutput`. @@ -669,10 +669,10 @@ Not ported: - `providerOptions` is a dictionary of JSON objects by provider. It rides every `ModelRequest`, and `OpenAiCompatibleModel` adds the fields under its `Provider` key (`openai` by default) to the chat-completions body as given, so write the wire names (`reasoning_effort`, not `reasoningEffort`). It cannot replace `model`, `messages`, or `tools`. Every other client does the same with its own key (`anthropic`, `google`, `bedrock`, `azure`, `xai`, `openrouter`, `gateway`) and refuses its own core fields. As upstream, requests to OpenAI (and Azure over Responses) carry `store: false` and a prompt cache key per system prompt, and Anthropic requests carry cache breakpoints on the system prompt and the newest message; provider options win over both - `assert` and `extract` make one model call and one repair round. `waitFor` judges at once, then again only after `Interval` and on a changed screen, and every call, repairs included, counts against `MaxModelCalls`. It ends `STEP_TIMEOUT` (`waitFor timed out; last judgment: ...`) or `STEP_BUDGET_EXHAUSTED` (`waitFor exhausted its model-call budget; last judgment: ...`) - As upstream, the judge prompt says a value shown in more than one place holds only when every place agrees, unless the statement names the place or is about some item among several. The wording follows upstream, with "instruction" as "statement". The judge answers with upstream's `agent-judgment-2` object (see below) -- **Judgment grammar** (`Agent/AgentProtocol.cs`, internal; upstream's `agent/protocol.ts`). `assert` and `waitFor` judge through structured output, as upstream does: the model answers one `agent-judgment-2` object (`protocolVersion`, `explanation`, then `verdict`: `holds`, `fails`, or `inconclusive`) that `AgentProtocol.ValidateJudgmentResponse` checks. `holds` passes, `fails` is `ASSERTION_FAILED`, `inconclusive` is `ASSERTION_INCONCLUSIVE`. The judge has no `done` tool, no tool at all. An answer that is not JSON or does not match gets one repair round (a `` turn with the validation issue), then `MODEL_OUTPUT_INVALID` (`The judge did not answer with an agent-judgment-2 object: ...`). `AgentProtocol.JudgmentSchema` is upstream's `JUDGMENT_SCHEMA`, with `explanation` before `verdict`. `ModelRequest.Schema` and `SchemaName` are upstream's `ModelCall.schema` and `schemaName`, .NET-only in their place: the port's models take a `ModelRequest`, not a `ModelCall`. **Breaking for a custom `IAgentModel`**, which must honor `Schema` (return the JSON object as `ModelResponse.Content`; `Tools` is empty). The providers: chat completions send `response_format` `json_schema`, Responses `text.format`, Google `generationConfig.responseMimeType` and `responseJsonSchema`; Anthropic and Bedrock, which have no such field, offer one `json` tool and force it (the AI SDK's own fallback) unless the request thinks, when the tool is only offered. `OpenAiCompatibleModelOptions.JsonSchemaResponseFormat` (.NET-only, default on) leaves `response_format` out for a server that rejects it; the judge's prompt still asks for the object. Chat completions and Responses send `strict: false`, because strict mode refuses `maxLength`; the agent checks the answer itself. `ModelResponses.Judgment` and `ModelHttp.StructuredAnswer`/`JsonToolName` are .NET-only (a scripted judge answer; the forced-tool answer as content). Not ported: upstream's `extractSchema` and `validateExtractResponse` (the `agent-extract-2` envelope). `extract` still answers through its `extract` and `done` tools, so its inconclusive outlet is the `done` tool. -- **Closed grammar tool inputs** (`AgentTools.SchemaTool`, upstream's `schemaTool` in `agent/primitives.ts`). A grammar tool call that carries a field its schema does not declare is refused and never runs; the refusal is the call's result (`failed: Invalid input for tool tap: Type validation failed: Value: {...}.` then `Error message: [{"code":"unrecognized_keys","keys":[...],...}]`, the AI SDK's text over a zod issue), it counts toward the failure streak, and in a batched turn the other calls still run. The schema sent to the model carries `additionalProperties: false` for every grammar tool and for `done` (upstream's `schemaTool`), at the top level only (the `to` of `drag` is an object in the port, a node id in upstream, and stays open). Differences: only undeclared fields are refused by the runtime (the tools read their own values, so a wrong type is the tool's own failure); project tools (`AgentTool`) and the extract tools are not closed (upstream's `defineTool` closes nothing); `done` drops a code a passed verdict carries, as upstream's verdict tool does, instead of refusing it, and refuses an undeclared field like every grammar tool. Tests: `AgentPolicy/ToolCallsOutsideTheVocabularyTests`, `AgentGrammarTools/TheGrammarToolsHaveClosedSchemasTests`, and `AgentGrammarTools/TheSchemaAProviderReceivesTests`. +- **Judgment grammar** (`Agent/AgentProtocol.cs`, internal; upstream's `agent/protocol.ts`). `assert` and `waitFor` judge through structured output, as upstream does: the model answers one `agent-judgment-2` object (`protocolVersion`, `explanation`, then `verdict`: `holds`, `fails`, or `inconclusive`) that `AgentProtocol.ValidateJudgmentResponse` checks. `holds` passes, `fails` is `ASSERTION_FAILED`, `inconclusive` is `ASSERTION_INCONCLUSIVE`. The judge has no `complete_step` tool, no tool at all. An answer that is not JSON or does not match gets one repair round (a `` turn with the validation issue), then `MODEL_OUTPUT_INVALID` (`The judge did not answer with an agent-judgment-2 object: ...`). `AgentProtocol.JudgmentSchema` is upstream's `JUDGMENT_SCHEMA`, with `explanation` before `verdict`. `ModelRequest.Schema` and `SchemaName` are upstream's `ModelCall.schema` and `schemaName`, .NET-only in their place: the port's models take a `ModelRequest`, not a `ModelCall`. **Breaking for a custom `IAgentModel`**, which must honor `Schema` (return the JSON object as `ModelResponse.Content`; `Tools` is empty). The providers: chat completions send `response_format` `json_schema`, Responses `text.format`, Google `generationConfig.responseMimeType` and `responseJsonSchema`; Anthropic and Bedrock, which have no such field, offer one `json` tool and force it (the AI SDK's own fallback) unless the request thinks, when the tool is only offered. `OpenAiCompatibleModelOptions.JsonSchemaResponseFormat` (.NET-only, default on) leaves `response_format` out for a server that rejects it; the judge's prompt still asks for the object. Chat completions and Responses send `strict: false`, because strict mode refuses `maxLength`; the agent checks the answer itself. `ModelResponses.Judgment` and `ModelHttp.StructuredAnswer`/`JsonToolName` are .NET-only (a scripted judge answer; the forced-tool answer as content). Not ported: upstream's `extractSchema` and `validateExtractResponse` (the `agent-extract-2` envelope). `extract` still answers through its `extract` and `complete_step` tools, so its inconclusive outlet is the `complete_step` tool. +- **Closed grammar tool inputs** (`AgentTools.SchemaTool`, upstream's `schemaTool` in `agent/primitives.ts`). A grammar tool call that carries a field its schema does not declare is refused and never runs; the refusal is the call's result (`failed: Invalid input for tool tap: Type validation failed: Value: {...}.` then `Error message: [{"code":"unrecognized_keys","keys":[...],...}]`, the AI SDK's text over a zod issue), it counts toward the failure streak, and in a batched turn the other calls still run. The schema sent to the model carries `additionalProperties: false` for every grammar tool and for `complete_step` (upstream's `schemaTool`), at the top level only (the `to` of `drag` is an object in the port, a node id in upstream, and stays open). Differences: only undeclared fields are refused by the runtime (the tools read their own values, so a wrong type is the tool's own failure); project tools (`AgentTool`) and the extract tools are not closed (upstream's `defineTool` closes nothing); `complete_step` drops a code a passed verdict carries, as upstream's verdict tool does, instead of refusing it, and refuses an undeclared field like every grammar tool. Tests: `AgentPolicy/ToolCallsOutsideTheVocabularyTests`, `AgentGrammarTools/TheGrammarToolsHaveClosedSchemasTests`, and `AgentGrammarTools/TheSchemaAProviderReceivesTests`. - `ExtractAsync`: the type argument is the schema, in place of a Standard Schema. The judge gets the JSON schema of `T` (`JsonSchemaExporter`), and the answer must deserialize into `T` with required members and nullable annotations respected. A failure gets one repair round with the validation error, then `MODEL_OUTPUT_INVALID` (`extracted data failed schema validation: ...`). A judge that says the data is not shown ends `ASSERTION_INCONCLUSIVE` (`nothing to extract: ...`) -- `tools` (`AgentOptions.Tools`, `AgentTools` on the default agent) are project tools the model is offered beside the agent's own. As upstream's `defineTool`, an `AgentTool` declares `mutates` and `platforms`: a tool is offered only on the platforms it names, and a mutating tool takes an action slot. Its output is redacted and bounded to 400 lines and 16 KB (16384 UTF-8 bytes), whichever is reached first: the cut never splits a line and ends with a notice that says how much was left out (`ToolOutput.BoundToolOutput`, `ToolOutput.MaxToolOutputLines`, and `ToolOutput.MaxToolOutputBytes`, upstream's `boundToolOutput`, `MAX_TOOL_OUTPUT_LINES`, and `MAX_TOOL_OUTPUT_BYTES` in `agent/tool-output.ts`, tester-army/e2e at `f185ca8`). A first line longer than the whole byte budget keeps its head. The port redacts the result before it bounds it, so a cut cannot split a secret in two; upstream bounds first. Earlier versions of this port cut at 16384 characters with `[output cut]`. A name the agent already uses (`observe`, `done`, `navigate`, `back`, `scroll`, `scroll_to`, or a verb) is refused with `INVALID_CONFIG`. Differences: the tool is an `AgentTool` built with its schema as a `JsonObject` and a delegate, since the port has no `defineTool`, AI SDK tool, or `toModelOutput`; a tool that wants upstream's `getToolContext` takes the second constructor, whose delegate gets an `AgentToolContext` (`ObserveAsync`, `AttachScreenshotAsync`), and the first constructor keeps working; and upstream's reserved names for tools it adds elsewhere are reserved as upstream's `HARNESS_TOOL_NAMES` has them (`AgentTools.HarnessToolNames`): `complete_step` (every agent step), `locate`, `run_code`, `test_lines`, `start_recording`, `stop_recording` (an e2e mcp session), and `report_finding` (explore). A project tool of one of those names is `INVALID_CONFIG`. The port's verdict tool is `done`, which is reserved with the grammar's names, and `run_code` has no port. Upstream records a mutating tool as a gap in the trace, and so does the port: a recording holds an action of kind `tool` with the tool name, a replay runs the actions before it and then hands the step to the model, and the step is never replayed whole. Older versions of this port cannot read that action and run the step live. As upstream, a recording that stops at such a gap (reason `gap`) is not stale: a strict cache run replays what it can and runs the rest of the step live. The recording keeps the tool's name only, never its arguments +- `tools` (`AgentOptions.Tools`, `AgentTools` on the default agent) are project tools the model is offered beside the agent's own. As upstream's `defineTool`, an `AgentTool` declares `mutates` and `platforms`: a tool is offered only on the platforms it names, and a mutating tool takes an action slot. Its output is redacted and bounded to 400 lines and 16 KB (16384 UTF-8 bytes), whichever is reached first: the cut never splits a line and ends with a notice that says how much was left out (`ToolOutput.BoundToolOutput`, `ToolOutput.MaxToolOutputLines`, and `ToolOutput.MaxToolOutputBytes`, upstream's `boundToolOutput`, `MAX_TOOL_OUTPUT_LINES`, and `MAX_TOOL_OUTPUT_BYTES` in `agent/tool-output.ts`, tester-army/e2e at `f185ca8`). A first line longer than the whole byte budget keeps its head. The port redacts the result before it bounds it, so a cut cannot split a secret in two; upstream bounds first. Earlier versions of this port cut at 16384 characters with `[output cut]`. A name the agent already uses (`observe`, `complete_step`, `navigate`, `back`, `scroll`, `scroll_to`, or a verb) is refused with `INVALID_CONFIG`. Differences: the tool is an `AgentTool` built with its schema as a `JsonObject` and a delegate, since the port has no `defineTool`, AI SDK tool, or `toModelOutput`; a tool that wants upstream's `getToolContext` takes the second constructor, whose delegate gets an `AgentToolContext` (`ObserveAsync`, `AttachScreenshotAsync`), and the first constructor keeps working; and upstream's reserved names for tools it adds elsewhere are reserved as upstream's `HARNESS_TOOL_NAMES` has them (`AgentTools.HarnessToolNames`): `complete_step` (every agent step), `locate`, `run_code`, `test_lines`, `start_recording`, `stop_recording` (an e2e mcp session), and `report_finding` (explore). A project tool of one of those names is `INVALID_CONFIG`. `complete_step` is reserved as upstream's `HARNESS_TOOL_NAMES` has it, and `run_code` has no port. Upstream records a mutating tool as a gap in the trace, and so does the port: a recording holds an action of kind `tool` with the tool name, a replay runs the actions before it and then hands the step to the model, and the step is never replayed whole. Older versions of this port cannot read that action and run the step live. As upstream, a recording that stops at such a gap (reason `gap`) is not stale: a strict cache run replays what it can and runs the rest of the step live. The recording keeps the tool's name only, never its arguments - `tap`, `double_tap`, and `right_click` take an optional `modifiers` list (`Shift`, `Control`, `Alt`, `Meta`, `ControlOrMeta`) on an engine that declares `EngineCapabilities.TapModifiers`, as upstream's `tap`, `double_tap`, and `right_click` do on an engine that declares `tapModifiers`. The web engine declares it. The device engines and the document engine do not, and refuse the field with `UNSUPPORTED_CAPABILITY`. The held keys are recorded as `modifiers` of the `tap`, `doubleTap`, or `secondaryTap` action and replayed from the cache. An entry whose `modifiers` is empty, repeats a key, or sits on another action is a miss (`invalid-entry`). The grammar tools close their schemas (see Closed grammar tool inputs), so a tool called with a field its engine does not declare is refused by the schema as `unrecognized_keys`, as upstream's closed schema does, and `UNSUPPORTED_CAPABILITY` is left to a field the schema declares. - A typed value the run derived is recorded as a gap, not replayed (upstream's `derivedReason`, `agent/derived.ts`; `Derived.DerivedReason`, `Agent/Derived.cs`). `ActionDispatcher` checks the value of a `TypeAsync` fill and of a `TypeTextAsync` against the step's instruction, its params (as the executor reads them, a `Unique` as its value), and what the step's screens showed (`ObservationFeed.ShownText`, upstream's `shownText`, kept redacted as upstream keeps the prepared capture; `PixelsShown` says a screenshot went to the model). A value the instruction or params spell out replays. Otherwise the first rule that fits names the gap's `RecordedAction.Derived` (`DerivedReason`, upstream's type of that name in `cache/trace.ts`, spelled `pixels`, `whole-node`, `minted-token`, or `date` in the entry): `Pixels` once the model was shown a screenshot, `WholeNode` when a node's name or text is exactly the value, `MintedToken` when a one-token value with a digit appears as a word of its own in shown text, `Date` for a date or clock time. `TraceRecorder.RecordDerivedGap` is upstream's `recordDerivedGap`: the gap is a `tool` action whose text is `type (run-time value)`, and a replay stops at it like any project-tool gap (reason `gap`). A value the model composed itself replays. A replay types the recording's own values, so it never records a gap (the port commits a replay through `ActReplayHost`, so upstream's `replayingTrace` check has no counterpart). `E2ESession`'s check for an unchanged flow ignores `derived`, as upstream's `flowOf` does. The MCP session tool path (`Agent.RunToolAsync`) builds its dispatcher with no instruction or params and records no trace, as upstream's interactive step records none. Tests: `AgentDerived/DerivedReasonTests` (the ten `it`s of `agent-derived.test.ts`) and, .NET-only, `AgentDerived/DerivedReasonWireTests` (the enum's wire spelling) and `AgentObservation/ObservationFeedTextTests` (the feed's redacted shown text, .NET-only). A replay that stops at the gap reports the rule: `ReplayOutcome.Derived` (upstream's `derived`, set from the gap's `RecordedAction.Derived`), `CacheInfo.Derived` (upstream's `StepCacheInfo.derived`, set on a hand-off or a miss that stopped at the gap, by `StepCache.HandOff` and `Missed`), and `derived` in the step's `cache` of `report.json` (`ReportWire`; the schema and fixture already hold it). Tests: the two `it`s of `trace-replay.test.ts` that assert `derived` (`ReplayTraceTests`), and the `step-cache.test.ts` `it` "leaves an entry untouched when only the rule that flagged a typed value differs" (`FlushPipelineTests`); `schema-fixtures.test.ts`'s `it` was ported earlier (`ReportV1SchemaTests`). - `act` also offers a `double_tap` tool, which upstream does not have. Without it, the agent cannot open a control that reacts only to a double-click, such as a TodoMVC label that opens its editor. It runs the locator `doubleTap`, and the replay cache records it as `doubleTap`. Upstream and older versions of this port cannot replay that entry, so they run the step live @@ -684,10 +684,10 @@ Not ported: - `executor` (`AgentOptions.Executor`, `AgentExecutor` on `E2ESessionOptions` and the fixtures) replaces the built-in agent for `act` and `assert`. The socket is `IStepExecutor`: `Name`, and `Version`, `Cache`, `Model`, and `Judge` with defaults, so a hand-rolled executor writes `Name` and `RunStepAsync(context, cancellationToken)`. The token is upstream's `signal`; the context's `Attempt.CancellationToken` is upstream's `attempt.signal`. `StepExecutorContext` is a class a test can build and override: `Step`, `Attempt`, `Target`, `ReplayedPrefix`, `Model`, `ProviderOptions`, `Ledger`, `AgentContext`, `Budgets`, `Actions`, `ObserveAsync`, `AttachTranscript`, and `AttachScreenshotAsync`. An executor is code, so `agents..executor` in `e2e.config.json` stays `INVALID_CONFIG` and names `AgentOptions.Executor`. As upstream, `System` and `Tools` are refused beside an executor, and the `Model` and `Judge` an executor brought are the agent's, for the judgments too. An entry that names another is `INVALID_CONFIG` (`agents.default.model (gpt-5.4) and the executor's own model (gpt-5.4-mini) differ; configure the model in one place`). Two models are the same when they are one instance or share a `Name`, where upstream compares the provider and the id - The `Ledger` text an act reads is upstream's `serializeLedger` over the attempt's step records: `1. app.open passed :: /` per completed step, then ` observed: ` when it has one (` recorded in an earlier run: ` for a step the cache replayed whole, upstream's `PriorStep.replayed`, #1434), up to 8192 bytes, newest kept, with `[n earlier step(s) omitted]` first. A failed step is in it, with its status. Each label and handoff is stripped of ANSI sequences and control characters and cut to 256 and 700 bytes. A step is recorded for `agent.act`, `agent.assert`, `agent.waitFor`, `agent.extract`, a locator action and `locator.waitFor`, an `expect` matcher on a locator, the browser, or the screen, and `app.open`, `app.back`, `app.restart`, and `app.clearState`. The `Browser` and `Device` fixture calls are steps (see App and browser fixtures); `expect.poll` is not a step yet. The records are internal, and the report writes them as the attempt's `steps`. A replayed act answers upstream's `replayed N recorded action(s) zero-turn from the trace cache; recorded verdict: ...` and hands later steps the recorded verdict alone (`StepTraceSession.RecordedVerdictOf`, upstream's `recordedVerdictOf`), so the line reads ` recorded in an earlier run: ` -- The tool loop's turn policy is upstream's (`agent/tool-loop.ts`): with five turns left (`WindDownTurns`) it adds a wind-down notice, and with two or fewer (`ForcedConclusionTurns`), after a loop-guard stop, or when the remaining step time falls under a quarter of the step's budget, capped at 60 s (`ClockWindDownMs`, `ClockWindDownFraction`, never on the first turn), it offers only `done` and, on a low clock, asks for a verdict. A call to another tool on such a turn is answered ` did not run: only done is available now. Call done with your verdict.` and costs the turn. `ToolLoopExecutor.ToolCallsOnlyRule` (upstream's `TOOL_CALLS_ONLY_RULE`) is in the instructions from the first turn, and `TextReplyNotice` (`TEXT_REPLY_NOTICE`) answers a reply with no tool call. A step that runs out of turns returns a failed verdict with `STEP_NO_CONCLUSION` (`the agent used N of M turn(s) without calling done`), where the port used to throw `STEP_BUDGET_EXHAUSTED`; the model-call budget still throws it for an executor that records past `MaxModelCalls`, and the action budget (`MaxSteps`) is unchanged. Differences: the tool is `done`, not `complete_step` (deliberate, see the tool names above), so the notices and the rule name `done`. The loop sends `ModelRequest.ToolChoice` (`ModelToolChoice`, the AI SDK's `ToolChoice`; port-only name for the SDK type): the loop's mode (`toolChoice`) decides it, as upstream's `prepareStep` returns `toolChoice: this.toolChoice`, or `{ type: 'tool', toolName: 'complete_step' }` on a forced turn when the mode is `required`. The loop starts in `required`, as upstream does. The wire forms are the AI SDK's (Anthropic `any` or `{ type: 'tool', name }`, OpenAI `required` or `{ type: 'function', ... }`, Gemini `ANY` with `allowedFunctionNames`, Bedrock `any` or `{ tool: { name } }`); a request with no choice still sends `auto`. A provider option can no longer replace `tool_choice` (Anthropic, chat completions, Responses) or `toolConfig` (Google; Bedrock already reserved it): it fails with `INVALID_CONFIG`, like the other fields the client sets (port-only guard, since the port builds the body by hand). The AiTrace records the choice as asked (`{ type: 'required' }`). `ModelResponse.Warnings` (`ModelWarning`, upstream's `CallWarning`) carries what a provider changed: `AnthropicModel` and `BedrockModel` send `auto` and warn `unsupported`/`toolChoice` for a forced choice while the model thinks (both providers refuse a forced tool then; the AI SDK's provider decides this per model). A model that refuses a forced choice is asked again with `auto` (`ToolChoice.IsForcedToolChoiceRejected`, upstream's `isForcedToolChoiceRejected` in `agent/model/tool-choice.ts`: an HTTP 4xx whose message matches one of upstream's patterns, such as Claude Fable 5.1's `tool_choice: type "tool" and "any" are not supported for this model`, DeepSeek's thinking mode, or Meta Muse Spark's auto-only message, read through the exception chain), on the same turn budget and the same instructions (`FreeToolChoiceRetry`, upstream's `freeToolChoiceRetry`); the refused request is not a model call. So is a forced turn that came back as a finished reply without a tool call (`IsForcedToolCallSkipped`, upstream's `isForcedToolCallSkipped`; the port reads it off the response through `ToolChoiceViolationException`, where the AI SDK throws `AI_ToolChoiceViolationError`), and a provider that reports a `toolChoice` warning (`IsForcedToolChoiceDowngraded`, `NoticeToolChoiceDowngrade`) continues the step with `auto`. Each of the three remembers the model (`ToolLoopExecutor.FreeToolChoiceModels`, upstream's `FREE_TOOL_CHOICE_MODELS`), so later steps on it start in `auto`. Upstream's `WeakSet` is keyed by the SDK model object, which a worker builds once and shares; the port builds a provider model per session, so a model object the caller owns (`ScriptedModel`, a custom `IAgentModel`) is held weakly by object, and a built-in provider model (any disposable `IAgentModel`) is remembered for the process by its type and model id, whatever the endpoint. Deliberate .NET-only difference. Differences: the port's failure message carries the response body (`HTTP 400: {body}`), where upstream reads `responseBody` apart, and the AI SDK's `RetryError` (`Errors`, `LastError`) is carried as the `InnerException` of the `AgentException` that `ModelHttp.SendAsync` throws once its retries are spent (#1311; the `AgentException` keeps its code, `ProviderStatus`, and message, which are the last attempt's, because it is sealed and public and callers match it), and `ToolChoice.IsForcedToolChoiceRejected` follows `LastError` as upstream follows `cause ?? lastError`; upstream's `isForcedToolCallMismatched` and `recordThrownTurn` have no counterpart, since a model returns the wrong-tool or prose reply as a response the loop already records as a turn and answers (` did not run: only done is available now`), where the SDK throws on it. The loop's notices to the model carry upstream's prefixes and wording with `done` for `complete_step` and `code` for `errorCode`: `[SYSTEM NOTICE]` for the going-in-circles warning, the failure-streak warning and the wind-down notice, `[SYSTEM]` for the loop-guard stop and the low-clock notice, and the loop records upstream's `loop guard warned before turn N: ` and `the step was stopped before turn N: ` notes on the turn. The loop-guard stop keeps its per-turn `PrepareMessages` stop, which does not persist across turns. Tests: `ToolLoopToolChoice` (`ToolLoopForcedToolChoiceTests`, `ToolLoopDowngradedToolChoiceTests`, `ToolLoopForcedConclusionAnsweredWithAnotherToolTests` (whole: a tool call carries `ModelToolCall.ProviderMetadata`, upstream's `providerMetadata` of a tool call part, and the refused call goes back with it), `IsForcedToolChoiceRejectedTests`, `IsForcedToolCallSkippedTests`). -- `ToolLoop.CreateToolLoopExecutor` and `DefaultAgent.CreateBuiltInAgent` are public. Differences: `ToolLoopExecutorOptions.Tools` returns a list of `ToolLoopTool` (a `ModelTool` and a delegate that answers a `ToolLoopResult`), where upstream's returns an AI SDK `ToolSet`; `BuildPrompt` answers a string, where upstream's can answer a whole message history; the chassis adds `done`, where upstream adds `complete_step`; there is no `maxTurns` to remove. `ToolLoopHelpers` has `GuardAsync`, `Concluding`, and `ReportHardStop`, and the loop guards every tool call itself. The built-in agent drives the context the runtime builds, so it runs as a step's executor or inside a custom executor that passes its context on, and not on a context built by hand +- The tool loop's turn policy is upstream's (`agent/tool-loop.ts`): with five turns left (`WindDownTurns`) it adds a wind-down notice, and with two or fewer (`ForcedConclusionTurns`), after a loop-guard stop, or when the remaining step time falls under a quarter of the step's budget, capped at 60 s (`ClockWindDownMs`, `ClockWindDownFraction`, never on the first turn), it offers only `complete_step` and, on a low clock, asks for a verdict. A call to another tool on such a turn is answered ` did not run: only done is available now. Call done with your verdict.` and costs the turn. `ToolLoopExecutor.ToolCallsOnlyRule` (upstream's `TOOL_CALLS_ONLY_RULE`) is in the instructions from the first turn, and `TextReplyNotice` (`TEXT_REPLY_NOTICE`) answers a reply with no tool call. A step that runs out of turns returns a failed verdict with `STEP_NO_CONCLUSION` (`the agent used N of M turn(s) without calling complete_step`), where the port used to throw `STEP_BUDGET_EXHAUSTED`; the model-call budget still throws it for an executor that records past `MaxModelCalls`, and the action budget (`MaxSteps`) is unchanged. Differences: the tool is `complete_step`, not `complete_step` (deliberate, see the tool names above), so the notices and the rule name `complete_step`. The loop sends `ModelRequest.ToolChoice` (`ModelToolChoice`, the AI SDK's `ToolChoice`; port-only name for the SDK type): the loop's mode (`toolChoice`) decides it, as upstream's `prepareStep` returns `toolChoice: this.toolChoice`, or `{ type: 'tool', toolName: 'complete_step' }` on a forced turn when the mode is `required`. The loop starts in `required`, as upstream does. The wire forms are the AI SDK's (Anthropic `any` or `{ type: 'tool', name }`, OpenAI `required` or `{ type: 'function', ... }`, Gemini `ANY` with `allowedFunctionNames`, Bedrock `any` or `{ tool: { name } }`); a request with no choice still sends `auto`. A provider option can no longer replace `tool_choice` (Anthropic, chat completions, Responses) or `toolConfig` (Google; Bedrock already reserved it): it fails with `INVALID_CONFIG`, like the other fields the client sets (port-only guard, since the port builds the body by hand). The AiTrace records the choice as asked (`{ type: 'required' }`). `ModelResponse.Warnings` (`ModelWarning`, upstream's `CallWarning`) carries what a provider changed: `AnthropicModel` and `BedrockModel` send `auto` and warn `unsupported`/`toolChoice` for a forced choice while the model thinks (both providers refuse a forced tool then; the AI SDK's provider decides this per model). A model that refuses a forced choice is asked again with `auto` (`ToolChoice.IsForcedToolChoiceRejected`, upstream's `isForcedToolChoiceRejected` in `agent/model/tool-choice.ts`: an HTTP 4xx whose message matches one of upstream's patterns, such as Claude Fable 5.1's `tool_choice: type "tool" and "any" are not supported for this model`, DeepSeek's thinking mode, or Meta Muse Spark's auto-only message, read through the exception chain), on the same turn budget and the same instructions (`FreeToolChoiceRetry`, upstream's `freeToolChoiceRetry`); the refused request is not a model call. So is a forced turn that came back as a finished reply without a tool call (`IsForcedToolCallSkipped`, upstream's `isForcedToolCallSkipped`; the port reads it off the response through `ToolChoiceViolationException`, where the AI SDK throws `AI_ToolChoiceViolationError`), and a provider that reports a `toolChoice` warning (`IsForcedToolChoiceDowngraded`, `NoticeToolChoiceDowngrade`) continues the step with `auto`. Each of the three remembers the model (`ToolLoopExecutor.FreeToolChoiceModels`, upstream's `FREE_TOOL_CHOICE_MODELS`), so later steps on it start in `auto`. Upstream's `WeakSet` is keyed by the SDK model object, which a worker builds once and shares; the port builds a provider model per session, so a model object the caller owns (`ScriptedModel`, a custom `IAgentModel`) is held weakly by object, and a built-in provider model (any disposable `IAgentModel`) is remembered for the process by its type and model id, whatever the endpoint. Deliberate .NET-only difference. Differences: the port's failure message carries the response body (`HTTP 400: {body}`), where upstream reads `responseBody` apart, and the AI SDK's `RetryError` (`Errors`, `LastError`) is carried as the `InnerException` of the `AgentException` that `ModelHttp.SendAsync` throws once its retries are spent (#1311; the `AgentException` keeps its code, `ProviderStatus`, and message, which are the last attempt's, because it is sealed and public and callers match it), and `ToolChoice.IsForcedToolChoiceRejected` follows `LastError` as upstream follows `cause ?? lastError`; upstream's `isForcedToolCallMismatched` and `recordThrownTurn` have no counterpart, since a model returns the wrong-tool or prose reply as a response the loop already records as a turn and answers (` did not run: only done is available now`), where the SDK throws on it. The loop's notices to the model carry upstream's prefixes and wording with `complete_step` for `complete_step` and `code` for `errorCode`: `[SYSTEM NOTICE]` for the going-in-circles warning, the failure-streak warning and the wind-down notice, `[SYSTEM]` for the loop-guard stop and the low-clock notice, and the loop records upstream's `loop guard warned before turn N: ` and `the step was stopped before turn N: ` notes on the turn. The loop-guard stop keeps its per-turn `PrepareMessages` stop, which does not persist across turns. Tests: `ToolLoopToolChoice` (`ToolLoopForcedToolChoiceTests`, `ToolLoopDowngradedToolChoiceTests`, `ToolLoopForcedConclusionAnsweredWithAnotherToolTests` (whole: a tool call carries `ModelToolCall.ProviderMetadata`, upstream's `providerMetadata` of a tool call part, and the refused call goes back with it), `IsForcedToolChoiceRejectedTests`, `IsForcedToolCallSkippedTests`). +- `ToolLoop.CreateToolLoopExecutor` and `DefaultAgent.CreateBuiltInAgent` are public. Differences: `ToolLoopExecutorOptions.Tools` returns a list of `ToolLoopTool` (a `ModelTool` and a delegate that answers a `ToolLoopResult`), where upstream's returns an AI SDK `ToolSet`; `BuildPrompt` answers a string, where upstream's can answer a whole message history; the chassis adds `complete_step`, where upstream adds `complete_step`; there is no `maxTurns` to remove. `ToolLoopHelpers` has `GuardAsync`, `Concluding`, and `ReportHardStop`, and the loop guards every tool call itself. The built-in agent drives the context the runtime builds, so it runs as a step's executor or inside a custom executor that passes its context on, and not on a context built by hand - `ActOptions.OnBeforeAction` is upstream's `onBeforeAction` (#1142, tester-army/e2e#1387 `4461e9d`): a hook that runs before each action, replayed ones included. `BeforeActionHook` (upstream's `BeforeActionHook`) is a delegate returning a `Task`, and `BeforeActionEvent(Index, Source)` is upstream's `BeforeActionEvent` (`Source` is the `ActionSource` enum, as for `ActionEvent`). `ActionDispatcher.BeforeAsync` is upstream's `before`, called from `RunNowAsync` (upstream's `runActionNow`) once the action's slot is taken, and from `StepAccounting.RunToolAsync` for a mutating project tool (upstream's `runTool`, which sits in the dispatcher); a read-only tool does not call it. `InvokeHookAsync` is upstream's `invokeHook`, shared with `NotifyAsync`, so a throw fails the step as `ACTION_FAILED` with `agent.act onBeforeAction failed: ...` and leaves the step's recording alone (`AgentException.FromActionHook`, upstream's `ActionHookError`). The event's `Index` is the action's slot, the one its `OnAction` event carries. Differences, all .NET-only: the port's replay takes each slot in `Replay` (`TryReserve`) rather than through the dispatcher's verbs, so the replay calls the host's `BeforeAsync` right after each successful `TryReserve` (`ReserveAsync` in `Replay.cs`; `IReplayHost.BeforeAsync` is a default member, .NET-only, which `ActReplayHost` forwards to the dispatcher with `ActionSource.Replay`, where upstream reads `accounting.replayingTrace`); `StepAccounting.Attach` takes the dispatcher's hook as `beforeTool`. A replayed upload that is refused after its slot is taken has already called the hook, where upstream authorizes the upload before it takes the slot. Upstream's check that `onBeforeAction` is a function has no counterpart, since the delegate type says it. Tests: `agent-act-before-action.test.ts` is `AgentActBeforeAction/` (6 of 6). The first is `Partial_`: the hooks record their labels in a list where upstream takes an `app.screenshot` step (the document engine keeps no screenshots), so it does not check that each hook step holds one screenshot artifact. The last is `Partial_`: it makes its refused dispatch (a tap of node `n99999`) from a hand-rolled executor, because the port's tool loop turns a target that names no control away before the dispatcher takes a slot. The `act-validation` `it` "takes an onBeforeAction function and refuses any other value" is skipped (a delegate cannot be anything but a function). The `sdk-types.ts` lines are `SdkTypes/SdkTypesTests.Partial_the_before_action_observer_sees_an_index_and_a_source_and_acts_take_it_but_assertions_do_not` (reflection checks; partial: the `onBeforeAction: true` line has no counterpart, since a delegate cannot be anything else). Known difference: the port's tool loop (`ActTools`) turns a target that names no control away before the dispatcher takes a slot, so the built-in loop does not call the hook for that refused attempt, where upstream's `tap n99999` reaches the dispatcher and gets a capture; that sits in the tool loop, not in this change. -- Upstream's `grammar-tools-export` changeset (`ffed34e`, tester-army/e2e#960) exports the built-in agent's tools. The port has `AgentTools.CreateGrammarTools(StepExecutorContext, GrammarToolsOptions?)` (upstream's `createGrammarTools(ctx, { guard })`), `AgentTools.CreateVerdictTool()` with `VerdictTool` (`Tool`, `Verdict`, `Concluded`, `Execute`), `AgentTools.VerdictRules` (`VERDICT_RULES`), `DefaultAgent.BaseRules` (`BASE_RULES`), and the existing `StepExecutors.IsRuntimeHardStop`. Differences: the grammar tools are `ToolLoopTool`s (a `ModelTool` and a delegate), where upstream's are AI SDK tools, and `AgentTools` and `VerdictTool` were internal before; `GrammarToolsOptions.Guard` takes the body as `Func>` and answers a string (`body => helpers.GuardAsync(body)` runs a tool under the loop's policy), where upstream's is generic; the built-in agent passes no guard, since the loop already ends the step on a hard stop. The `done` tool stays `done` (not `complete_step`) and takes `code` (not `errorCode`), so `VerdictRules` names `done`. The built-in rules were one text; they are now `BaseRules` (the action guidance) and `VerdictRules`, and `ToolLoop` appends `VerdictRules` after the executor's guidance and the project context, as upstream's chassis appends `VERDICT_RULES`, so every loop executor, the built-in one included, sends the verdict rules last. `VerdictTool.Execute` keeps the first accepted verdict, answers `Step concluded.` or `The step already concluded.`, drops a code sent beside `passed` and says so, and refuses a blocked verdict without a blockable code; `ToolLoop` uses the same policy through a stateless internal member, so the loop's own `done` answers the same text (`Step concluded.`, or the note that a code sent beside `passed` was ignored, which names the code only when it is from the closed set, and says a code was ignored without echoing any other). A call to a tool the step does not offer is answered `failed: Model tried to call unavailable tool 'click'. Available tools: observe, tap, fill, done.` (the list is the step's own tools, so it varies with the engine and the project tools), the AI SDK's `NoSuchToolError` text that upstream's `agent-policy` and `loop-guards` tests read, with `done` for `complete_step`; the turn records it as `[click] failed: ...`. `ToolOutcome.Finish` takes the dropped code (`droppedCode`, .NET-only: upstream's tool answers the text itself, and the port's loop reads the text off the stateless outcome). `ScreenOutput` has no counterpart: the port's tool answer, `ToolLoopResult` (`Content`, `Failed`, `Pixels`), is public and carries the text and the screenshot. The screen presenter and `onActionFailed` stay internal (`ScreenPresenter`, `ActTools`). `AgentTools` members the port had made public by mistake (`MaxScrollTimes`, `ModelErrorCodes`, `IsReserved`, `Project`, `ExtractFor`, and the others) are internal now. The internal definition builder `CreateGrammarTools(EngineCapabilities, ...)` is `GrammarToolDefinitions` (.NET-only, since the public name belongs to upstream's). Tests: `SdkTypes/SdkTypesTests` is the new lines of `tests/types/sdk-types.ts` as reflection checks (the four `Partial_` ones check a shape the type checker checks upstream); `AgentGrammarTools/CreateGrammarToolsTests` is .NET-only (a loop that spreads the grammar tools beside its own, the guard, the verdict tool). +- Upstream's `grammar-tools-export` changeset (`ffed34e`, tester-army/e2e#960) exports the built-in agent's tools. The port has `AgentTools.CreateGrammarTools(StepExecutorContext, GrammarToolsOptions?)` (upstream's `createGrammarTools(ctx, { guard })`), `AgentTools.CreateVerdictTool()` with `VerdictTool` (`Tool`, `Verdict`, `Concluded`, `Execute`), `AgentTools.VerdictRules` (`VERDICT_RULES`), `DefaultAgent.BaseRules` (`BASE_RULES`), and the existing `StepExecutors.IsRuntimeHardStop`. Differences: the grammar tools are `ToolLoopTool`s (a `ModelTool` and a delegate), where upstream's are AI SDK tools, and `AgentTools` and `VerdictTool` were internal before; `GrammarToolsOptions.Guard` takes the body as `Func>` and answers a string (`body => helpers.GuardAsync(body)` runs a tool under the loop's policy), where upstream's is generic; the built-in agent passes no guard, since the loop already ends the step on a hard stop. The `complete_step` tool stays `complete_step` (not `complete_step`) and takes `code` (not `errorCode`), so `VerdictRules` names `complete_step`. The built-in rules were one text; they are now `BaseRules` (the action guidance) and `VerdictRules`, and `ToolLoop` appends `VerdictRules` after the executor's guidance and the project context, as upstream's chassis appends `VERDICT_RULES`, so every loop executor, the built-in one included, sends the verdict rules last. `VerdictTool.Execute` keeps the first accepted verdict, answers `Step concluded.` or `The step already concluded.`, drops a code sent beside `passed` and says so, and refuses a blocked verdict without a blockable code; `ToolLoop` uses the same policy through a stateless internal member, so the loop's own `complete_step` answers the same text (`Step concluded.`, or the note that a code sent beside `passed` was ignored, which names the code only when it is from the closed set, and says a code was ignored without echoing any other). A call to a tool the step does not offer is answered `failed: Model tried to call unavailable tool 'click'. Available tools: observe, tap, fill, done.` (the list is the step's own tools, so it varies with the engine and the project tools), the AI SDK's `NoSuchToolError` text that upstream's `agent-policy` and `loop-guards` tests read, with `complete_step` for `complete_step`; the turn records it as `[click] failed: ...`. `ToolOutcome.Finish` takes the dropped code (`droppedCode`, .NET-only: upstream's tool answers the text itself, and the port's loop reads the text off the stateless outcome). `ScreenOutput` has no counterpart: the port's tool answer, `ToolLoopResult` (`Content`, `Failed`, `Pixels`), is public and carries the text and the screenshot. The screen presenter and `onActionFailed` stay internal (`ScreenPresenter`, `ActTools`). `AgentTools` members the port had made public by mistake (`MaxScrollTimes`, `ModelErrorCodes`, `IsReserved`, `Project`, `ExtractFor`, and the others) are internal now. The internal definition builder `CreateGrammarTools(EngineCapabilities, ...)` is `GrammarToolDefinitions` (.NET-only, since the public name belongs to upstream's). Tests: `SdkTypes/SdkTypesTests` is the new lines of `tests/types/sdk-types.ts` as reflection checks (the four `Partial_` ones check a shape the type checker checks upstream); `AgentGrammarTools/CreateGrammarToolsTests` is .NET-only (a loop that spreads the grammar tools beside its own, the guard, the verdict tool). - The verbs are the ones the port's engines have: `TapAsync`, `DoubleTapAsync`, and `SecondaryTapAsync` (all take `ClickOptions`, whose `Modifiers` need `Target.TapModifiers`, as upstream's `tap(target, { modifiers })` needs `target.tapModifiers`), `LongPressAsync`, `HoverAsync`, `DragAsync` (upstream's `longPress`, `secondaryTap`, `hover`, and `drag`; `Target.Verbs` lists them by the action kinds the engine declares in `IEngineSession.ActionKinds`: `longPress`, `secondaryTap`, `hover`, `dragTo`), `TypeAsync`, `TypeSecretAsync`, `ClearAsync` (upstream clears with `type`), `PressAsync`, `SelectAsync`, `CheckAsync`, `UploadAsync` (upstream's `upload`), `ScrollToAsync`, `ScrollUntilAsync`, `ScrollAsync`, `NavigateAsync`, and `BackAsync`. `TypeTextAsync` (with `TypeTextOptions.Replace`) and `PressKeyAsync` are upstream's `typeText` and `pressKey`, listed when the engine declares `Keyboard`. `DismissKeyboardAsync` is upstream's `dismissKeyboard`, listed when the engine can hide its keyboard (`IEngineSession.CanDismissKeyboard`, with `Keyboard`); it is a default member of `IExecutorActions` that refuses with `UNSUPPORTED_CAPABILITY`, so an existing implementation keeps compiling. `Target.Verbs` follows the engine's capabilities: the scroll verbs need `Scroll`, `Back` needs `History`, `Upload` needs `EngineCapabilities.Upload` (.NET-only name for the `upload` verb an upstream engine lists; the web engine declares it). A node is named by its ref (`new ExecutorTarget("e12")`, the `[ref=e12]` of a line), where upstream names `n42`. Actions and looks run in call order, mutating project tools included, and a mutating `RunToolAsync` body that calls an action or a look waits behind itself, as upstream's does. A verb checks its argument before it takes a slot (`INVALID_ARGUMENT`), but the port has no key grammar, so `PressAsync` takes any non-blank key - `ObserveAsync` answers `Text`, `Truncated`, `KeyboardVisible`, `Path`, and `Revision`, and with `ExecutorObserveOptions` `Tree` (a list of root `ExecutorNode`s, redacted, with no value or selection on a secure field, a link target absolute and cut as a line cuts it) and `Pixels` (the masked screenshot sized for the model, or `PixelsWithheld`, a `VisionDegradation`). `Revision` is upstream's `revision` (`Observation.Revision`, null when the capture carries none); it has no `Viewport` and no `framePath`, since the port has none. `ReplayedPrefix` carries the actions as the text the hand-off notice lists, `ReplayHandOffReason` adds the port's `ActionBudget` and `InvalidEntry`, and has no `viewport-changed`. A replay that meets `ACTION_MAY_HAVE_COMMITTED` stops as `action-uncertain` and names the action, which the built-in agent's notice repeats. `KeyboardVisible` is upstream's `keyboardVisible` (commit `5fce7b2`): `Observation.KeyboardVisible` carries the engine's measured on-screen keyboard state to `ExecutorObservation.KeyboardVisible`, and the mobile `DeviceSession` sets it from the capture's keyboard fact (`visible` is true, `absent` is false, and an unmeasured capture leaves it null). The web engine measures none, so it is always null there. Not ported: the part of that commit in `@e2e-dev/decision` (`dismiss_keyboard` and the page fingerprint), and the `sdk-types.ts` checks, which the C# types make. Tests: `FixtureRuntime/ProjectToolDispatchTests` and `E2E.Mobile.Tests`' `Engine/ObservationTests`. - The runtime owns the budgets and the clock. `RecordModelCall` throws `STEP_BUDGET_EXHAUSTED` past `MaxModelCalls`, and a verdict never outranks a hard stop: a passed one over a spent budget, a timeout, or a code-less failure after one ends with the runtime's code. An executor that ignores its token gets one second to unwind after the step is stopped, and then the step settles without it. The closed verdict grammar is upstream's: an empty summary, an unknown code, a passed verdict with a code, or a blocked one without a blockable code is `MODEL_OUTPUT_INVALID`, and so is a runtime code the runtime did not record. A thrown error that is no agent error becomes `ACTION_FAILED`, as upstream's `toAgentError` does, and a cancel by the caller stays an `OperationCanceledException`. The message of a failed or blocked verdict is upstream's `agent.act failed: ` or `agent.act blocked: ` (`agent.assert` for an assert). The step keeps the bare summary as its explanation, as upstream does. A hard stop (`Step timed out`, a spent budget) that ends the step with no verdict, or over a passed or code-less one, has a message with no prefix (the reason, then the verdict's summary if there is one); a coded failed or blocked verdict gets the prefix, also after a hard stop @@ -702,7 +702,7 @@ Not ported: ## Explore -`Explore.ExploreAsync` ports upstream's `explore` (`packages/e2e/src/explore`), the library under `e2e explore`, and `e2e explore` ports the command (see below). An exploration is the planner (`ExtractAsync`), the explorer (the built-in agent with the exploration rules and `report_finding`), and the loop of charters under the exploration's budgets, with upstream's outcome policy: a failed step is not a finding, a step at its limit is neither, three unproductive steps in a row end the run, reported issues fail it, and a run that explored nothing, or whose charters were all blocked, stays blocked. The planner's prompt, the repairs of its answers (padded digit runs, a lone title or charter), and the explorer's rules are upstream's text, with `done` where upstream says `complete_step` and `fill_secret` where it says `type_secret`. +`Explore.ExploreAsync` ports upstream's `explore` (`packages/e2e/src/explore`), the library under `e2e explore`, and `e2e explore` ports the command (see below). An exploration is the planner (`ExtractAsync`), the explorer (the built-in agent with the exploration rules and `report_finding`), and the loop of charters under the exploration's budgets, with upstream's outcome policy: a failed step is not a finding, a step at its limit is neither, three unproductive steps in a row end the run, reported issues fail it, and a run that explored nothing, or whose charters were all blocked, stays blocked. The planner's prompt, the repairs of its answers (padded digit runs, a lone title or charter), and the explorer's rules are upstream's text, with `complete_step` where upstream says `complete_step` and `fill_secret` where it says `type_secret`. - The port has no runner, so `ExploreAsync` starts one `E2ESession` and returns an `ExploreOutcome`: `Status` (`RunStatus`: `Passed`, `Failed`, `Blocked`), `ExitCode` (0, 1, 2, or 3, from the error that ended the run by upstream's table of agent codes), `Error` (the `E2EException` that ended the run), and `Explore` (upstream's `run.explore` block as `ReportExplore`). A config, agent, target, goal, or budget error throws before anything starts, as upstream does. A cancelled token throws `OperationCanceledException`, where upstream's interrupt is exit code 130. - A failure while closing the exploration (the engine's `finish` hook, or the session's own release) does not throw and does not replace the outcome: as upstream's runner records it as a run error of phase `cleanup` and folds its category into the exit code (precedence 130, 4, 3, 2, 1, 0), `ExploreOutcome.ExitCode` is the verdict's combined with the cleanup failure's (an engine failure is 3), `Error` is the verdict's error or else the first cleanup failure, and a failure that is not an e2e error, or an engine error of a closed engine code, is an `ENGINE_FAILURE`, as upstream's `translateProvisioningError` makes it. The findings stay in `Explore`. @@ -711,7 +711,7 @@ Not ported: - A target is named by `Target`, or is the only one the config declares (`TARGET_REQUIRED` for several, `UNKNOWN_TARGET` for a name it does not declare). The config's `credentials` are the accounts, as upstream's `credentialAccounts(resolved.credentials)` carries them. `Credentials` adds accounts in code (.NET-only), by name, as `UserCredential`s, and one replaces a configured account of its name. Each travels into every step as the username and the secret `.password` (purpose `password`), and a secret anywhere in the params of an `act`, at any depth, is declared and remembered, as upstream's walk of the params finds it. The size bound of the account inventory, a quarter of the params limit, is upstream's. - A target's engine is chosen from its platform, a .NET-only bridge: upstream's config holds a live engine handle, and `e2e.config.json` still rejects an `engine` key. `web` is a `WebEngine`. `ios` and `android` are a `MobileEngine` (`MobileTargetEngine.Create`, `E2E.Mobile`) built from `app.bundleId` or `app.appPath` and `E2E_MOBILE_DEVICE`, as for `MobileAppTest`. `windows` is a `WinAppEngine` (`WinAppTargetEngine.Create`, `E2E.WinApp`; the CLI passes `WindowsTargets.Create` as the `windows` factory of `TargetEngines.Create`, which is `INVALID_CONFIG` off Windows) built from `app.appPath` or `app.bundleId`, as for `WinAppTest`. `ExploreOptions.Engine` still wins. The device engine needs Node 22.12 or newer and a booted simulator or emulator. `E2E_MOBILE_PROVIDER=limrun` or `devflow` makes the engine lease a hosted device, as the config of upstream's CLI does with `mobile({ provider: limrun() })` (the port names the provider by variable, like the adapters). Any other value fails with `INVALID_CONFIG`; `E2E.Expo`, `E2E.BrowserStack`, and `E2E.TestMu.Mobile` are not referenced by `E2E.Cli`, because upstream's CLI loads providers from the config and the port's tool ships only Limrun and DevFlow. Upstream's `openStandaloneAttempt` (`run/standalone.ts`) prepares the engine for one slot of a run of its own and finishes that run when the attempt closes, on every path (`engines.prepare` and `engines.finish`, `run/provision.ts`). The port has the same: `Standalone.OpenAttemptAsync` is `openStandaloneAttempt`, over `Standalone.OpenWorkerAsync` (`openWorker`) and its `StandaloneHost` (`openAttempt`, `close`), and `PreparedEngines` is `provision.ts`'s class of that name. Only upstream's one-shot kind exists: the hosted kind publishes each session for another test runner, and the port's tests run in their own. An engine takes part by implementing `IPreparableEngine` (.NET-only: upstream's engine handle carries optional `prepare` and `finish` members). `MobileEngine` does; `WebEngine` does not, because it leases for the attempt it starts (below). `e2e explore` and `e2e mcp` open their session through the host, so the lease happens when the session opens, never when the engine is built or when `ENGINE_IN_USE` refuses a second session, and `MobileEngine.StartAsync` leases nothing itself. What prepare leaves (the device binding) reaches the worker in `EngineStartOptions.Env`, under the run's own variables. The finish runs once, within the cleanup timeout, and a hook that outlives it is abandoned and reported as `CLEANUP_TIMEOUT` whether or not it honors its token, as upstream does. A finish that fails is an error (`close_session` shows it as a `Cleanup:` line, and `e2e explore` ends with it), because a hosted device bills until it is released. A start that fails or is cancelled after the lease was granted finishes the engine, with a failed finish added to the start's message; a caller cancel surfaces as `CANCELLED` and not as a provider failure; a lease the provider grants after the caller stopped waiting is released where it lands; and the end of the process finishes an engine still held (`AppDomain.ProcessExit`, .NET-only; a killed process is covered by the provider's own time limit, 15 minutes for Limrun). Progress lines of the provider (session URLs, lease ids) have nowhere to go in a session and are dropped, as in `WebEngine`'s standalone run; `LIMRUN_API_KEY` is registered as a secret, so an error that quotes it is redacted. DevFlow's agent runs inside the app, so `IDeviceProvider.AttachesToRunningApp` (.NET-only; a default interface member, false) makes the first `app.open()` attach with no relaunch. Live runs are not possible without Limrun credits or a DevFlow agent: the tests use a fake provider that counts leases and releases. Ported, in part, through the host: `OpenHostLifecycleTests.Partial_Host_close_cancels_an_opening_session_and_waits_for_disposal_and_finish_exactly_once` (upstream's `open-host-lifecycle` `it`, without the count of worker disposals, because the port has no worker `dispose` hook) and `EngineTargetsTests.Partial_Ends_the_run_before_any_worker_starts_when_prepare_fails` and `Partial_Abandons_a_finish_hook_that_outlives_the_cleanup_budget_as_a_cleanup_error_instead_of_hanging_the_run` (upstream's `engine-target` `it`, without the run's status, results, and error events, because the port has no run). .NET-only: `StandaloneHostTests`, `PreparableEngineTests`, `HostedDeviceSessionTests`, `ExploreHostedDeviceTests`, and `DeviceTargetsTests`. The .NET-only symbols of this change are: `IPreparableEngine`, `StandaloneAttempt` (upstream's type of that name, over the port's `E2ESession`), the process-exit finish of `StandaloneHost`, `PrepareScope` and `FinishScope` (upstream's, in C# casing), `E2ESession.LaunchAsync` (the start with the worker's environment), `DeviceTargets` and `HostedDeviceProviders` (they stand in for the provider upstream's config holds as code), `IDeviceProvider.AttachesToRunningApp`, the `MobileTargetEngine.Create` overloads that take a provider, `SessionHostOptions.Credentials`, and `FakeCloud` (test helper). The `E2E_MOBILE_PROVIDER` mapping to Limrun and DevFlow is .NET-only, because upstream names a provider in `e2e.config.ts` and the port's config file cannot hold code. DevFlow is .NET-only too: upstream has no DevFlow package, so the attach behavior has no upstream counterpart. `EngineCapabilities.PinnedApp` is the .NET-only stand-in for upstream's test of the engine handle (a session with `restart` and no `open`): `MobileEngine` declares it. `TargetEngines.LaunchesPinnedApp(target, engine)` is upstream's `launchesPinnedApp` (`config/targets.ts`, upstream #1252): the target has no `app.url` and the engine pins an app. The port's config holds a platform and no engine handle, so it takes the engine too. Explore opens the app first when the target has a URL or `LaunchesPinnedApp` holds (`openApp`), so a device target's pinned app launches before the planner's first look. To make that call work, `App.OpenAsync` on a target with no URL on a pinned-app engine launches the app through `RestartAsync`, as upstream's `launchPinnedApp` does in `run/fixtures.ts`, and refuses a path with `APP_URL_REQUIRED`; upstream's `UNSUPPORTED_CAPABILITY` for an engine without `restart` has no port, because `PinnedApp` implies it. Tests: `TargetEngineChoiceTests` and `MobileTargetEngineTests` (.NET-only); `E2eExploreTests.Launches_a_device_targets_pinned_app_before_the_planners_first_look` and `SessionHostTests.Launches_a_device_targets_pinned_app_before_the_first_observation` port upstream's two `it` of #1252. `AppSteeringHooksTests` ports three `it` of upstream's `app steering hooks` for a device (`app.open()` launches through restart, `app.open(path)` is `APP_URL_REQUIRED`, and the engine's own reason is kept); the fourth, an engine with no relaunch hook, has no port, because `PinnedApp` stands for that hook. .NET-only: `SessionHostTests.Surfaces_a_cancel_during_the_launch_of_a_pinned_app_as_a_cancel_and_disposes_the_engine`. The explore test reads the attempt's steps from `report.json`, as upstream reads the report. - The goal is redacted for the secrets of the config and the credentials' passwords, which the session also redacts wherever they appear. -- The tool is named `report_finding`. A project tool of that name is `INVALID_CONFIG` for every agent, as upstream's `HARNESS_TOOL_NAMES` has it (`done` is reserved with the grammar's names). The finding is recorded with `ArtifactId`, the id of the `screenshot` artifact that holds its evidence, as upstream's is (`attachScreenshot` writes the file, registers it with the attempt's artifacts, attaches it to the step, and answers its id; the file is `screenshots/-finding-.png` under the attempt's results directory, where upstream names it `finding-.png`). The list reporter finds the screenshot among the run's artifacts by that id, as upstream's `result(artifacts)` does; the port hands them over as the .NET-only internal `ExploreOutcome.Artifacts` and `ArtifactsRoot`, and shows the path relative to the working directory. The exploration is a test of the virtual file `explore` (`Explore.ExploreFile`, upstream's `EXPLORE_FILE`): the file joins the goal in the result id and leads `` (`explore--`), and every prompt opens with `Test: `, as upstream's does, where an `e2e mcp` session, which runs no test (no `TestFile`), gets no such line. The explore executor keeps the model's screenshot, also after a secret fill (upstream #1437), so none is kept when the engine captures none, when it cannot prove its masking covered every secure node (`MaskingUnproven`, as upstream's `pixelsProvenMasked`), or when the observation or the attachment fails (the finding is still recorded). A finding records `ObservationRevision`, the revision of the screen the finding tool observed (`Observation.Revision`, #648), as upstream's `observationRevision`; it is null when the observation failed, and the report writes it as `observationRevision` on the finding. +- The tool is named `report_finding`. A project tool of that name is `INVALID_CONFIG` for every agent, as upstream's `HARNESS_TOOL_NAMES` has it (`complete_step` is in that list too). The finding is recorded with `ArtifactId`, the id of the `screenshot` artifact that holds its evidence, as upstream's is (`attachScreenshot` writes the file, registers it with the attempt's artifacts, attaches it to the step, and answers its id; the file is `screenshots/-finding-.png` under the attempt's results directory, where upstream names it `finding-.png`). The list reporter finds the screenshot among the run's artifacts by that id, as upstream's `result(artifacts)` does; the port hands them over as the .NET-only internal `ExploreOutcome.Artifacts` and `ArtifactsRoot`, and shows the path relative to the working directory. The exploration is a test of the virtual file `explore` (`Explore.ExploreFile`, upstream's `EXPLORE_FILE`): the file joins the goal in the result id and leads `` (`explore--`), and every prompt opens with `Test: `, as upstream's does, where an `e2e mcp` session, which runs no test (no `TestFile`), gets no such line. The explore executor keeps the model's screenshot, also after a secret fill (upstream #1437), so none is kept when the engine captures none, when it cannot prove its masking covered every secure node (`MaskingUnproven`, as upstream's `pixelsProvenMasked`), or when the observation or the attachment fails (the finding is still recorded). A finding records `ObservationRevision`, the revision of the screen the finding tool observed (`Observation.Revision`, #648), as upstream's `observationRevision`; it is null when the observation failed, and the report writes it as `observationRevision` on the finding. - `e2e explore [goal]` (`E2E.Cli`, `ExploreCommand`) takes `--config`, `--target`, `--agent`, `--max-steps`, `--timeout` (milliseconds), `--headed`, `--output`, `--debug`, `--ai-trace`, `--trace`, and `--video`, with upstream's bounds, and exits with the outcome's code: 0, 1, 2, 3, and 130 for an interrupt. A bad flag exits 2 with commander's message (`option '--max-steps ' argument '101' is invalid. must be an integer from 1 through 100`). Each `E2E_USER__USERNAME` and `_PASSWORD` pair in the environment that no configured credential owns is passed as `Credentials` (named `` in lower case; .NET-only, upstream carries the config's `credentials` alone). A username with no password, or a short one, exits 2. - The output is upstream's list reporter, fed by the run's events (see [List reporter](#list-reporter) and [Run events](#run-events)): `explore-text.ts` is `ExploreText` (the severity words, why-it-ended text, step counts, and the order of findings), and `list-explore.ts` is `ExploreView` in `E2E.Cli`. `ExploreCommand` subscribes a `ListReporter` to the process's events, `Explore.ExploreAsync` writes the run's start (`HostRun.Started`, with the explored target and agent) and plan, and the command ends the run with `HostRun.FinishAsync`. The output is the `RUN` header, the header of the exploration, each finding and each step as they happen, then the `Findings` section, the `Assessment`, and the summary rows (`Findings`, `Steps`, `AI`, `Start at`, `Duration`). Differences: no live window (the command always prints the output a pipe gets, where upstream paints a live window on a terminal), the model and the target text the command redacted before are now redacted where the run writes its events (the explore state and the session's results), and a failure that is not the findings' verdict prints as `CODE: message` below the record. An interrupt (exit 130) prints no summary. An error thrown after the run started (the engine failed to start) is one `error CODE: message` line on standard error, below the header. - `--debug`, `--ai-trace`, `--trace [mode]`, and `--video [mode]` are upstream's, with its parser: a bare `--trace` or `--video` is `on`, the optional value is greedy (`--trace "Explore checkout"` reads the goal as the mode and is refused with `write --trace=, or put test files before --trace`), and the old spellings `all` and `retries` are refused with the mode they meant. `ExploreOptions.Trace`, `Video`, `Debug`, and `AiTrace` carry them; a `Trace` or `Video` is a `Run` source over the target's and the config's, else the target's mode. `E2E_TRACE`, `E2E_VIDEO`, `E2E_DEBUG`, and `E2E_AI_TRACE` set them without the flag. An exploration is one attempt, so a retry mode keeps nothing, and `ExploreOptions.Notice` says to pass `--trace on` or `--video on` (upstream's `recordingNotices` for `explore`, as `Explore.RecordingNotices`, internal), as it says that a `--video` the engine cannot record is skipped. An exploration's verdict (`ASSERTION_FAILED` with an issue finding) gets no trace page (`isExploreVerdict`, as `TracePages.IsExploreVerdict`), and `ReportRun.Explore` carries the record for that. `e2e mcp` takes none of the four, as upstream. @@ -835,7 +835,7 @@ The walk goes through open shadow roots and closed ones that page script attache - The lease tests are `BrowserProvider/` in `E2E.Tests`, from `browser-provider.test.ts`. Not ported: "rejects a sweep that is not a function" and "rejects a record that is not a function" (both are methods of the interface), "releases the replacement on dispose even when ending the attempt failed" (the port's `DisposeAsync` does not end an attempt), and "refuses, at the start and named, a recording without a start time and a stop" (a recording is typed). The other tests of the recording group are ported in full, on a real Chromium the provider leases (upstream drives a scripted page), through `StartVideoAsync` and `StopVideoAsync` of the engine session. Also not ported: "knows a recording by a parseable start time and a stop" of `provider-recording.test.ts`. The rest of that file is `ProviderRecording/`, and `provider-downloads.test.ts` is `ProviderDownloads/`, on a real Chrome over CDP, with the `Browser` fixture over the engine session - `Browser.OnDialogAsync` and `Browser.WaitForDownloadAsync` are upstream's `onDialog` and `waitForDownload`. `DialogHandler` is upstream's `DialogHandler` (`DialogHandler.Accept` and `Dismiss` are its `'accept'` and `'dismiss'`, and a `Func` converts to it), `Dialog.Type` is a `DialogType` (`Alert`, `Confirm`, `Prompt`, `BeforeUnload`) where upstream's is a string union, and `OnDialogAsync` returns `Func` for upstream's unsubscribe. The engine side is `DialogRouter` (`dialogs.ts`) and `ErrorLatch` and `WebSupport.SanitizeFilename` (`support.ts`), and `WebSession.ArtifactPath` is the surface's `artifactPath`. Differences: the engine session has `OnDialogAsync` and `WaitForDownloadAsync` (default members of `IBrowserSession`), and the `Browser` fixture attaches the file to the step (`attachArtifact`), since the port's fixture holds no context; `BrowserDownload` is the result upstream types inline. A dialog handler's failure is latched and fails the next step, as a route handler's does, and one that no step follows fails the attempt at its end: `ErrorLatch.Track` and `SettleAsync` (`support.ts`) are behind `IEngineSession.SettleAttemptAsync` (upstream's optional `settleAttempt`, whose `EngineCleanupContext` is the timeout and the token) and `E2ESession.SettleAttemptAsync` (the runner's `settleAttempt`), which the NUnit and xUnit fixtures call when the body settles and again after the teardown, within the cleanup timeout. A handler still running at the budget fails the attempt with `CLEANUP_TIMEOUT`, as upstream does, and what it throws later is dropped. The MCP server, `Explore`, and standalone sessions have no attempt end to settle at, so they do not call it, and the attempt record has no phase, so the failure is not marked `body`. A test framework's assertion is not an `E2EException`, so one thrown in a handler fails the next step as `ENGINE_FAILURE`, `dialog handler failed: ...`, where upstream's `expect` failure keeps its `ASSERTION_FAILED`. A download is a `download` artifact of the `browser.waitForDownload` step, with the media type of its extension. `AttemptScope.AttachArtifact` registers the file through the attempt's sink (`ArtifactPaths.CreateAttemptArtifacts`, see [Artifact store](#artifact-store)), as upstream's `attachArtifact` does, and the sink rewrites a download as `sink.register` does: when the session has a secret value and the media type is text-like (`text/*` or `application/json`), it rewrites the file through the redactor (`ArtifactPaths.RedactDownload`, upstream's `redactDownload`) and the record says `complete`; any other download, one it cannot decode as UTF-8, one containing NUL bytes, or a session without a secret value stays as served and the record says `incomplete`. Unlike upstream, the port rejects NUL-containing text conservatively so BOM-less UTF-16 cannot hide secrets between NULs and still be marked `complete`. `DownloadRedactionSessionTests` extends the binary-download case with a BOM-less UTF-16LE CSV regression. `AttemptScope.ArtifactsDir` is upstream's `ArtifactSink.dir`. The redaction `it`s of `engine-secrets.test.ts` (the page's echo of the password, and of the base64 credential in the `Authorization` header) are `Partial_`: they assert the failure message and the report, and scan every file the sessions wrote under their results directories (the screen, the trace page, and the download among them), where upstream scans the reporters' files as well and the download comes from the same run. The handler-assertion tests of `web-platform.test.ts` are ported with a stand-in for `expect(value).toBe`, which the port lacks: the handler throws the `ASSERTION_FAILED` it would. "fails a %s handler assertion at the attempt end when no step follows it" and "takes the failure evidence of a late handler assertion before a teardown navigates away" are `Partial_`: they do not read the phase (`body`), `attempt.cleanup`, the run's errors, the steps list, or `failure.url` from `report.json`. The `ErrorLatch` tests of `dialogs.test.ts` are `Dialogs/ErrorLatchTests`. The frame tests of `dialogs-frames.e2e.ts` are `DialogsFrames/FramesTests`, and the download tests trigger the link with `browser.locator('a[download]')` as upstream does - `Browser.WaitForDownloadsAsync(trigger, count, timeout?)` is upstream's `waitForDownloads(trigger, { count, timeout })` (#1048). It calls the trigger once and returns the first `count` downloads in start order, each its own `download` artifact; `WebSession.CollectDownloadsAsync` is upstream's `collectDownloads`, which `WaitForDownloadAsync` shares with a count of one. One timeout bounds the start, completion, and save of every file (also for `WaitForDownloadAsync`): a timeout before the trigger returns fails with `ACTION_FAILED` `browser.waitForDownload(s) exceeded its timeout of ms`, and one after names ` of downloads started within ms; the trigger resolved after ms` (`no download started` for a count of one) or `did not finish within ms`. On any failure the listener is removed, the downloads that started are canceled, and the provider's `ReadAsync` token is canceled. A `count` below one fails with `INVALID_ARGUMENT` before the trigger runs. Differences: the engine session has `WaitForDownloadsAsync` (a default member of `IBrowserSession`, beside `WaitForDownloadAsync`), and the `Browser` fixture attaches the files once all are saved, where upstream attaches each as it is saved, so a failure after the first file attaches none. `count` is an `int`, so upstream's "safe integer" check is the `count < 1` check. The wait budget in the messages is the time left less Playwright's lead, as before. `ProviderDownloads/` ports the two `it`s of `provider-downloads.test.ts` in full, with the `%i` and `%s` cases as theory data, and `Routes/BrowserRouteDecisionsTests` the five new `it`s of `routes.test.ts`; the page's `download` listener count is read from the Playwright page's event field by reflection. Not ported: `tests/types/downloads.ts`, because the C# signature refuses a missing count and a synchronous trigger at compile time. -- Tabs in the browser fixture, and the `switch_tab` and `close_tab` agent verbs (#1602, tester-army/e2e#1710 `0920ef5`). `Browser.Tabs()`, `WaitForPopupAsync(trigger, timeout?)`, and `NewTabAsync()` are upstream's `tabs()`, `waitForPopup(trigger, { timeout })`, and `newTab()`; `BrowserTab` is upstream's `BrowserTab` (`IsActive` and `IsClosed` are properties where upstream's are `isActive()` and `isClosed()`, as Playwright for .NET has `IsClosed`). The engine side is `WebEngine.Tabs.cs`, upstream's tab half of `attempt-session.ts`: `AttemptTab` is its `AttemptTab`, and `OpenTabs`, `ShownTab`, `ShowTabAsync`, `CloseTabAtAsync`, `NewTabAsync`, `NextPopupCore` (`nextPopup`), `Adopt`, `Follow`, `Orphaned`, `Join`, `Closed`, `ReturnTo`, `ShowAsync` (`show`), `Publish`, `InTurnAsync` (`inTurn`), and `RemapTabsAsync` (`remapTabs`) are its functions. `WebSession.WaitForPopupAsync` is upstream's `waitForTab` (`browser.ts`); `ReadingTabAsync` is `readingTab` and `TabsReport` is `withTabs` (`surface.ts`); `WebErrors.DispatchedInput` is `dispatchedInput` (`support.ts`), read from the Playwright for .NET message, which holds the call log; `TargetIdentity.OpenerId` is `openerId`; `IAttemptVideo.PageClosedAsync` is `pageClosed`; `AppLog` tells a tab the app opened as `the app opened a new tab at ` and a later navigation of a background tab as `a background tab navigated to `. The agent side: `EngineCapabilities.SwitchTab` and `CloseTab` with `IEngineSession.SwitchTabAsync` and `CloseTabAsync` (.NET-only capability names for upstream's optional `session.switchTab` and `session.closeTab`, whose being defined lists the verbs, as for `Resize`), `EngineSessionSpec.SwitchTab` and `CloseTab`, `Observation.Tabs` and `EngineSnapshot.Tabs` (`EngineTabs`, `validTabs`), `ExecutorVerb.SwitchTab` and `CloseTab`, `IExecutorActions.SwitchTabAsync` and `CloseTabAsync`, `ExecutorObservation.Tabs` (`ExecutorTabs`, active numbered from 1), `ActionDispatcher.SwitchTabAsync` and `CloseTabAsync`, the `switch_tab` and `close_tab` tools of `AgentTools` (run by `ActTools`), `ScreenPresenter`'s `TabNote` (`tabNote`) and the `, tab 1 of 2` of a screen header, the `switchTab` and `closeTab` recorded actions (`RecordedAction.Tab`, zero-based, read back by `TraceRead`, replayed by `Replay` and `ActReplayHost`), `SettlePolicy`, `ActionName.SwitchTab` and `CloseTab`, and the MCP catalog and `test_lines` (`switch_tab` and `close_tab` print no line). Differences: the engine session carries the tab members as default members of `IBrowserSession` over an `IAttemptTab` handle, since the port's fixture holds no surface; `Orphaned` waits `OrphanTurn` (100 ms) for the opener's `popup` where upstream waits one event-loop turn, since Playwright for .NET does not fix the order in which it delivers the two events; the tab wait before an operation is `SettleTabsAsync`, bounded by the action timeout. Not ported: the engine manifest schema and the `session/switchTab` and `session/closeTab` protocol methods (the port has no engine process, see below), and the `attempt-session.test.ts`, `browser-provider.test.ts`, `observation-capture.test.ts`, `semantic-fallback.test.ts`, and `surface-lifecycle.test.ts` edits, which only add `on` and `once` to their fake pages: the port's tests drive a real Playwright page. The 'defineEngine' unknown-key message of `engine.test.ts` has no counterpart: `EngineSessionSpec` is a typed record. 'offers the default toolset and concludes through complete_step' (`agent-act.test.ts`) stays a partial test over the document engine, which has no tabs: `TheGrammarToolsHaveClosedSchemasTests` offers the tools. `Tabs/TabsTests` ports `tabs.test.ts` in full; its video test needs ffmpeg and runs on CI, not in a sandbox without it. `NewTabFlow/NewTabFlowTests` and the `new-tab-flow` scenario of `TestsAgent/Scenarios` port the benchmark tests; the scenario is a real-model test. The `NewTabFlow` page now opens its tab from a script on the page (`webBenchmarkNewTabFlow.open`): its click handler returned the opened window through Blazor's interop, which cannot serialize a window, so the click threw and ended the circuit. +- Tabs in the browser fixture, and the `switch_tab` and `close_tab` agent verbs (#1602, tester-army/e2e#1710 `0920ef5`). `Browser.Tabs()`, `WaitForPopupAsync(trigger, timeout?)`, and `NewTabAsync()` are upstream's `tabs()`, `waitForPopup(trigger, { timeout })`, and `newTab()`; `BrowserTab` is upstream's `BrowserTab` (`IsActive` and `IsClosed` are properties where upstream's are `isActive()` and `isClosed()`, as Playwright for .NET has `IsClosed`). The engine side is `WebEngine.Tabs.cs`, upstream's tab half of `attempt-session.ts`: `AttemptTab` is its `AttemptTab`, and `OpenTabs`, `ShownTab`, `ShowTabAsync`, `CloseTabAtAsync`, `NewTabAsync`, `NextPopupCore` (`nextPopup`), `Adopt`, `Follow`, `Orphaned`, `Join`, `Closed`, `ReturnTo`, `ShowAsync` (`show`), `Publish`, `InTurnAsync` (`inTurn`), and `RemapTabsAsync` (`remapTabs`) are its functions. `WebSession.WaitForPopupAsync` is upstream's `waitForTab` (`browser.ts`); `ReadingTabAsync` is `readingTab` and `TabsReport` is `withTabs` (`surface.ts`); `WebErrors.DispatchedInput` is `dispatchedInput` (`support.ts`), read from the Playwright for .NET message, which holds the call log; `TargetIdentity.OpenerId` is `openerId`; `IAttemptVideo.PageClosedAsync` is `pageClosed`; `AppLog` tells a tab the app opened as `the app opened a new tab at ` and a later navigation of a background tab as `a background tab navigated to `. The agent side: `EngineCapabilities.SwitchTab` and `CloseTab` with `IEngineSession.SwitchTabAsync` and `CloseTabAsync` (.NET-only capability names for upstream's optional `session.switchTab` and `session.closeTab`, whose being defined lists the verbs, as for `Resize`), `EngineSessionSpec.SwitchTab` and `CloseTab`, `Observation.Tabs` and `EngineSnapshot.Tabs` (`EngineTabs`, `validTabs`), `ExecutorVerb.SwitchTab` and `CloseTab`, `IExecutorActions.SwitchTabAsync` and `CloseTabAsync`, `ExecutorObservation.Tabs` (`ExecutorTabs`, active numbered from 1), `ActionDispatcher.SwitchTabAsync` and `CloseTabAsync`, the `switch_tab` and `close_tab` tools of `AgentTools` (run by `ActTools`), `ScreenPresenter`'s `TabNote` (`tabNote`) and the `, tab 1 of 2` of a screen header, the `switchTab` and `closeTab` recorded actions (`RecordedAction.Tab`, zero-based, read back by `TraceRead`, replayed by `Replay` and `ActReplayHost`), `SettlePolicy`, `ActionName.SwitchTab` and `CloseTab`, and the MCP catalog and `test_lines` (`switch_tab` and `close_tab` print no line). Differences: the engine session carries the tab members as default members of `IBrowserSession` over an `IAttemptTab` handle, since the port's fixture holds no surface; `Orphaned` waits `OrphanTurn` (100 ms) for the opener's `popup` where upstream waits one event-loop turn, since Playwright for .NET does not fix the order in which it delivers the two events; the tab wait before an operation is `SettleTabsAsync`, bounded by the action timeout. Not ported: the engine manifest schema and the `session/switchTab` and `session/closeTab` protocol methods (the port has no engine process, see below), and the `attempt-session.test.ts`, `browser-provider.test.ts`, `observation-capture.test.ts`, `semantic-fallback.test.ts`, and `surface-lifecycle.test.ts` edits, which only add `on` and `once` to their fake pages: the port's tests drive a real Playwright page. The 'defineEngine' unknown-key message of `engine.test.ts` has no counterpart: `EngineSessionSpec` is a typed record. 'offers the default toolset and concludes through complete_step' (`agent-act.test.ts`) runs in full over the web engine (`AgentAct/AgentActWithTheDefaultToolLoopAgentExecutorTests`, #1630). `Tabs/TabsTests` ports `tabs.test.ts` in full; its video test needs ffmpeg and runs on CI, not in a sandbox without it. `NewTabFlow/NewTabFlowTests` and the `new-tab-flow` scenario of `TestsAgent/Scenarios` port the benchmark tests; the scenario is a real-model test. The `NewTabFlow` page now opens its tab from a script on the page (`webBenchmarkNewTabFlow.open`): its click handler returned the opened window through Blazor's interop, which cannot serialize a window, so the click threw and ended the circuit. - Aborted navigations and screenshots across navigations (`fix(web): classify aborted navigations and capture screenshots across navigations`, `4e7e8b8`, tester-army/e2e#1550, #1095). A `GotoAsync`, `App.OpenAsync`, back, forward, or reload whose document request a route aborted fails with `ACTION_FAILED` (`navigation failed: a route aborted the request for : ...`) instead of `ENGINE_FAILURE`; `WebSession.NavigatingAsync` is upstream's `navigating` and `NoteRouteAbort` is `noteRouteAbort`, called by `PlaywrightRoute` before every abort it makes (upstream's `abort` in `browser.ts`). A route handler that throws, such as a `FulfillAsync` with an invalid header, aborts the request, and the step fails with the handler's error: `WebSession.GuardAsync` (upstream's `PlaywrightSurface.guard`, `RunAsync` before) rethrows a latched error before it runs the step, prefers one latched while it ran, and refuses a cancelled token or stops waiting on the call once it cancels, both as `CANCELLED`. On a persistent context (`cdpEndpoint`) a caller's own cancel stays an `OperationCanceledException`, as before, because the operation waits on a shared reconnect and `Propagates_the_recovery_owner_cancellation_to_another_waiting_operation` documents that type; so the exception type depends on how the engine started. A cancelled call whose work honours the token gets `CancelGrace` (one second, .NET-only) to undo itself, such as a drag releasing the button, before the step reports. `App.ScreenshotAsync` taken while the page navigates is taken again, within the step's deadline, of the document that replaced the one its masks were resolved in (`DocumentWatch` is `watchDocuments`); a timeout is `OPERATION_TIMEOUT` `screenshot timed out: the page replaced its document during every capture for ms`. A capture that fails with no document replaced is `ENGINE_FAILURE`, as upstream (the port's artifact screenshot answered `NODE_STALE` for a navigation race before). Differences: the `framenavigated` event can arrive after the failure it explains, so the port makes one to five round trips to the page (`SettleDocumentEventsAsync`) before it reads the watch, where upstream's caret cleanup is that round trip. The port's artifact screenshot does not hide the caret (see the caret paragraph). .NET-only seams: `WebEngine.CapturePage` (what upstream's tests do by spying on `page.screenshot`), and `WebEngine.Guard` and `WebEngine.Latch` (upstream's tests call `surface.guard` and `surface.latch.latch`). Tests: `ScreenshotNavigation/` (the five its of `screenshot-navigation.test.ts`), `Routes/BrowserRouteDecisionsTests` (the two new its of `routes.test.ts`), `SurfaceGuard/` (the four its of `surface-guard.test.ts`). The `should work while navigating` it of `screenshot.e2e.ts` (whose skip the commit removes) is in the screenshot port of #1060. - With `ReconnectEndpoint`, as upstream, each attempt rides the default context of a fresh browser from `CdpEndpoint`, refuses a browser an earlier attempt of the same `WebEngine` rode, and reconnects once to the same browser and page after a transport drop. `Headers`, `BasicAuth`, `UserAgent`, `Locale`, and `TimezoneId` are `INVALID_CONFIG`, and `app.clearState()` is `UNSUPPORTED_CAPABILITY`. After a reconnect, `PerformAsync`, the agent's key press, and the viewport swipe fail with `NODE_STALE` until the next observation; the browser fixture's keyboard and mouse are direct input and keep working. As upstream, the reconnect is spent from the budget of the operation that found the transport dropped, both for the CDP dial and for the operation itself - `BasicAuth` takes a `Secret` password. As upstream, the attempt redacts the password and the `Authorization` credential the browser sends with it, padded and not, as text only: a page that shows the value outside a secure field is not masked in screenshots, and the agent cannot fill it by name. `ASecretInAnEngineOptionTests.Protects_the_secret_as_text_only_screenshots_are_kept_the_text_download_rewritten` ports upstream's `it` of that name with `App.ScreenshotAsync` and `WaitForDownloadAsync`: one session per test of upstream's suite, against one site that echoes the `Authorization` header. The port has no engine secrets to declare, so the engine resolves the password through the attempt when it starts. A plain string password is not redacted diff --git a/skills/e2e-dotnet/references/agent.md b/skills/e2e-dotnet/references/agent.md index 5dd657da2..4f5774267 100644 --- a/skills/e2e-dotnet/references/agent.md +++ b/skills/e2e-dotnet/references/agent.md @@ -529,8 +529,8 @@ they cost nothing and answer with the line that says to take one. keyboard (the web and document engines): `type_at` and `press_at` then tap the point to focus it and go through the keyboard. `type_at` takes `replace` to clear the field first. A listed node the engine cannot fill (a canvas, a - custom widget) takes `fill` through the keyboard too, after a tap that - focuses it, and the result says so. `fill` with no target types into the + custom widget) takes `type` through the keyboard too, after a tap that + focuses it, and the result says so. `type` with no target types into the focused field. `select_at` always needs a listed control. Over MCP, the session's `screenshot` is the one the point tools read their diff --git a/skills/e2e-dotnet/references/explore.md b/skills/e2e-dotnet/references/explore.md index 4b3420998..285f12328 100644 --- a/skills/e2e-dotnet/references/explore.md +++ b/skills/e2e-dotnet/references/explore.md @@ -213,7 +213,7 @@ the harness adds (see the agent reference). `Credentials` gives the accounts the explorer can sign in with, by name (read one with `Credentials.User`). The planner knows the account names and -usernames, and the agent fills each password with `fill_secret` by the +usernames, and the agent fills each password with `type_secret` by the secret `.password`; the password never reaches a model. Give only the accounts the exploration needs. Without credentials, the agent signs in with made-up ones. diff --git a/src/E2E.Acp/AcpExecutor.cs b/src/E2E.Acp/AcpExecutor.cs index a35d315dc..f7500a8df 100644 --- a/src/E2E.Acp/AcpExecutor.cs +++ b/src/E2E.Acp/AcpExecutor.cs @@ -19,7 +19,7 @@ public static class AcpExecutor private const string SessionRules = "This conversation drives a real application for an end-to-end test, one step per message: an action to perform, or an assertion to judge, with the current screen.\n" + "Use only the tools of this conversation. You have no access to the application's source, files, shell, or network.\n" + - "The rules below are for each step. A step ends when you call done; then wait for the next message. For an assertion, judge the condition from the screen without changing anything."; + "The rules below are for each step. A step ends when you call complete_step; then wait for the next message. For an assertion, judge the condition from the screen without changing anything."; private static readonly string Rules = $"{SessionRules}\n\n{DefaultAgent.BaseRules}\n\n{AgentTools.VerdictRules}"; @@ -402,7 +402,7 @@ private static StepVerdict NoConclusion(StepExecutorContext context, TurnReport said = Redact(context, said.Length > 300 ? said[..300] : said); return new StepVerdict( StepVerdictStatus.Failed, - $"The agent ended its turn ({report.StopReason}) without done{(said == "" ? "" : $": {said}")}", + $"The agent ended its turn ({report.StopReason}) without complete_step{(said == "" ? "" : $": {said}")}", "STEP_NO_CONCLUSION"); } diff --git a/src/E2E.Cli/Mcp/SessionCatalog.cs b/src/E2E.Cli/Mcp/SessionCatalog.cs index a58da6a3e..d9675de4f 100644 --- a/src/E2E.Cli/Mcp/SessionCatalog.cs +++ b/src/E2E.Cli/Mcp/SessionCatalog.cs @@ -164,7 +164,7 @@ public static SessionCatalog Create(ISessionContext context, EngineCapabilities if (verbs.Contains(ExecutorVerb.Clear)) { - Add("clear", Verb(context, screen, "clear", "type", "Clear a field.", Schema(Target), extra: new JsonObject { ["value"] = "" })); + Add("clear", Verb(context, screen, "clear", "clear", "Clear a field.", Schema(Target))); } // Upstream's grammarTools serves upload with the agent's description; the session takes the agent tool's text and names its node as target. diff --git a/src/E2E/Agent/ActTools.cs b/src/E2E/Agent/ActTools.cs index c5854612d..b1b4e289f 100644 --- a/src/E2E/Agent/ActTools.cs +++ b/src/E2E/Agent/ActTools.cs @@ -72,7 +72,9 @@ public async Task ExecuteAsync( } // A verb is a tool only where the engine's declarations unlock it, so a model that calls one anyway is told so. - if (!ActionTools.Contains(call.Name) || !AgentTools.OffersVerb(call.Name, Verbs)) + // An MCP session also runs a clear on an engine that declares the clear kind, which has no agent tool of its own (upstream has none). + var sessionClear = _session && call.Name == "clear" && Verbs.Contains(ExecutorVerb.Clear); + if (!sessionClear && (!ActionTools.Contains(call.Name) || !AgentTools.OffersVerb(call.Name, Verbs))) { return ToolOutcome.Fail("Unknown tool " + call.Name + "."); } @@ -106,7 +108,7 @@ public async Task ExecuteAsync( // The one dispatcher action the call makes spends the slot. if (!budget.TryPrepay()) { - return ToolOutcome.Fail(budget.Message + ". Take no more actions. Call done with a verdict."); + return ToolOutcome.Fail(budget.Message + ". Take no more actions. Call " + VerdictTool.Name + " with a verdict."); } try @@ -195,6 +197,7 @@ private async Task RunActionAsync(ModelToolCall call, IExecutorActi "type_secret" => actions.TypeSecretAsync(target, Args.String(arguments, "secret") ?? "", token), "select" => actions.SelectAsync(target, Args.String(arguments, "value") ?? "", token), "check" => actions.CheckAsync(target, Args.Bool(arguments, "checked") != false, token), + "clear" => actions.ClearAsync(target, token), _ => throw new TestException("INVALID_ARGUMENT", "Unknown tool " + call.Name + "."), }, token).ConfigureAwait(false); @@ -501,6 +504,7 @@ private static ActionLabel NodeLabel(string tool, JsonElement arguments, string false), "press" => new("Pressed " + key + " on " + reference + ".", "press " + key + " on " + reference, !MovesFocusOnly(key)), "select" => new("Selected \"" + value + "\" in " + reference + ".", "select \"" + value + "\" in " + reference), + "clear" => new("Cleared " + reference + ".", "clear " + reference), "check" => new((Args.Bool(arguments, "checked") == false ? "Unchecked " : "Checked ") + reference + ".", "check " + reference), _ => new(tool + " " + reference, tool + " " + reference), }; @@ -523,7 +527,7 @@ private async Task RunProjectToolAsync(ModelToolCall call, AgentToo if (tool.Mutates && !budget.TryPrepay()) { - return ToolOutcome.Fail(budget.Message + ". Take no more actions. Call done with a verdict."); + return ToolOutcome.Fail(budget.Message + ". Take no more actions. Call " + VerdictTool.Name + " with a verdict."); } string output; diff --git a/src/E2E/Agent/Agent.cs b/src/E2E/Agent/Agent.cs index aee1c10d3..5a25b9458 100644 --- a/src/E2E/Agent/Agent.cs +++ b/src/E2E/Agent/Agent.cs @@ -32,11 +32,11 @@ public sealed partial class Agent "Write the explanation first and let the verdict follow from it: the two must agree. " + "Respond with exactly one JSON object and nothing else: { \"protocolVersion\": \"agent-judgment-2\", \"explanation\": , \"verdict\": \"holds\" | \"fails\" | \"inconclusive\" }."; - // Upstream's EXTRACT_REQUEST, for the extract and done tools. Extract cannot share the judge's - // text: that says to call only done, and a model that follows its system prompt then never extracts. + // Upstream's EXTRACT_REQUEST, for the extract and complete_step tools. Extract cannot share the judge's + // text: that says to call only complete_step, and a model that follows its system prompt then never extracts. private const string ExtractSystem = "You read the requested data from the current screen. Call extract with the data when the screen shows it. " + - "Call done only when it does not: the data is not on screen, is still loading, or cannot be read; say what is missing in summary. " + + "Call complete_step only when it does not: the data is not on screen, is still loading, or cannot be read; say what is missing in summary. " + "Never fill a gap with a placeholder such as an empty string or 0. " + "An instruction that says what to answer when the data is absent, such as \"or null when none is shown\", is answered by calling extract with that value. " + "Take every value from the screen as it reads; never invent data. " + @@ -310,7 +310,7 @@ private async Task ExtractCoreAsync(string instruction, ExtractOptions? op issue, previous, fields, - "Correct a misread or a wrong shape. When the screen cannot satisfy the errors, call done with status failed and code ASSERTION_INCONCLUSIVE and say in summary what is missing, rather than change values.")), + "Correct a misread or a wrong shape. When the screen cannot satisfy the errors, call complete_step with status failed and code ASSERTION_INCONCLUSIVE and say in summary what is missing, rather than change values.")), }); } } diff --git a/tests/E2E.Tests/AgentAct/LoopGuardsAndTranscriptsTests.cs b/tests/E2E.Tests/AgentAct/LoopGuardsAndTranscriptsTests.cs index 6948e1d08..0f3e5e1f6 100644 --- a/tests/E2E.Tests/AgentAct/LoopGuardsAndTranscriptsTests.cs +++ b/tests/E2E.Tests/AgentAct/LoopGuardsAndTranscriptsTests.cs @@ -12,7 +12,7 @@ public sealed class LoopGuardsAndTranscriptsTests // Upstream also persists the executor transcript as a log artifact under --debug and checks it. // The port has no debug transcript, so only the verdict and the call count are ported. [Fact] - public async Task Forces_a_verdict_when_the_model_repeats_itself() + public async Task Partial_Forces_a_verdict_when_the_model_repeats_itself() { var model = new ScriptedModel(request => { From e0fb39d4bfb1e8d892426f15b858789e065dbdeb Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 20:48:45 +0000 Subject: [PATCH 04/13] Fix the KindsByVerb comment: MCP clear runs through the session-only clear (#1630) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1xnBhxbpAgMwCuPBUA --- src/E2E/Agent/ActContext.cs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/E2E/Agent/ActContext.cs b/src/E2E/Agent/ActContext.cs index a700367df..2894d9360 100644 --- a/src/E2E/Agent/ActContext.cs +++ b/src/E2E/Agent/ActContext.cs @@ -184,7 +184,7 @@ public void WriteTranscript() // Upstream's KINDS_BY_VERB (engine/session.ts): the action kinds an engine must declare for each kind-derived grammar verb. `check` sets a // state rather than flipping one, so it needs both directions; `fill` unlocks the plain and the secret fill alike. The port's `clear` verb - // follows its own kind and serves the MCP `clear` tool through `type`: upstream has no clear verb. + // follows its own kind and serves the MCP `clear` tool through the session-only `clear` of `ActTools`: upstream has no clear verb. private static readonly (ExecutorVerb Verb, string[] Kinds)[] KindsByVerb = [ (ExecutorVerb.Tap, ["tap"]), From 1560c6b262a306f37c91e2b3847a7f45db28dd65 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 21:35:56 +0000 Subject: [PATCH 05/13] Reserve type_secret for project tools and fix stale tool names in COMPATIBILITY (#1630) IsReserved builds the grammar with a secret, so a project tool cannot shadow type_secret. COMPATIBILITY.md names the current agent tools throughout, and the ActionNames doc comment is complete. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1xnBhxbpAgMwCuPBUA --- COMPATIBILITY.md | 20 +++++++++---------- src/E2E/Agent/ActionNames.cs | 2 +- src/E2E/Agent/AgentTools.cs | 2 +- .../TheBuiltInAgentOptionsTests.cs | 2 +- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index c8097f5d6..668261dce 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -57,7 +57,7 @@ Names are C# versions of the JavaScript API: `agent.act` is `ActAsync`, `screen. - `press` keys are checked in core, as upstream's `requireKey` (`internal/keys.ts`, `Keys.RequireKey`) does: `Locator.PressAsync` checks the key before the node is resolved (in `LocatorEngine.CheckAction`, upstream's `checkAction`), and the agent's `press` and `pressKey` check it in the dispatcher before the action is committed. A key outside the grammar (`KeyGrammar.Parse`, upstream's `parseKey`: zero or more modifiers and one key joined by `+`) is `INVALID_ARGUMENT` on every engine, where the mobile and Windows engines alone refused it with `UNSUPPORTED_CAPABILITY` (those checks stay, as upstream's engines keep theirs). `Locator.PressAsync` no longer throws an `ArgumentException` for an empty or blank key: `""` is `INVALID_ARGUMENT` like any key outside the grammar, and `" "` is a space character key. `Browser.Keyboard.PressAsync` takes any Playwright key (`Alt+KeyA`), as upstream's `browser.keyboard` does, and is not checked. The agent's `press` still refuses an empty or blank key first with `press requires a key` (a .NET-only pre-check). - Agent budgets: `ActOptions.MaxSteps`, `ActOptions.MaxModelCalls`, and `WaitForOptions.MaxModelCalls`. A per-call budget can only lower the agent's configured one. A higher or non-positive value throws `INVALID_ARGUMENT` - An `act` past its action budget ends `STEP_BUDGET_EXHAUSTED`, blocked. The model is told and may still conclude: a passing verdict, or a failure without a code, becomes the budget error. Replayed actions draw on the same budget. `navigate`, `back`, `scroll`, and `scroll_to` take a slot; `observe` does not. An `act` whose tool loop uses every model turn without a `complete_step` call ends `STEP_NO_CONCLUSION`, a failed verdict, as upstream's does (see the tool loop bullet); an executor that records one model call more than the budget allows still ends `STEP_BUDGET_EXHAUSTED`, blocked (`agent.act exhausted its model-call budget of N`) -- As upstream, an `act` guards against a model that repeats itself. The same call with the same input 3 times in a row, or a sequence of 2 to 4 calls twice, adds a warning. 5 identical calls, or a sequence 3 times, offers the model only `complete_step`, so it still writes its own verdict. The `complete_step` call never counts. The failure streak is upstream's `checkFailureStreak` (`LoopGuards.CheckFailureStreak`, tester-army/e2e `loop-guards.ts` at `f4be1c4`): 3 failed calls in a row add a notice, 5 offer the model only `complete_step`, and a call that worked starts the count over. The loop reads each tool result's text as upstream does: `LoopGuards.ExtractToolResults` (upstream's `extractToolResults`) over the tool messages, and `LoopGuards.IsFailedResult` (upstream's `isFailedResult`) for the failure shape. A call the SDK refused or a tool that threw is an `error-text` result (`ModelMessage.ErrorText`, `ToolOutcome.ErrorText`), so it counts as failed whatever its text says. Differences: a project tool's structured result has no port counterpart (the port's tool results are text, so the `it` that reads a structured result skips it), and the grammar's failure leads are the port's own text: `ActionNames.GrammarToolNames` also holds the port's `fill`, `fill_secret`, `uncheck`, and `clear`, which upstream does not have, so their failures count too. The partial read of a structured result is `Partial_Reads_text_results_...` in `LoopGuards/FailureStreakTests`. The thresholds are upstream's `LoopGuardThresholds` and `DEFAULT_LOOP_GUARD_THRESHOLDS` (`LoopGuards.DefaultLoopGuardThresholds`), and `CheckLoopGuards` and `CheckFailureStreak` take an optional `thresholds` argument; the loop passes none, and the type is internal, so no public option exposes it. A turn with no tool call counts as a failure. An executor's transcript is kept only under `E2E_DEBUG=1`, as upstream keeps it under `--debug`. +- As upstream, an `act` guards against a model that repeats itself. The same call with the same input 3 times in a row, or a sequence of 2 to 4 calls twice, adds a warning. 5 identical calls, or a sequence 3 times, offers the model only `complete_step`, so it still writes its own verdict. The `complete_step` call never counts. The failure streak is upstream's `checkFailureStreak` (`LoopGuards.CheckFailureStreak`, tester-army/e2e `loop-guards.ts` at `f4be1c4`): 3 failed calls in a row add a notice, 5 offer the model only `complete_step`, and a call that worked starts the count over. The loop reads each tool result's text as upstream does: `LoopGuards.ExtractToolResults` (upstream's `extractToolResults`) over the tool messages, and `LoopGuards.IsFailedResult` (upstream's `isFailedResult`) for the failure shape. A call the SDK refused or a tool that threw is an `error-text` result (`ModelMessage.ErrorText`, `ToolOutcome.ErrorText`), so it counts as failed whatever its text says. Differences: a project tool's structured result has no port counterpart (the port's tool results are text, so the `it` that reads a structured result skips it), and the grammar's failure leads are the port's own text: `ActionNames.GrammarToolNames` is upstream's `GRAMMAR_TOOL_NAMES`. The partial read of a structured result is `Partial_Reads_text_results_...` in `LoopGuards/FailureStreakTests`. The thresholds are upstream's `LoopGuardThresholds` and `DEFAULT_LOOP_GUARD_THRESHOLDS` (`LoopGuards.DefaultLoopGuardThresholds`), and `CheckLoopGuards` and `CheckFailureStreak` take an optional `thresholds` argument; the loop passes none, and the type is internal, so no public option exposes it. A turn with no tool call counts as a failure. An executor's transcript is kept only under `E2E_DEBUG=1`, as upstream keeps it under `--debug`. - `ActResult.ModelCalls` (0 for a full replay) and `ActResult.Actions` (replayed and live actions, counting failed attempts) - `AgentException.Blocked` and `AgentException.Explanation` (the same text as `Message`) - OpenAI-compatible tool calling @@ -171,7 +171,7 @@ Differences: - Model reasoning on step events, #1196 (tester-army/e2e `agent/executor.ts`, `agent/usage.ts`): the reasoning a turn returns rides its `model` event as `Reasoning`, for an executor turn (`ExecutorModelCall.Reasoning`, set by `ToolLoop` and `StepRecorder.AddModelCall`) and for a judgment call alike, and the list reporter prints its excerpt (`ListSteps`). `ModelResponse.ReasoningText` is upstream's `StepResult.reasoningText`: each provider fills it from the reasoning text it returned (Anthropic `thinking` blocks, Responses reasoning summaries, Gemini thought parts, Bedrock `reasoningText`, chat `reasoning_content`/`reasoning`), so an unsigned Anthropic thinking block shows in the report although it is never replayed. `ModelUsage.ReasoningField` is upstream's `reasoningField`: the text is trimmed, redacted, and cut to `ModelUsage.MaxReasoningChars` (600, upstream's `MAX_REASONING_CHARS`); a turn with no reasoning leaves the field out of the report. The AI trace keeps the full text. The port has no `ModelCallTracker`; `ToolLoop` and the judgment call pass `ReasoningText` on directly. - The `dismiss_keyboard` tool and the `dismissKeyboard` executor verb (#1014, tester-army/e2e `f4be1c4`): `dismiss_keyboard` in `AgentTools.GrammarToolDefinitions` (upstream's `dismiss_keyboard` in `createGrammarTools`, `agent/primitives.ts`), run by `ActTools`; `IExecutorActions.DismissKeyboardAsync` and `ExecutorVerb.DismissKeyboard` (upstream's `dismissKeyboard` of `ExecutorActions` and `target.verbs`), run by `ActionDispatcher.DismissKeyboardAsync` (upstream's `dismissKeyboard`). The trace records `dismissKeyboard` (`RecordedAction.Kind`), `Replay` replays it as given, `SettlePolicy` settles it like a key, and `ActionName.DismissKeyboard` reports it to `OnAction` (upstream's `actions.ts` summary is `dismiss the keyboard`). Upstream offers the verb when the engine defines the optional `keyboard.dismiss` hook; C# has no optional member to test, so `IEngineSession.CanDismissKeyboard` declares it (.NET-only, false by default, like `CanLocate`) and `IEngineSession.DismissKeyboardAsync` is the hook (a default member that refuses with `UNSUPPORTED_CAPABILITY`, so no engine breaks). `DeviceSession` is the one engine that says true, and its hook runs the dismissal under the `keyboard dismiss` input budget, as upstream's `keyboard.dismiss` does. `IExecutorActions.DismissKeyboardAsync` is a default member that refuses with `UNSUPPORTED_CAPABILITY` (.NET-only), so a custom executor context written before it keeps compiling. A session over MCP lists `dismiss_keyboard` after `back` where the engine can dismiss, and it answers with the reason `a test hides the keyboard only when it covers the next control` instead of a test line (upstream's `test-lines.ts`). Like upstream's, the tool tells the model nothing about the keyboard leaving and does not expect a change. Not ported: the decision executor's `dismiss_keyboard` (`packages/decision`, offered while `keyboardVisible`), which the port does not have. Tests: the keyboard lines of `engine.test.ts`'s verbs `it` are `Engine/CreateEngineSessionTests.Partial_Derives_the_keyboard_and_pointer_verbs_from_the_declared_hooks` (partial: the port has no `createEngineSession` or `open` verb, so it checks `ActContext.VerbsOf`); the `dismiss_keyboard` input of `agent-grammar-tools.test.ts` is checked in `AgentGrammarTools/TheGrammarToolsHaveClosedSchemasTests` (the port's schemas are not closed); `mcp-test-lines.test.ts` in `McpTestLines/SessionTestLinesTests`; the `dismissKeyboard` of `trace-replay.test.ts`'s host in `TraceReplay/TestHost`; the `keyboard dismiss` row of the mobile `engine.test.ts` input-budget `it` now calls the engine's hook (`IEngineSession.DismissKeyboardAsync`); and `agent-act-pixel-tools.test.ts` already checks that no `dismiss_keyboard` is offered without the hook. - The `resize_viewport` tool and the `resize` executor verb (#1035, tester-army/e2e#1524 `cba5e0e`): `resize_viewport` in `AgentTools.GrammarToolDefinitions` (upstream's `resize_viewport` in `createGrammarTools`, `agent/primitives.ts`), run by `ActTools` with `ExpectChange: false` as upstream's `acting`; `IExecutorActions.ResizeAsync` and `ExecutorVerb.Resize` (upstream's `resize` of `ExecutorActions` and `target.verbs`), run by `ActionDispatcher.ResizeAsync` (upstream's `resize`). The size is whole CSS pixels from 200 to 4096 per side (`Geometry.ViewportLimits`, upstream's `VIEWPORT_LIMITS`; `Geometry.RequireViewportSize`, upstream's `requireViewportSize`); a size outside them is `INVALID_ARGUMENT` and takes no action slot. The step's viewport becomes the new size (upstream's `attachViewport`, with scale 1). The trace records `resize` with its `Size` (`RecordedAction.Size`, upstream's `ResizeAction.size`; an entry whose size is outside the limits is a miss), `Replay` replays it as given, `Decide.OpensWithNavigate` passes over a leading resize, `SettlePolicy` settles it like a scroll, and `ActionName.Resize` reports it to `OnAction` (summary `resize the viewport to 390x844`). Upstream offers the verb when the engine defines the optional `session.resize` hook; C# has no optional member to test, so `EngineCapabilities.Resize` declares it (.NET-only, as `History` is for `session.back`) and `IEngineSession.ResizeAsync` is the hook (a default member that refuses with `UNSUPPORTED_CAPABILITY`). `WebEngine` declares it and sets the viewport as `Browser.SetViewportAsync` does, then hands the trace the screen. `IExecutorActions.ResizeAsync` is a default member that refuses with `UNSUPPORTED_CAPABILITY` (.NET-only), so a custom executor context written before it keeps compiling. A session over MCP lists `resize_viewport` after `back` where the engine can resize, and answers with the reason `set 390x844 with the engine fixture's viewport call` instead of a test line. `ExecutorObservation.Viewport` (internal) is now set on the executor's observation, as upstream's `viewport` is. The document engine and the device engines do not resize, so the port's tool list of the default-executor test (`AgentActWithTheDefaultToolLoopAgentExecutorTests`, which runs on the document engine) stays as it was. Not ported: the engine protocol's `session/resize` method and the manifest schema (the port has no engine protocol, #873), the decision package's test helper, and the `engine.test.ts` defineEngine key check (the port has no `defineEngine`). Tests: `AgentActVerbs/` (the resize flow and its `it`; the engine-name line of the record-then-replay `it` is not ported, as the port's replay records no engine events), `AgentAct/AgentActResizeTests`, `AgentGrammarTools/TheGrammarToolsHaveClosedSchemasTests` (partial: the port's schemas are not closed), `Engine/CreateEngineSessionTests`, `TraceCache/` (the leading resize in `DecideTraceReplayTests`; the limits in `Trace1EntryTests`), `TraceRecorder/TraceRecorderTests` (the recording, and the stored size check as a partial), `TraceReplay/ReplayTraceTests`, `McpTestLines/SessionTestLinesTests`, and the tool list and count in `McpServer/E2EMcpTests`. -- Point tools (#632, tester-army/e2e#1284 `90e6fb8`, with `drag_at` of upstream main): `tap_at`, `hover_at`, `drag_at`, `type_at`, `press_at`, and `select_at` in `AgentTools.CreatePointTools` (upstream's `createPointTools` and `POINT_TOOLS`, `agent/primitives.ts`), run by `ActTools.PointToolAsync`. A point is read off the latest screenshot (`ScreenPresenter.LatestScreenshot`, upstream's `latestScreenshot`) and scaled by `PointTap.ImagePointToViewport`; `PointTap.HitTest` is `hitTest`, and `DescribePointAction`, `DescribePointDrag`, and `DescribePointHit` are upstream's. The executor gets `TapAtAsync`, `HoverAtAsync`, `DragAtAsync`, and `HitTestAsync` (`ActionDispatcher.Points.cs`: `pointVerb`, `pointDrag`, `hitTest`); the bare point goes through `IEngineSession.PerformAtAsync` when the engine declares the pointer kind (`tap`, `hover`, `dragTo`), else a tap acts on the control by id. The trace records `tapAt`, `hoverAt`, and `dragAt` with the point, the viewport (`RecordedAction.Viewport`), and the place inside the node under it (`RecordedAction.Under`, `RecordedPointPlace`: upstream's `within`); `Replay.PlacePoint` and `PlaceWithin` are `placePoint` and `placeWithin`, and a viewport of another size hands off with `viewport-changed`. `tap_at`, `hover_at`, and `drag_at` are offered when the engine takes the bare point or lists the node verb (`tap`, `hover`, `dragTo` in `ActionKinds`), and `type_at`, `press_at`, and `select_at` with the node verb they act through (see the Agent verbs entry), as upstream's `POINT_VERBS` pairing decides; on an engine without the bare point a hover acts on the listed control by id, and a drag between points on two different listed controls is joined by id (`ActionDispatcher.DragOntoAsync`, upstream's `dragOnto`, shared with the `drag` verb), while a drag inside one control or from a point on nothing listed is `UNSUPPORTED_CAPABILITY`. Differences: a port tree lists the page's top-level nodes with no root for the page, so a point inside a named top-level node is placed in it, where upstream skips its root; there is no twin position among look-alikes (the port records none), so an ambiguous place hands off; the failure lead of a point tool carries the error code (`select_at (30, 30) failed: ACTION_FAILED: ...`), where upstream's code-less error reads without one; the replay of a recorded point emits no step engine event, as no other replayed verb does. Tests: `AgentPointTap/` (the 12 its of `agent-point-tap`, one partial), `TraceReplay/ReplayTraceBarePointTapsTests` and `ReplayTraceBarePointDragsTests` (the bare-point hover `it` of `replayTrace` is in `ReplayTraceTests`), `TraceRecorder/TraceRecorderBarePointTapsTests` and `TraceRecorderBarePointDragsTests` (the eight its of `trace-recorder.test.ts`, all `Partial_`: the port records no summary (hardkoded/e2e-dotnet#1283) and no truncation flag (#1187), and its typed entry cannot hold a field of the wrong JSON type; `TraceRecorder.Record(kind, PlacedPoint, viewport)` and `Record(from, to, viewport)` are upstream's `record` of a point action, with `PlacedPoint` and the recorder's `withinOf`; a stored place whose `fx` or `fy` is outside 0 to 1 is a miss, as upstream's `isFraction`), and the point its of `AgentActPixelTools/` and `AgentTraceCache/` on the canvas and repeats pages. `AgentActPixelTools/` has all the its of `agent-act-pixel-tools`; `AgentActVerbs/` has the repeated `press` its and the bare-point hover of `agent-act-verbs`. Keyboard (#632, `IEngineSession.TypeTextAsync`, upstream's `keyboard.type`, with `PressAsync` for keys): `ActionDispatcher.TypeTextAsync` and `PressKeyAsync` are upstream's `typeText` and `pressKey`, recorded as `typeText` (`Value`, `Replace`) and `pressKey` (`Key`) and replayed by `Replay` (`ReplayHost`'s `ActReplayHost.TypeTextAsync` and `PressKeyAsync`). `type_at` and `press_at` on a point on nothing listed tap it to focus it and go through the keyboard (`ActTools.FocusAtAsync`, upstream's `focusAt`); a listed input under the point is filled by id, and a node the engine cannot fill takes the value through the keyboard (`ActTools.TypeIntoNodeAsync`, upstream's `typeIntoNode`, `TypedThroughKeyboard`). A fill with no target types into the focused field. Differences: the port's tool is `fill`, where upstream's is `type`, so the descriptions say `fill`; `IEngineSession.TypeTextAsync` is a default member that refuses with `UNSUPPORTED_CAPABILITY` (an engine that declares `Keyboard` implements it; the default keeps custom engines compiling); the options of `typeText` are `TypeTextOptions` (C# has no inline types); `dismissKeyboard` is not ported (#1014). The `press` tool takes `times` (`AgentTools.MaxPressTimes`, upstream's `MAX_PRESS_TIMES`, 20): the first press goes to the node and the rest to the focused field through `PressKeyAsync`, one action each. The repaint report (#1012, `screen-update.ts` at `f4be1c4`): once the step shows pixels, the built-in agent compares each fresh screenshot with the one the model held (`ScreenPresenter.Attach` returns upstream's `PixelEvidence`, the digest of the image bytes, as `pixelDigest` does) and the listing with the one it read (by the shape `AgentObservation.Shape` gives). A listing that stood still under a moved screenshot opens the screen with `listed nodes unchanged; the screenshot changed. Read the effect off the image ...`, and one that stood still under the same image after an action that should have changed it says `had no visible effect` (`ScreenPresenter.RenderUnchanged`, upstream's `renderUnchanged`). With no listed control to act on, no control is blamed. Point verbs expect a change by default, as upstream's `acting`, except `type_at` on a node typed through the keyboard. The note replaces the screen as upstream's does; the `none` and `unavailable` texts are the presenter's. `Reports_a_moved_screenshot_under_an_unchanged_listing_and_blames_the_control_only_when_the_image_stood_still_too` is a whole port: the first miss reads as the `changed #n status "Hit" ...` diff line. +- Point tools (#632, tester-army/e2e#1284 `90e6fb8`, with `drag_at` of upstream main): `tap_at`, `hover_at`, `drag_at`, `type_at`, `press_at`, and `select_at` in `AgentTools.CreatePointTools` (upstream's `createPointTools` and `POINT_TOOLS`, `agent/primitives.ts`), run by `ActTools.PointToolAsync`. A point is read off the latest screenshot (`ScreenPresenter.LatestScreenshot`, upstream's `latestScreenshot`) and scaled by `PointTap.ImagePointToViewport`; `PointTap.HitTest` is `hitTest`, and `DescribePointAction`, `DescribePointDrag`, and `DescribePointHit` are upstream's. The executor gets `TapAtAsync`, `HoverAtAsync`, `DragAtAsync`, and `HitTestAsync` (`ActionDispatcher.Points.cs`: `pointVerb`, `pointDrag`, `hitTest`); the bare point goes through `IEngineSession.PerformAtAsync` when the engine declares the pointer kind (`tap`, `hover`, `dragTo`), else a tap acts on the control by id. The trace records `tapAt`, `hoverAt`, and `dragAt` with the point, the viewport (`RecordedAction.Viewport`), and the place inside the node under it (`RecordedAction.Under`, `RecordedPointPlace`: upstream's `within`); `Replay.PlacePoint` and `PlaceWithin` are `placePoint` and `placeWithin`, and a viewport of another size hands off with `viewport-changed`. `tap_at`, `hover_at`, and `drag_at` are offered when the engine takes the bare point or lists the node verb (`tap`, `hover`, `dragTo` in `ActionKinds`), and `type_at`, `press_at`, and `select_at` with the node verb they act through (see the Agent verbs entry), as upstream's `POINT_VERBS` pairing decides; on an engine without the bare point a hover acts on the listed control by id, and a drag between points on two different listed controls is joined by id (`ActionDispatcher.DragOntoAsync`, upstream's `dragOnto`, shared with the `drag` verb), while a drag inside one control or from a point on nothing listed is `UNSUPPORTED_CAPABILITY`. Differences: a port tree lists the page's top-level nodes with no root for the page, so a point inside a named top-level node is placed in it, where upstream skips its root; there is no twin position among look-alikes (the port records none), so an ambiguous place hands off; the failure lead of a point tool carries the error code (`select_at (30, 30) failed: ACTION_FAILED: ...`), where upstream's code-less error reads without one; the replay of a recorded point emits no step engine event, as no other replayed verb does. Tests: `AgentPointTap/` (the 12 its of `agent-point-tap`, one partial), `TraceReplay/ReplayTraceBarePointTapsTests` and `ReplayTraceBarePointDragsTests` (the bare-point hover `it` of `replayTrace` is in `ReplayTraceTests`), `TraceRecorder/TraceRecorderBarePointTapsTests` and `TraceRecorderBarePointDragsTests` (the eight its of `trace-recorder.test.ts`, all `Partial_`: the port records no summary (hardkoded/e2e-dotnet#1283) and no truncation flag (#1187), and its typed entry cannot hold a field of the wrong JSON type; `TraceRecorder.Record(kind, PlacedPoint, viewport)` and `Record(from, to, viewport)` are upstream's `record` of a point action, with `PlacedPoint` and the recorder's `withinOf`; a stored place whose `fx` or `fy` is outside 0 to 1 is a miss, as upstream's `isFraction`), and the point its of `AgentActPixelTools/` and `AgentTraceCache/` on the canvas and repeats pages. `AgentActPixelTools/` has all the its of `agent-act-pixel-tools`; `AgentActVerbs/` has the repeated `press` its and the bare-point hover of `agent-act-verbs`. Keyboard (#632, `IEngineSession.TypeTextAsync`, upstream's `keyboard.type`, with `PressAsync` for keys): `ActionDispatcher.TypeTextAsync` and `PressKeyAsync` are upstream's `typeText` and `pressKey`, recorded as `typeText` (`Value`, `Replace`) and `pressKey` (`Key`) and replayed by `Replay` (`ReplayHost`'s `ActReplayHost.TypeTextAsync` and `PressKeyAsync`). `type_at` and `press_at` on a point on nothing listed tap it to focus it and go through the keyboard (`ActTools.FocusAtAsync`, upstream's `focusAt`); a listed input under the point is filled by id, and a node the engine cannot fill takes the value through the keyboard (`ActTools.TypeIntoNodeAsync`, upstream's `typeIntoNode`, `TypedThroughKeyboard`). A fill with no target types into the focused field. Differences: `IEngineSession.TypeTextAsync` is a default member that refuses with `UNSUPPORTED_CAPABILITY` (an engine that declares `Keyboard` implements it; the default keeps custom engines compiling); the options of `typeText` are `TypeTextOptions` (C# has no inline types); `dismissKeyboard` is not ported (#1014). The `press` tool takes `times` (`AgentTools.MaxPressTimes`, upstream's `MAX_PRESS_TIMES`, 20): the first press goes to the node and the rest to the focused field through `PressKeyAsync`, one action each. The repaint report (#1012, `screen-update.ts` at `f4be1c4`): once the step shows pixels, the built-in agent compares each fresh screenshot with the one the model held (`ScreenPresenter.Attach` returns upstream's `PixelEvidence`, the digest of the image bytes, as `pixelDigest` does) and the listing with the one it read (by the shape `AgentObservation.Shape` gives). A listing that stood still under a moved screenshot opens the screen with `listed nodes unchanged; the screenshot changed. Read the effect off the image ...`, and one that stood still under the same image after an action that should have changed it says `had no visible effect` (`ScreenPresenter.RenderUnchanged`, upstream's `renderUnchanged`). With no listed control to act on, no control is blamed. Point verbs expect a change by default, as upstream's `acting`, except `type_at` on a node typed through the keyboard. The note replaces the screen as upstream's does; the `none` and `unavailable` texts are the presenter's. `Reports_a_moved_screenshot_under_an_unchanged_listing_and_blames_the_control_only_when_the_image_stood_still_too` is a whole port: the first miss reads as the `changed #n status "Hit" ...` diff line. - Upstream's docs site pages that have no counterpart in the port: the page objects page (`docs/page-objects.mdx`, moved to `/page-object-model` in `d30114a`), `e2e feedback` and the `docs/reference/cli.mdx` text about its opt-out variables (`4c1e2e8`), and `examples/` with its registry (`b3ae3a4`, `38b802d`, `a29edcb`, `b54646f`). The migration guides (`docs/migrate/*.mdx`, `8537abc`) are ported: see [Migration guides](#migration-guides). The docs site has .NET-only guides for upstream's five example apps of `caf51eb` (tester-army/e2e#1497: i18n, file upload, location, biometrics, deep links), under [Examples](docs-site/src/content/docs/examples/index.mdx). They are not ports: that commit only adds registry entries that link to separate repositories, so there was no guide text. Each page writes C# tests for the upstream app. The self-healing deep dive (`6fa99ea`) is ported as the [Self-healing tests](README.md#self-healing-tests) section of the guide, since the port has no docs pages for the cache, the trace page, or executors to link to. Known gap: only `e2e explore` prints a run summary, since it is the one command that feeds a run to `ListReporter` (see [List reporter](#list-reporter)): the `Cache 4 replayed · 1 handed off · 1 missed` row is `Format.CacheText` over `Format.StepsCacheTally` (`E2E.Cli`, upstream's `cacheText` and `stepsCacheTally` of `report/format.ts`), which an exploration, with its cache off, never fills, and a test run does not print ([#799](https://github.com/hardkoded/e2e-dotnet/issues/799)) - Parallel workers and serial suites - `test.fail` and the `fail` test option (tester-army/e2e#1539, `732ea8a`), partly ported. A test is marked expected to fail with NUnit's own property `E2E.Fail` (`[Property("E2E.Fail", "reason")]` on the test, or set in its body through `TestExecutionContext.CurrentContext.CurrentTest.Properties`), which needs no public API. `open_session {test}` reads the mark (`SessionTestOutcome.ExpectedFailure`, `NUnitSessionTestRunner`): a body that fails reads `failed as expected` with `Expected to fail: `, a body that passes reads `UNEXPECTED_PASS: test is marked test.fail ("") but passed`, and a `beforeAll` failure under a mark reads `failed`, as upstream. Not ported: the verdict of the run. NUnit and xUnit v3 have no expected-failure outcome, so `dotnet test` still reports a marked test that fails as failed, and a marked test that passes as passed. The port has none of upstream's registry, scheduler, retry, or reporter parts (`settleExpectedFailure`). xUnit v3 has no session runner, so the mark is read for NUnit only. @@ -333,7 +333,7 @@ Upstream's bug bash (`skills/e2e/references/bug-bash.md`, last changed in `94e6e - The config is `e2e.bugbash.config.json`. The port's config is JSON, so it cannot spread the project's config (`...base`), cannot import a `gateway(...)` model, and is a copy with the keys a bug bash changes: a target without `app.command`, `credentials`, and the three agents (`default`, `skeptic`, `fuzzer`) with `maxSteps` and `maxModelCalls` of 40, a `context`, and a `system`. An agent inherits nothing from `default`, so each repeats the model and the `context`. The credentials leave the password to `E2E_USER__PASSWORD`. - There is no `tests` key. The repro tests live in the test project's `BugBash` folder, tagged `bugbash` (`[Category("bugbash")]` or `[Trait("TestCategory", "bugbash")]`), and `dotnet test --filter "TestCategory!=bugbash"` replaces `e2e run --exclude-tag bugbash`. A repro test runs with `dotnet test --filter "FullyQualifiedName~BugBash."`. -- `e2e explore --output ` puts `report.json` beside ``, so the fan-out passes `--output .e2e/bugbash//results`. `--reporter` exits 2, so there is no `summary.md`; the log holds the `Findings`. `--session` exits 2: a charter that needs an account names its credential and the explorer signs in with `fill_secret`. +- `e2e explore --output ` puts `report.json` beside ``, so the fan-out passes `--output .e2e/bugbash//results`. `--reporter` exits 2, so there is no `summary.md`; the log holds the `Findings`. `--session` exits 2: a charter that needs an account names its credential and the explorer signs in with `type_secret`. - The docs page drops upstream's demo video, `ShowMore`, and `CodeGroup` blocks (one package manager, `dotnet`), and the card links to pages the docs site does not have. `e2e list` is not ported, so no command checks `e2e.bugbash.config.json`. ## Coding agents @@ -503,7 +503,7 @@ Not ported: upstream's `describe('explicit fixture operations')` in `packages/e2 `AcpExecutor` (the `E2E.Acp` package, namespace `E2E.Acp`) ports `@e2e-dev/acp` (tester-army/e2e#960, `0410fc9`): an `IStepExecutor` that runs `agent.act` and `agent.assert` steps on a coding agent that speaks the Agent Client Protocol on stdio. It starts the agent, opens one session per attempt, and serves the built-in action tools and the verdict tool to it as an MCP server on `127.0.0.1`. Every action goes through `ctx.Actions`, so the runner authorizes, counts, and records it, and cached steps replay as the built-in agent's do. -- Upstream's `acp({ command, args, ... })` is `AcpExecutor.Create(command, args, options)`; `claudeCode(options)` and `codex(options)` are `AcpExecutor.ClaudeCode(options)` and `AcpExecutor.Codex(options)` (`presets.ts` is the internal `Presets`). `AcpAgentOptions` (`Env`, `Model`, `System`, `Name`) and `AcpOptions` (adds `Mode` and `SessionMeta`) are records. The step is `complete_step` with a `code`, where upstream's takes `errorCode`. The tools are the port's `AgentTools.CreateGrammarTools`, so the agent gets `fill` and `fill_secret`, not `type` and `type_secret`. +- Upstream's `acp({ command, args, ... })` is `AcpExecutor.Create(command, args, options)`; `claudeCode(options)` and `codex(options)` are `AcpExecutor.ClaudeCode(options)` and `AcpExecutor.Codex(options)` (`presets.ts` is the internal `Presets`). `AcpAgentOptions` (`Env`, `Model`, `System`, `Name`) and `AcpOptions` (adds `Mode` and `SessionMeta`) are records. The step is `complete_step` with a `code`, where upstream's takes `errorCode`. The tools are the port's `AgentTools.CreateGrammarTools`, so the agent gets `type` and `type_secret`. - Upstream uses the ACP and MCP TypeScript SDKs. There is no .NET ACP SDK, so `E2E.Acp` has an internal JSON-RPC peer over newline-delimited JSON (`JsonRpcConnection`), an internal ACP session (`AcpSession`, `AgentLaunch`), and an internal MCP server over `HttpListener` (`McpToolServer`, stateless Streamable HTTP, a random path segment, a request with an `Origin` header is refused). `StepTools` builds the served tools from upstream's `tools.ts`; its per-step gate runs batched action calls one at a time, so a call queued behind `complete_step` is skipped. These are .NET-only. - Permission requests fail closed: an unknown request is answered `cancelled`. A tool of the agent's own (read from the adapter's marks and the tool kind, never from the agent's text) ends the step with `POLICY_DENIED`. - `InternalsVisibleTo E2E.Acp` on `E2E` reaches `ctx.Host.Scope.Redactor` and `ReplayHandOffReasons`, which upstream's package imports from `e2e`. .NET-only. @@ -684,16 +684,16 @@ Not ported: - `executor` (`AgentOptions.Executor`, `AgentExecutor` on `E2ESessionOptions` and the fixtures) replaces the built-in agent for `act` and `assert`. The socket is `IStepExecutor`: `Name`, and `Version`, `Cache`, `Model`, and `Judge` with defaults, so a hand-rolled executor writes `Name` and `RunStepAsync(context, cancellationToken)`. The token is upstream's `signal`; the context's `Attempt.CancellationToken` is upstream's `attempt.signal`. `StepExecutorContext` is a class a test can build and override: `Step`, `Attempt`, `Target`, `ReplayedPrefix`, `Model`, `ProviderOptions`, `Ledger`, `AgentContext`, `Budgets`, `Actions`, `ObserveAsync`, `AttachTranscript`, and `AttachScreenshotAsync`. An executor is code, so `agents..executor` in `e2e.config.json` stays `INVALID_CONFIG` and names `AgentOptions.Executor`. As upstream, `System` and `Tools` are refused beside an executor, and the `Model` and `Judge` an executor brought are the agent's, for the judgments too. An entry that names another is `INVALID_CONFIG` (`agents.default.model (gpt-5.4) and the executor's own model (gpt-5.4-mini) differ; configure the model in one place`). Two models are the same when they are one instance or share a `Name`, where upstream compares the provider and the id - The `Ledger` text an act reads is upstream's `serializeLedger` over the attempt's step records: `1. app.open passed :: /` per completed step, then ` observed: ` when it has one (` recorded in an earlier run: ` for a step the cache replayed whole, upstream's `PriorStep.replayed`, #1434), up to 8192 bytes, newest kept, with `[n earlier step(s) omitted]` first. A failed step is in it, with its status. Each label and handoff is stripped of ANSI sequences and control characters and cut to 256 and 700 bytes. A step is recorded for `agent.act`, `agent.assert`, `agent.waitFor`, `agent.extract`, a locator action and `locator.waitFor`, an `expect` matcher on a locator, the browser, or the screen, and `app.open`, `app.back`, `app.restart`, and `app.clearState`. The `Browser` and `Device` fixture calls are steps (see App and browser fixtures); `expect.poll` is not a step yet. The records are internal, and the report writes them as the attempt's `steps`. A replayed act answers upstream's `replayed N recorded action(s) zero-turn from the trace cache; recorded verdict: ...` and hands later steps the recorded verdict alone (`StepTraceSession.RecordedVerdictOf`, upstream's `recordedVerdictOf`), so the line reads ` recorded in an earlier run: ` -- The tool loop's turn policy is upstream's (`agent/tool-loop.ts`): with five turns left (`WindDownTurns`) it adds a wind-down notice, and with two or fewer (`ForcedConclusionTurns`), after a loop-guard stop, or when the remaining step time falls under a quarter of the step's budget, capped at 60 s (`ClockWindDownMs`, `ClockWindDownFraction`, never on the first turn), it offers only `complete_step` and, on a low clock, asks for a verdict. A call to another tool on such a turn is answered ` did not run: only done is available now. Call done with your verdict.` and costs the turn. `ToolLoopExecutor.ToolCallsOnlyRule` (upstream's `TOOL_CALLS_ONLY_RULE`) is in the instructions from the first turn, and `TextReplyNotice` (`TEXT_REPLY_NOTICE`) answers a reply with no tool call. A step that runs out of turns returns a failed verdict with `STEP_NO_CONCLUSION` (`the agent used N of M turn(s) without calling complete_step`), where the port used to throw `STEP_BUDGET_EXHAUSTED`; the model-call budget still throws it for an executor that records past `MaxModelCalls`, and the action budget (`MaxSteps`) is unchanged. Differences: the tool is `complete_step`, not `complete_step` (deliberate, see the tool names above), so the notices and the rule name `complete_step`. The loop sends `ModelRequest.ToolChoice` (`ModelToolChoice`, the AI SDK's `ToolChoice`; port-only name for the SDK type): the loop's mode (`toolChoice`) decides it, as upstream's `prepareStep` returns `toolChoice: this.toolChoice`, or `{ type: 'tool', toolName: 'complete_step' }` on a forced turn when the mode is `required`. The loop starts in `required`, as upstream does. The wire forms are the AI SDK's (Anthropic `any` or `{ type: 'tool', name }`, OpenAI `required` or `{ type: 'function', ... }`, Gemini `ANY` with `allowedFunctionNames`, Bedrock `any` or `{ tool: { name } }`); a request with no choice still sends `auto`. A provider option can no longer replace `tool_choice` (Anthropic, chat completions, Responses) or `toolConfig` (Google; Bedrock already reserved it): it fails with `INVALID_CONFIG`, like the other fields the client sets (port-only guard, since the port builds the body by hand). The AiTrace records the choice as asked (`{ type: 'required' }`). `ModelResponse.Warnings` (`ModelWarning`, upstream's `CallWarning`) carries what a provider changed: `AnthropicModel` and `BedrockModel` send `auto` and warn `unsupported`/`toolChoice` for a forced choice while the model thinks (both providers refuse a forced tool then; the AI SDK's provider decides this per model). A model that refuses a forced choice is asked again with `auto` (`ToolChoice.IsForcedToolChoiceRejected`, upstream's `isForcedToolChoiceRejected` in `agent/model/tool-choice.ts`: an HTTP 4xx whose message matches one of upstream's patterns, such as Claude Fable 5.1's `tool_choice: type "tool" and "any" are not supported for this model`, DeepSeek's thinking mode, or Meta Muse Spark's auto-only message, read through the exception chain), on the same turn budget and the same instructions (`FreeToolChoiceRetry`, upstream's `freeToolChoiceRetry`); the refused request is not a model call. So is a forced turn that came back as a finished reply without a tool call (`IsForcedToolCallSkipped`, upstream's `isForcedToolCallSkipped`; the port reads it off the response through `ToolChoiceViolationException`, where the AI SDK throws `AI_ToolChoiceViolationError`), and a provider that reports a `toolChoice` warning (`IsForcedToolChoiceDowngraded`, `NoticeToolChoiceDowngrade`) continues the step with `auto`. Each of the three remembers the model (`ToolLoopExecutor.FreeToolChoiceModels`, upstream's `FREE_TOOL_CHOICE_MODELS`), so later steps on it start in `auto`. Upstream's `WeakSet` is keyed by the SDK model object, which a worker builds once and shares; the port builds a provider model per session, so a model object the caller owns (`ScriptedModel`, a custom `IAgentModel`) is held weakly by object, and a built-in provider model (any disposable `IAgentModel`) is remembered for the process by its type and model id, whatever the endpoint. Deliberate .NET-only difference. Differences: the port's failure message carries the response body (`HTTP 400: {body}`), where upstream reads `responseBody` apart, and the AI SDK's `RetryError` (`Errors`, `LastError`) is carried as the `InnerException` of the `AgentException` that `ModelHttp.SendAsync` throws once its retries are spent (#1311; the `AgentException` keeps its code, `ProviderStatus`, and message, which are the last attempt's, because it is sealed and public and callers match it), and `ToolChoice.IsForcedToolChoiceRejected` follows `LastError` as upstream follows `cause ?? lastError`; upstream's `isForcedToolCallMismatched` and `recordThrownTurn` have no counterpart, since a model returns the wrong-tool or prose reply as a response the loop already records as a turn and answers (` did not run: only done is available now`), where the SDK throws on it. The loop's notices to the model carry upstream's prefixes and wording with `complete_step` for `complete_step` and `code` for `errorCode`: `[SYSTEM NOTICE]` for the going-in-circles warning, the failure-streak warning and the wind-down notice, `[SYSTEM]` for the loop-guard stop and the low-clock notice, and the loop records upstream's `loop guard warned before turn N: ` and `the step was stopped before turn N: ` notes on the turn. The loop-guard stop keeps its per-turn `PrepareMessages` stop, which does not persist across turns. Tests: `ToolLoopToolChoice` (`ToolLoopForcedToolChoiceTests`, `ToolLoopDowngradedToolChoiceTests`, `ToolLoopForcedConclusionAnsweredWithAnotherToolTests` (whole: a tool call carries `ModelToolCall.ProviderMetadata`, upstream's `providerMetadata` of a tool call part, and the refused call goes back with it), `IsForcedToolChoiceRejectedTests`, `IsForcedToolCallSkippedTests`). +- The tool loop's turn policy is upstream's (`agent/tool-loop.ts`): with five turns left (`WindDownTurns`) it adds a wind-down notice, and with two or fewer (`ForcedConclusionTurns`), after a loop-guard stop, or when the remaining step time falls under a quarter of the step's budget, capped at 60 s (`ClockWindDownMs`, `ClockWindDownFraction`, never on the first turn), it offers only `complete_step` and, on a low clock, asks for a verdict. A call to another tool on such a turn is answered ` did not run: only done is available now. Call done with your verdict.` and costs the turn. `ToolLoopExecutor.ToolCallsOnlyRule` (upstream's `TOOL_CALLS_ONLY_RULE`) is in the instructions from the first turn, and `TextReplyNotice` (`TEXT_REPLY_NOTICE`) answers a reply with no tool call. A step that runs out of turns returns a failed verdict with `STEP_NO_CONCLUSION` (`the agent used N of M turn(s) without calling complete_step`), where the port used to throw `STEP_BUDGET_EXHAUSTED`; the model-call budget still throws it for an executor that records past `MaxModelCalls`, and the action budget (`MaxSteps`) is unchanged. The tool is `complete_step`, as upstream's, and the notices and the rule name it. The loop sends `ModelRequest.ToolChoice` (`ModelToolChoice`, the AI SDK's `ToolChoice`; port-only name for the SDK type): the loop's mode (`toolChoice`) decides it, as upstream's `prepareStep` returns `toolChoice: this.toolChoice`, or `{ type: 'tool', toolName: 'complete_step' }` on a forced turn when the mode is `required`. The loop starts in `required`, as upstream does. The wire forms are the AI SDK's (Anthropic `any` or `{ type: 'tool', name }`, OpenAI `required` or `{ type: 'function', ... }`, Gemini `ANY` with `allowedFunctionNames`, Bedrock `any` or `{ tool: { name } }`); a request with no choice still sends `auto`. A provider option can no longer replace `tool_choice` (Anthropic, chat completions, Responses) or `toolConfig` (Google; Bedrock already reserved it): it fails with `INVALID_CONFIG`, like the other fields the client sets (port-only guard, since the port builds the body by hand). The AiTrace records the choice as asked (`{ type: 'required' }`). `ModelResponse.Warnings` (`ModelWarning`, upstream's `CallWarning`) carries what a provider changed: `AnthropicModel` and `BedrockModel` send `auto` and warn `unsupported`/`toolChoice` for a forced choice while the model thinks (both providers refuse a forced tool then; the AI SDK's provider decides this per model). A model that refuses a forced choice is asked again with `auto` (`ToolChoice.IsForcedToolChoiceRejected`, upstream's `isForcedToolChoiceRejected` in `agent/model/tool-choice.ts`: an HTTP 4xx whose message matches one of upstream's patterns, such as Claude Fable 5.1's `tool_choice: type "tool" and "any" are not supported for this model`, DeepSeek's thinking mode, or Meta Muse Spark's auto-only message, read through the exception chain), on the same turn budget and the same instructions (`FreeToolChoiceRetry`, upstream's `freeToolChoiceRetry`); the refused request is not a model call. So is a forced turn that came back as a finished reply without a tool call (`IsForcedToolCallSkipped`, upstream's `isForcedToolCallSkipped`; the port reads it off the response through `ToolChoiceViolationException`, where the AI SDK throws `AI_ToolChoiceViolationError`), and a provider that reports a `toolChoice` warning (`IsForcedToolChoiceDowngraded`, `NoticeToolChoiceDowngrade`) continues the step with `auto`. Each of the three remembers the model (`ToolLoopExecutor.FreeToolChoiceModels`, upstream's `FREE_TOOL_CHOICE_MODELS`), so later steps on it start in `auto`. Upstream's `WeakSet` is keyed by the SDK model object, which a worker builds once and shares; the port builds a provider model per session, so a model object the caller owns (`ScriptedModel`, a custom `IAgentModel`) is held weakly by object, and a built-in provider model (any disposable `IAgentModel`) is remembered for the process by its type and model id, whatever the endpoint. Deliberate .NET-only difference. Differences: the port's failure message carries the response body (`HTTP 400: {body}`), where upstream reads `responseBody` apart, and the AI SDK's `RetryError` (`Errors`, `LastError`) is carried as the `InnerException` of the `AgentException` that `ModelHttp.SendAsync` throws once its retries are spent (#1311; the `AgentException` keeps its code, `ProviderStatus`, and message, which are the last attempt's, because it is sealed and public and callers match it), and `ToolChoice.IsForcedToolChoiceRejected` follows `LastError` as upstream follows `cause ?? lastError`; upstream's `isForcedToolCallMismatched` and `recordThrownTurn` have no counterpart, since a model returns the wrong-tool or prose reply as a response the loop already records as a turn and answers (` did not run: only done is available now`), where the SDK throws on it. The loop's notices to the model carry upstream's prefixes and wording with `complete_step` for `complete_step` and `code` for `errorCode`: `[SYSTEM NOTICE]` for the going-in-circles warning, the failure-streak warning and the wind-down notice, `[SYSTEM]` for the loop-guard stop and the low-clock notice, and the loop records upstream's `loop guard warned before turn N: ` and `the step was stopped before turn N: ` notes on the turn. The loop-guard stop keeps its per-turn `PrepareMessages` stop, which does not persist across turns. Tests: `ToolLoopToolChoice` (`ToolLoopForcedToolChoiceTests`, `ToolLoopDowngradedToolChoiceTests`, `ToolLoopForcedConclusionAnsweredWithAnotherToolTests` (whole: a tool call carries `ModelToolCall.ProviderMetadata`, upstream's `providerMetadata` of a tool call part, and the refused call goes back with it), `IsForcedToolChoiceRejectedTests`, `IsForcedToolCallSkippedTests`). - `ToolLoop.CreateToolLoopExecutor` and `DefaultAgent.CreateBuiltInAgent` are public. Differences: `ToolLoopExecutorOptions.Tools` returns a list of `ToolLoopTool` (a `ModelTool` and a delegate that answers a `ToolLoopResult`), where upstream's returns an AI SDK `ToolSet`; `BuildPrompt` answers a string, where upstream's can answer a whole message history; the chassis adds `complete_step`, where upstream adds `complete_step`; there is no `maxTurns` to remove. `ToolLoopHelpers` has `GuardAsync`, `Concluding`, and `ReportHardStop`, and the loop guards every tool call itself. The built-in agent drives the context the runtime builds, so it runs as a step's executor or inside a custom executor that passes its context on, and not on a context built by hand - `ActOptions.OnBeforeAction` is upstream's `onBeforeAction` (#1142, tester-army/e2e#1387 `4461e9d`): a hook that runs before each action, replayed ones included. `BeforeActionHook` (upstream's `BeforeActionHook`) is a delegate returning a `Task`, and `BeforeActionEvent(Index, Source)` is upstream's `BeforeActionEvent` (`Source` is the `ActionSource` enum, as for `ActionEvent`). `ActionDispatcher.BeforeAsync` is upstream's `before`, called from `RunNowAsync` (upstream's `runActionNow`) once the action's slot is taken, and from `StepAccounting.RunToolAsync` for a mutating project tool (upstream's `runTool`, which sits in the dispatcher); a read-only tool does not call it. `InvokeHookAsync` is upstream's `invokeHook`, shared with `NotifyAsync`, so a throw fails the step as `ACTION_FAILED` with `agent.act onBeforeAction failed: ...` and leaves the step's recording alone (`AgentException.FromActionHook`, upstream's `ActionHookError`). The event's `Index` is the action's slot, the one its `OnAction` event carries. Differences, all .NET-only: the port's replay takes each slot in `Replay` (`TryReserve`) rather than through the dispatcher's verbs, so the replay calls the host's `BeforeAsync` right after each successful `TryReserve` (`ReserveAsync` in `Replay.cs`; `IReplayHost.BeforeAsync` is a default member, .NET-only, which `ActReplayHost` forwards to the dispatcher with `ActionSource.Replay`, where upstream reads `accounting.replayingTrace`); `StepAccounting.Attach` takes the dispatcher's hook as `beforeTool`. A replayed upload that is refused after its slot is taken has already called the hook, where upstream authorizes the upload before it takes the slot. Upstream's check that `onBeforeAction` is a function has no counterpart, since the delegate type says it. Tests: `agent-act-before-action.test.ts` is `AgentActBeforeAction/` (6 of 6). The first is `Partial_`: the hooks record their labels in a list where upstream takes an `app.screenshot` step (the document engine keeps no screenshots), so it does not check that each hook step holds one screenshot artifact. The last is `Partial_`: it makes its refused dispatch (a tap of node `n99999`) from a hand-rolled executor, because the port's tool loop turns a target that names no control away before the dispatcher takes a slot. The `act-validation` `it` "takes an onBeforeAction function and refuses any other value" is skipped (a delegate cannot be anything but a function). The `sdk-types.ts` lines are `SdkTypes/SdkTypesTests.Partial_the_before_action_observer_sees_an_index_and_a_source_and_acts_take_it_but_assertions_do_not` (reflection checks; partial: the `onBeforeAction: true` line has no counterpart, since a delegate cannot be anything else). Known difference: the port's tool loop (`ActTools`) turns a target that names no control away before the dispatcher takes a slot, so the built-in loop does not call the hook for that refused attempt, where upstream's `tap n99999` reaches the dispatcher and gets a capture; that sits in the tool loop, not in this change. -- Upstream's `grammar-tools-export` changeset (`ffed34e`, tester-army/e2e#960) exports the built-in agent's tools. The port has `AgentTools.CreateGrammarTools(StepExecutorContext, GrammarToolsOptions?)` (upstream's `createGrammarTools(ctx, { guard })`), `AgentTools.CreateVerdictTool()` with `VerdictTool` (`Tool`, `Verdict`, `Concluded`, `Execute`), `AgentTools.VerdictRules` (`VERDICT_RULES`), `DefaultAgent.BaseRules` (`BASE_RULES`), and the existing `StepExecutors.IsRuntimeHardStop`. Differences: the grammar tools are `ToolLoopTool`s (a `ModelTool` and a delegate), where upstream's are AI SDK tools, and `AgentTools` and `VerdictTool` were internal before; `GrammarToolsOptions.Guard` takes the body as `Func>` and answers a string (`body => helpers.GuardAsync(body)` runs a tool under the loop's policy), where upstream's is generic; the built-in agent passes no guard, since the loop already ends the step on a hard stop. The `complete_step` tool stays `complete_step` (not `complete_step`) and takes `code` (not `errorCode`), so `VerdictRules` names `complete_step`. The built-in rules were one text; they are now `BaseRules` (the action guidance) and `VerdictRules`, and `ToolLoop` appends `VerdictRules` after the executor's guidance and the project context, as upstream's chassis appends `VERDICT_RULES`, so every loop executor, the built-in one included, sends the verdict rules last. `VerdictTool.Execute` keeps the first accepted verdict, answers `Step concluded.` or `The step already concluded.`, drops a code sent beside `passed` and says so, and refuses a blocked verdict without a blockable code; `ToolLoop` uses the same policy through a stateless internal member, so the loop's own `complete_step` answers the same text (`Step concluded.`, or the note that a code sent beside `passed` was ignored, which names the code only when it is from the closed set, and says a code was ignored without echoing any other). A call to a tool the step does not offer is answered `failed: Model tried to call unavailable tool 'click'. Available tools: observe, tap, fill, done.` (the list is the step's own tools, so it varies with the engine and the project tools), the AI SDK's `NoSuchToolError` text that upstream's `agent-policy` and `loop-guards` tests read, with `complete_step` for `complete_step`; the turn records it as `[click] failed: ...`. `ToolOutcome.Finish` takes the dropped code (`droppedCode`, .NET-only: upstream's tool answers the text itself, and the port's loop reads the text off the stateless outcome). `ScreenOutput` has no counterpart: the port's tool answer, `ToolLoopResult` (`Content`, `Failed`, `Pixels`), is public and carries the text and the screenshot. The screen presenter and `onActionFailed` stay internal (`ScreenPresenter`, `ActTools`). `AgentTools` members the port had made public by mistake (`MaxScrollTimes`, `ModelErrorCodes`, `IsReserved`, `Project`, `ExtractFor`, and the others) are internal now. The internal definition builder `CreateGrammarTools(EngineCapabilities, ...)` is `GrammarToolDefinitions` (.NET-only, since the public name belongs to upstream's). Tests: `SdkTypes/SdkTypesTests` is the new lines of `tests/types/sdk-types.ts` as reflection checks (the four `Partial_` ones check a shape the type checker checks upstream); `AgentGrammarTools/CreateGrammarToolsTests` is .NET-only (a loop that spreads the grammar tools beside its own, the guard, the verdict tool). +- Upstream's `grammar-tools-export` changeset (`ffed34e`, tester-army/e2e#960) exports the built-in agent's tools. The port has `AgentTools.CreateGrammarTools(StepExecutorContext, GrammarToolsOptions?)` (upstream's `createGrammarTools(ctx, { guard })`), `AgentTools.CreateVerdictTool()` with `VerdictTool` (`Tool`, `Verdict`, `Concluded`, `Execute`), `AgentTools.VerdictRules` (`VERDICT_RULES`), `DefaultAgent.BaseRules` (`BASE_RULES`), and the existing `StepExecutors.IsRuntimeHardStop`. Differences: the grammar tools are `ToolLoopTool`s (a `ModelTool` and a delegate), where upstream's are AI SDK tools, and `AgentTools` and `VerdictTool` were internal before; `GrammarToolsOptions.Guard` takes the body as `Func>` and answers a string (`body => helpers.GuardAsync(body)` runs a tool under the loop's policy), where upstream's is generic; the built-in agent passes no guard, since the loop already ends the step on a hard stop. The `complete_step` tool takes `code` (not `errorCode`, #1643), and `VerdictRules` names `complete_step`. The built-in rules were one text; they are now `BaseRules` (the action guidance) and `VerdictRules`, and `ToolLoop` appends `VerdictRules` after the executor's guidance and the project context, as upstream's chassis appends `VERDICT_RULES`, so every loop executor, the built-in one included, sends the verdict rules last. `VerdictTool.Execute` keeps the first accepted verdict, answers `Step concluded.` or `The step already concluded.`, drops a code sent beside `passed` and says so, and refuses a blocked verdict without a blockable code; `ToolLoop` uses the same policy through a stateless internal member, so the loop's own `complete_step` answers the same text (`Step concluded.`, or the note that a code sent beside `passed` was ignored, which names the code only when it is from the closed set, and says a code was ignored without echoing any other). A call to a tool the step does not offer is answered `failed: Model tried to call unavailable tool 'click'. Available tools: observe, tap, type, complete_step.` (the list is the step's own tools, so it varies with the engine and the project tools), the AI SDK's `NoSuchToolError` text that upstream's `agent-policy` and `loop-guards` tests read, with `complete_step` for `complete_step`; the turn records it as `[click] failed: ...`. `ToolOutcome.Finish` takes the dropped code (`droppedCode`, .NET-only: upstream's tool answers the text itself, and the port's loop reads the text off the stateless outcome). `ScreenOutput` has no counterpart: the port's tool answer, `ToolLoopResult` (`Content`, `Failed`, `Pixels`), is public and carries the text and the screenshot. The screen presenter and `onActionFailed` stay internal (`ScreenPresenter`, `ActTools`). `AgentTools` members the port had made public by mistake (`MaxScrollTimes`, `ModelErrorCodes`, `IsReserved`, `Project`, `ExtractFor`, and the others) are internal now. The internal definition builder `CreateGrammarTools(EngineCapabilities, ...)` is `GrammarToolDefinitions` (.NET-only, since the public name belongs to upstream's). Tests: `SdkTypes/SdkTypesTests` is the new lines of `tests/types/sdk-types.ts` as reflection checks (the four `Partial_` ones check a shape the type checker checks upstream); `AgentGrammarTools/CreateGrammarToolsTests` is .NET-only (a loop that spreads the grammar tools beside its own, the guard, the verdict tool). - The verbs are the ones the port's engines have: `TapAsync`, `DoubleTapAsync`, and `SecondaryTapAsync` (all take `ClickOptions`, whose `Modifiers` need `Target.TapModifiers`, as upstream's `tap(target, { modifiers })` needs `target.tapModifiers`), `LongPressAsync`, `HoverAsync`, `DragAsync` (upstream's `longPress`, `secondaryTap`, `hover`, and `drag`; `Target.Verbs` lists them by the action kinds the engine declares in `IEngineSession.ActionKinds`: `longPress`, `secondaryTap`, `hover`, `dragTo`), `TypeAsync`, `TypeSecretAsync`, `ClearAsync` (upstream clears with `type`), `PressAsync`, `SelectAsync`, `CheckAsync`, `UploadAsync` (upstream's `upload`), `ScrollToAsync`, `ScrollUntilAsync`, `ScrollAsync`, `NavigateAsync`, and `BackAsync`. `TypeTextAsync` (with `TypeTextOptions.Replace`) and `PressKeyAsync` are upstream's `typeText` and `pressKey`, listed when the engine declares `Keyboard`. `DismissKeyboardAsync` is upstream's `dismissKeyboard`, listed when the engine can hide its keyboard (`IEngineSession.CanDismissKeyboard`, with `Keyboard`); it is a default member of `IExecutorActions` that refuses with `UNSUPPORTED_CAPABILITY`, so an existing implementation keeps compiling. `Target.Verbs` follows the engine's capabilities: the scroll verbs need `Scroll`, `Back` needs `History`, `Upload` needs `EngineCapabilities.Upload` (.NET-only name for the `upload` verb an upstream engine lists; the web engine declares it). A node is named by its ref (`new ExecutorTarget("e12")`, the `[ref=e12]` of a line), where upstream names `n42`. Actions and looks run in call order, mutating project tools included, and a mutating `RunToolAsync` body that calls an action or a look waits behind itself, as upstream's does. A verb checks its argument before it takes a slot (`INVALID_ARGUMENT`), but the port has no key grammar, so `PressAsync` takes any non-blank key - `ObserveAsync` answers `Text`, `Truncated`, `KeyboardVisible`, `Path`, and `Revision`, and with `ExecutorObserveOptions` `Tree` (a list of root `ExecutorNode`s, redacted, with no value or selection on a secure field, a link target absolute and cut as a line cuts it) and `Pixels` (the masked screenshot sized for the model, or `PixelsWithheld`, a `VisionDegradation`). `Revision` is upstream's `revision` (`Observation.Revision`, null when the capture carries none); it has no `Viewport` and no `framePath`, since the port has none. `ReplayedPrefix` carries the actions as the text the hand-off notice lists, `ReplayHandOffReason` adds the port's `ActionBudget` and `InvalidEntry`, and has no `viewport-changed`. A replay that meets `ACTION_MAY_HAVE_COMMITTED` stops as `action-uncertain` and names the action, which the built-in agent's notice repeats. `KeyboardVisible` is upstream's `keyboardVisible` (commit `5fce7b2`): `Observation.KeyboardVisible` carries the engine's measured on-screen keyboard state to `ExecutorObservation.KeyboardVisible`, and the mobile `DeviceSession` sets it from the capture's keyboard fact (`visible` is true, `absent` is false, and an unmeasured capture leaves it null). The web engine measures none, so it is always null there. Not ported: the part of that commit in `@e2e-dev/decision` (`dismiss_keyboard` and the page fingerprint), and the `sdk-types.ts` checks, which the C# types make. Tests: `FixtureRuntime/ProjectToolDispatchTests` and `E2E.Mobile.Tests`' `Engine/ObservationTests`. - The runtime owns the budgets and the clock. `RecordModelCall` throws `STEP_BUDGET_EXHAUSTED` past `MaxModelCalls`, and a verdict never outranks a hard stop: a passed one over a spent budget, a timeout, or a code-less failure after one ends with the runtime's code. An executor that ignores its token gets one second to unwind after the step is stopped, and then the step settles without it. The closed verdict grammar is upstream's: an empty summary, an unknown code, a passed verdict with a code, or a blocked one without a blockable code is `MODEL_OUTPUT_INVALID`, and so is a runtime code the runtime did not record. A thrown error that is no agent error becomes `ACTION_FAILED`, as upstream's `toAgentError` does, and a cancel by the caller stays an `OperationCanceledException`. The message of a failed or blocked verdict is upstream's `agent.act failed: ` or `agent.act blocked: ` (`agent.assert` for an assert). The step keeps the bare summary as its explanation, as upstream does. A hard stop (`Step timed out`, a spent budget) that ends the step with no verdict, or over a passed or code-less one, has a message with no prefix (the reason, then the verdict's summary if there is one); a coded failed or blocked verdict gets the prefix, also after a hard stop - `AssertAsync` on an agent with an executor goes to it as a step of kind `Assert`, with the agent's budgets and `JudgmentTimeout`. An assert is never cached, and a passing one verifies the acts before it. `Vision` is refused with `UNSUPPORTED_CAPABILITY`, since the executor decides what its model sees, and `Screenshot` keeps its evidence after the verdict, bounded by the action timeout: a capture that outlives it is dropped and the verdict stands. `WaitForAsync` and `ExtractAsync` keep the judge, which is the executor's `Judge` or `Model`. `ExecutorCache.Off` never replays or records, and takes no look before the step. A step with the cache on that cannot read its first screen (no app opened) still runs, and records nothing -- `TypeSecretAsync` fills a secret the step's params declared, and refuses with `POLICY_DENIED` any other name, a disabled or non-editable field, and a field whose purpose does not fit a password secret. A secret the attempt does not know is `AUTH_CREDENTIAL_UNAVAILABLE`. The built-in agent's `fill_secret` goes through the same check, so its refusals read as the dispatcher's messages and not as `Secret fill refused: ...`. `ExecutorStep.Params` is JSON: a `Secret` is its name and, when set, its purpose, and every string and key that holds a secret value the attempt knows is redacted. `ExecutorAttempt` has the session's `TestTitle` as `TestId`, a fresh id per session, and `Index` as the attempt number minus one; `ExecutorStep.Index` counts the agent steps of the attempt, where upstream's counts every step. `AttachTranscript` and `AttachScreenshotAsync` write under the attempt's results directory (`transcript-.txt`, `screenshots/-