diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index c109cd20..27defa3e 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -57,7 +57,7 @@ Names are C# versions of the JavaScript API: `agent.act` is `ActAsync`, `screen. - `press` keys are checked in core, as upstream's `requireKey` (`internal/keys.ts`, `Keys.RequireKey`) does: `Locator.PressAsync` checks the key before the node is resolved (in `LocatorEngine.CheckAction`, upstream's `checkAction`), and the agent's `press` and `pressKey` check it in the dispatcher before the action is committed. A key outside the grammar (`KeyGrammar.Parse`, upstream's `parseKey`: zero or more modifiers and one key joined by `+`) is `INVALID_ARGUMENT` on every engine, where the mobile and Windows engines alone refused it with `UNSUPPORTED_CAPABILITY` (those checks stay, as upstream's engines keep theirs). `Locator.PressAsync` no longer throws an `ArgumentException` for an empty or blank key: `""` is `INVALID_ARGUMENT` like any key outside the grammar, and `" "` is a space character key. `Browser.Keyboard.PressAsync` takes any Playwright key (`Alt+KeyA`), as upstream's `browser.keyboard` does, and is not checked. The agent's `press` still refuses an empty or blank key first with `press requires a key` (a .NET-only pre-check). - Agent budgets: `ActOptions.MaxSteps`, `ActOptions.MaxModelCalls`, and `WaitForOptions.MaxModelCalls`. A per-call budget can only lower the agent's configured one. A higher or non-positive value throws `INVALID_ARGUMENT` - An `act` past its action budget ends `STEP_BUDGET_EXHAUSTED`, blocked. The model is told and may still conclude: a passing verdict, or a failure without a code, becomes the budget error. Replayed actions draw on the same budget. `navigate`, `back`, `scroll`, and `scroll_to` take a slot; `observe` does not. An `act` whose tool loop uses every model turn without a `done` call ends `STEP_NO_CONCLUSION`, a failed verdict, as upstream's does (see the tool loop bullet); an executor that records one model call more than the budget allows still ends `STEP_BUDGET_EXHAUSTED`, blocked (`agent.act exhausted its model-call budget of N`) -- As upstream, an `act` guards against a model that repeats itself. The same call with the same input 3 times in a row, or a sequence of 2 to 4 calls twice, adds a warning. 5 identical calls, or a sequence 3 times, offers the model only `done`, so it still writes its own verdict. The `done` call never counts. The failure streak is upstream's `checkFailureStreak` (`LoopGuards.CheckFailureStreak`, tester-army/e2e `loop-guards.ts` at `f4be1c4`): 3 failed calls in a row add a notice, 5 offer the model only `done`, and a call that worked starts the count over. The loop reads each tool result's text as upstream does: `LoopGuards.ExtractToolResults` (upstream's `extractToolResults`) over the tool messages, and `LoopGuards.IsFailedResult` (upstream's `isFailedResult`) for the failure shape. A call the SDK refused or a tool that threw is an `error-text` result (`ModelMessage.ErrorText`, `ToolOutcome.ErrorText`), so it counts as failed whatever its text says. Differences: a project tool's structured result has no port counterpart (the port's tool results are text, so the `it` that reads a structured result skips it), and the grammar's failure leads are the port's own text: `ActionNames.GrammarToolNames` also holds the port's `fill`, `fill_secret`, `uncheck`, and `clear`, which upstream does not have, so their failures count too. The partial read of a structured result is `Partial_Reads_text_results_...` in `LoopGuards/FailureStreakTests`. The thresholds are upstream's `LoopGuardThresholds` and `DEFAULT_LOOP_GUARD_THRESHOLDS` (`LoopGuards.DefaultLoopGuardThresholds`), and `CheckLoopGuards` and `CheckFailureStreak` take an optional `thresholds` argument; the loop passes none, and the type is internal, so no public option exposes it. A turn with no tool call counts as a failure. An executor's transcript is kept only under `E2E_DEBUG=1`, as upstream keeps it under `--debug`. +- As upstream, an `act` guards against a model that repeats itself. The same call with the same input 3 times in a row, or a sequence of 2 to 4 calls twice, adds a warning. 5 identical calls, or a sequence 3 times, offers the model only `done`, so it still writes its own verdict. The `done` call never counts. The failure streak is upstream's `checkFailureStreak` (`LoopGuards.CheckFailureStreak`, tester-army/e2e `loop-guards.ts` at `f4be1c4`): 3 failed calls in a row add a notice, 5 offer the model only `done`, and a call that worked starts the count over. The loop reads each tool result's text as upstream does: `LoopGuards.ExtractToolResults` (upstream's `extractToolResults`) over the tool messages, and `LoopGuards.IsFailedResult` (upstream's `isFailedResult`) for the failure shape. A call the SDK refused or a tool that threw is an `error-text` result (`ModelMessage.ErrorText`, `ToolOutcome.ErrorText`), so it counts as failed whatever its text says. Differences: a project tool's structured result has no port counterpart (the port's tool results are text, so the `it` that reads a structured result skips it), and the grammar's failure leads are the port's own text: `ActionNames.GrammarToolNames` also holds the port's `fill`, `fill_secret`, `uncheck`, and `clear`, which upstream does not have, so their failures count too. The partial read of a structured result is `Partial_Reads_text_results_...` in `LoopGuards/FailureStreakTests`. The thresholds are upstream's `LoopGuardThresholds` and `DEFAULT_LOOP_GUARD_THRESHOLDS` (`LoopGuards.DefaultLoopGuardThresholds`), and `CheckLoopGuards` and `CheckFailureStreak` take an optional `thresholds` argument; the loop passes none, and the type is internal, so no public option exposes it. A turn with no tool call counts as a failure. An agent step's transcript is kept only under `E2E_DEBUG=1`, as upstream keeps it under `--debug`: the loop attaches it, and it is a step-attributed `log` artifact (see Custom executors). - `ActResult.ModelCalls` (0 for a full replay) and `ActResult.Actions` (replayed and live actions, counting failed attempts) - `AgentException.Blocked` and `AgentException.Explanation` (the same text as `Message`) - OpenAI-compatible tool calling @@ -230,7 +230,7 @@ Upstream's `--debug` prints the run's phase timings and a table of agent steps o - The phases are `session.launch` (the engine's start), `test.body` (from the start of the session until `Complete` or dispose), `session.close` (the engine's disposal), and the agent's `agent.observe`, `agent.model`, and `agent.action` (`Phases`, internal; upstream `agent/phases.ts`, where `instrumentPhase` also records step events). `agent.cache` is declared upstream and used nowhere, so the port omits it. Upstream also has `config.load`, `engine.prepare`, `engine.init`, `app.start(...)`, `collect`, and `scheduler`: the port has no such phase, because it has no runner, no app process, and no engine prepare step. - The table is written to stderr when each session is released, once per test, where upstream prints one table for the run. The title is the same, `[e2e debug] phase timings (wall ...)`. - The agent step table is `DebugSteps.AgentStepTable` (internal, `Report/DebugSteps.cs`), upstream's `agentStepTable` in `report/debug-steps.ts`: one row per agent step in execution order, with the step's time, the time in its model, observe, and action events, its model calls, tokens in and out, the share the provider served from its prompt cache, and its cost. A `via` column appears when the steps named several models. It is written to stderr after the phase table when each session is released, from that session's step records (upstream takes the run's results and serial groups). Differences: `StepModelInfo` has no provenance (`endpoint`, `adapterVersion`, `policyVersion`), and `IAgentModel.Name` spells provider and model together, so a step's label is that name alone unless an executor reports a provider. The `model` column sums the step's `Model` events, one per model call (`StepRecorder.AddModelCall`). The `observe` and `action` columns sum the step's `Observation` and `Engine` events (see [Trace page](#trace-page)). `cached` and `cost` read `-` unless the provider or an executor reports them in `ExecutorModelCall`. Every built-in provider returns the cache read tokens (and the write tokens, where it has them) in `ModelCallUsage`, and `ToolLoop` and the agent's own loop (`Agent.cs`) pass them on. The AI trace still writes only the input and output tokens. Bedrock and Anthropic report the cached tokens apart from the input tokens, so the port adds them back into `InputTokens`, as the AI SDK does (`convertAmazonBedrockUsage`: `total: inputTokens + cacheReadTokens + cacheWriteTokens`), and upstream's `ModelUsage.record` reads that total. Google counts the cached tokens inside `promptTokenCount` already. The OpenRouter cost is the field upstream's `readCost` reads (`metadata?.['openrouter']?.['usage']`, then `cost`). A count that is negative or not whole is dropped, as upstream's `tokenCount` does. Only an OpenAI-compatible endpoint reports a cost, and only when it sends `usage.cost`, as OpenRouter does. Upstream's `readCost` also reads the AI Gateway metadata; the port has no gateway. `debug-steps.test.ts` is `DebugSteps/AgentStepTableTests` (both `it`; the first is partial: the port has no serial groups, so its third step is a second attempt's). `.NET-only`: `ModelCallUsage` (the per-call usage a model returns, in `ModelResponse.Usage`; upstream reads the AI SDK's usage object; a breaking rename of the port's former public `ModelUsage`, whose name is now upstream's accumulator, see [Model usage](#model-usage)), the agent step table case of `SessionDebugTests`, `ModelCallUsageTests` and `Provider_usage_carries_the_cache_split_and_the_cost_when_reported`. -- An executor's `AttachTranscript` text is kept as `transcript-.txt` only under debug. `.NET-only`: `SessionDebugTests`, and the debug-on and debug-off cases of `AttachedTranscriptTests`. +- An executor's `AttachTranscript` text is kept as a step `log` artifact, `transcript-.txt`, only under debug. `.NET-only`: `SessionDebugTests`, and the debug-on and debug-off cases of `AttachedTranscriptTests`. ## AI trace @@ -692,7 +692,7 @@ Not ported: - `ObserveAsync` answers `Text`, `Truncated`, `KeyboardVisible`, `Path`, and `Revision`, and with `ExecutorObserveOptions` `Tree` (a list of root `ExecutorNode`s, redacted, with no value or selection on a secure field, a link target absolute and cut as a line cuts it) and `Pixels` (the masked screenshot sized for the model, or `PixelsWithheld`, a `VisionDegradation`). `Revision` is upstream's `revision` (`Observation.Revision`, null when the capture carries none); it has no `Viewport` and no `framePath`, since the port has none. `ReplayedPrefix` carries the actions as the text the hand-off notice lists, `ReplayHandOffReason` adds the port's `ActionBudget` and `InvalidEntry`, and has no `viewport-changed`. A replay that meets `ACTION_MAY_HAVE_COMMITTED` stops as `action-uncertain` and names the action, which the built-in agent's notice repeats. `KeyboardVisible` is upstream's `keyboardVisible` (commit `5fce7b2`): `Observation.KeyboardVisible` carries the engine's measured on-screen keyboard state to `ExecutorObservation.KeyboardVisible`, and the mobile `DeviceSession` sets it from the capture's keyboard fact (`visible` is true, `absent` is false, and an unmeasured capture leaves it null). The web engine measures none, so it is always null there. Not ported: the part of that commit in `@e2e-dev/decision` (`dismiss_keyboard` and the page fingerprint), and the `sdk-types.ts` checks, which the C# types make. Tests: `FixtureRuntime/ProjectToolDispatchTests` and `E2E.Mobile.Tests`' `Engine/ObservationTests`. - The runtime owns the budgets and the clock. `RecordModelCall` throws `STEP_BUDGET_EXHAUSTED` past `MaxModelCalls`, and a verdict never outranks a hard stop: a passed one over a spent budget, a timeout, or a code-less failure after one ends with the runtime's code. An executor that ignores its token gets one second to unwind after the step is stopped, and then the step settles without it. The closed verdict grammar is upstream's: an empty summary, an unknown code, a passed verdict with a code, or a blocked one without a blockable code is `MODEL_OUTPUT_INVALID`, and so is a runtime code the runtime did not record. A thrown error that is no agent error becomes `ACTION_FAILED`, as upstream's `toAgentError` does, and a cancel by the caller stays an `OperationCanceledException`. The message of a failed or blocked verdict is upstream's `agent.act failed: ` or `agent.act blocked: ` (`agent.assert` for an assert). The step keeps the bare summary as its explanation, as upstream does. A hard stop (`Step timed out`, a spent budget) that ends the step with no verdict, or over a passed or code-less one, has a message with no prefix (the reason, then the verdict's summary if there is one); a coded failed or blocked verdict gets the prefix, also after a hard stop - `AssertAsync` on an agent with an executor goes to it as a step of kind `Assert`, with the agent's budgets and `JudgmentTimeout`. An assert is never cached, and a passing one verifies the acts before it. `Vision` is refused with `UNSUPPORTED_CAPABILITY`, since the executor decides what its model sees, and `Screenshot` keeps its evidence after the verdict, bounded by the action timeout: a capture that outlives it is dropped and the verdict stands. `WaitForAsync` and `ExtractAsync` keep the judge, which is the executor's `Judge` or `Model`. `ExecutorCache.Off` never replays or records, and takes no look before the step. A step with the cache on that cannot read its first screen (no app opened) still runs, and records nothing -- `TypeSecretAsync` fills a secret the step's params declared, and refuses with `POLICY_DENIED` any other name, a disabled or non-editable field, and a field whose purpose does not fit a password secret. A secret the attempt does not know is `AUTH_CREDENTIAL_UNAVAILABLE`. The built-in agent's `fill_secret` goes through the same check, so its refusals read as the dispatcher's messages and not as `Secret fill refused: ...`. `ExecutorStep.Params` is JSON: a `Secret` is its name and, when set, its purpose, and every string and key that holds a secret value the attempt knows is redacted. `ExecutorAttempt` has the session's `TestTitle` as `TestId`, a fresh id per session, and `Index` as the attempt number minus one; `ExecutorStep.Index` counts the agent steps of the attempt, where upstream's counts every step. `AttachTranscript` and `AttachScreenshotAsync` write under the attempt's results directory (`transcript-.txt`, `screenshots/-