From 9b1cc480118d58bb27ab775684dfb0921eee477e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dar=C3=ADo=20Kondratiuk?= Date: Sun, 1 Nov 2026 10:22:31 -0300 Subject: [PATCH 1/2] Bound observe and screenshots by the operation budget (upstream #1464, in part) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1yaZFgZ3FEJkYScfkK --- COMPATIBILITY.md | 4 +- docs-site/src/content/docs/docs/index.mdx | 2 +- skills/e2e-dotnet/references/mobile.md | 2 +- src/E2E.Mobile/DeviceActions.cs | 4 +- src/E2E.Mobile/DeviceSession.Screen.cs | 87 ++++++++++--------- src/E2E.Mobile/DeviceSession.cs | 32 ++++--- src/E2E.Mobile/DeviceSupport.cs | 3 +- src/E2E.Mobile/MobileEngine.cs | 2 +- src/E2E/Agent/Agent.cs | 9 +- src/E2E/Engine/IEngine.cs | 19 +++- src/E2E/Expect/ScreenshotCapture.cs | 8 +- src/E2E/Locator/Screen.cs | 20 +++-- src/E2E/Run/E2ESession.cs | 5 +- src/E2E/Run/ScreenshotContext.cs | 2 +- .../Engine/ObservationTests.cs | 51 +++++++++++ 15 files changed, 172 insertions(+), 78 deletions(-) diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 4dc323303..b8c306ee6 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -35,7 +35,7 @@ Names are C# versions of the JavaScript API: `agent.act` is `ActAsync`, `screen. - Actions: tap, click (an alias of tap), doubleTap (`DblClickAsync` is an alias), fill (a string or a `Secret`), pressSequentially, press, check, uncheck, clear, selectOption, focus, scrollIntoView. They wait up to the action timeout for exactly one enabled, visible match. `selectOption` takes a string that matches an option value or label. `ActionOptions.Timeout` overrides the wait per action (`PressSequentiallyOptions` adds `Delay`). The web engine's own Playwright wait still uses the configured action timeout - Locator reads: `textContent`, `inputValue`, `getAttribute`, `isVisible`, `isHidden`, `isEnabled`, `isDisabled`, `isChecked`, `boundingBox`, `count`, `all`, `allTextContents`. A read of one node (`textContent`, `inputValue`, `getAttribute`, `isEnabled`, `isDisabled`, `isChecked`, `boundingBox`) waits for its one match until the action timeout, then reads once with no retry for a value; `isVisible`, `isHidden`, `count`, `all`, and `allTextContents` answer from the current screen at once. No read verifies an earlier `act`. `textContent` and `allTextContents` collapse each whitespace run to one space and trim, as upstream. A single-node read fails with `NOT_FOUND` when nothing matches before the action timeout ends - `screen.tree()` and `screen.debug()` (commit `d7ec0ec`) are `Screen.TreeAsync` and `Screen.DebugAsync`. One observation of the whole screen, rendered by `FailureEvidence.Listing` (the trace's screen at failure, upstream's `prepareObservation`): one node per line, children indented, secrets redacted, cut at the observation byte limit (262144). A node the page does not show is left out. A step of kind `screen`. `DebugAsync` writes the text to `Console.Out`, where a test framework picks it up. Differences: `Timeout` bounds the read with a linked token and fails with `OPERATION_TIMEOUT`, where upstream passes it to the engine operation. A `Locator` has no `TreeAsync`, so upstream's refusal on a locator is a compile error. On a `FrameScreen` they refuse with `INVALID_ARGUMENT`, as upstream's do. The same commit's change to print test output without the `list` reporter, and to keep stdout for `--reporter json`, is not ported: the port has no runner. `screen-tree.test.ts` is `ScreenTree/ScreenTreeTests` and `ScreenDebugTests` (not ported: the refusal on a locator, the unknown option, and `sdk-types.ts`'s checks, which a typed API makes compile errors); `screen-tree-secrets.test.ts` is `ScreenTreeSecrets/ScreenTreeTests` (through a session over a fake engine). `run-user-output.test.ts`'s two new cases are not ported with the runner change. -- The last read of a wait gets grace (`fix: give the last read of a wait the time of the slowest one`, `57f9ed9`, tester-army/e2e#1464). `Locator.WaitForAsync`, a locator `Expect`, and `ToHaveScreenshotAsync` read through a `ReadWindow` (`Internal/Time.cs`, upstream's `ReadWindow` in `internal/time.ts`; `Screen.Reads` is `LocatorEngine.reads`). Each engine locate runs until the wait's deadline plus the slowest read the wait finished (the budget `IEngineSession.LocateAsync` gets), capped by the test timeout (`Screen.BudgetDeadline`, upstream's `budget.deadline`), so a screen that shows up during a slow last read passes. `Time.PollReadsAsync` is `pollReads`, and `PollConditionAsync` is `pollCondition` over a window with no cap. A read the deadline cut off is `ReadWindow.CutOff`: an `OPERATION_TIMEOUT` of a read that started with under one poll tick, or under the slowest read, left (it replaces `cutOffAtDeadline`). A negation counts as held only until the deadline. `Locator.ResolveExactlyOneAsync` and the other waiting resolves end a wait whose last read was cut off after a read that found nothing as `NOT_FOUND`, not as `ACTION_FAILED`, and a retry the deadline cut off ends on the failure it was retrying (upstream's `resolveCounted` and `resolveOnce`). An action whose deadline passed before it was sent fails with `ACTION_FAILED` (upstream's `lateAction`). A capture of `ToHaveScreenshotAsync` that outlives its budget is an `OPERATION_TIMEOUT` the poll judges as it judges any read; the message of a failure with no capture is `the last screenshot ran past the timeout`. Differences: the port polls every 50 ms in the locator and 100 ms in screenshots where upstream polls every 100 ms. Only an engine that resolves locators itself (`CanLocate`) is handed the read's budget; the observation, the screenshot, and the web engine's reads take none, so the window only times them (a screenshot is cancelled at its budget). `Locator.WaitForAsync` keeps the port's `TIMEOUT` error where upstream's is `LOCATOR_NOT_FOUND`. Not ported: the positioned tap (`performAt`) `lateAction` check, which needs the locator engine's `performWithin`. Tests: `Time/PollConditionTests` (the five `pollReads` and `ReadWindow` its), `LocatorWaitFor/ASlowLastReadTests`, `LocatorEngine/LocatorEngineResolveRetryContractTests` (four its) and `LocatorEnginePerformContractTests` ('never sends an action whose target resolved past the deadline'), and `ScreenshotAssertions/ToHaveScreenshotTests` (the two capture-timeout its). +- The last read of a wait gets grace (`fix: give the last read of a wait the time of the slowest one`, `57f9ed9`, tester-army/e2e#1464). `Locator.WaitForAsync`, a locator `Expect`, and `ToHaveScreenshotAsync` read through a `ReadWindow` (`Internal/Time.cs`, upstream's `ReadWindow` in `internal/time.ts`; `Screen.Reads` is `LocatorEngine.reads`). Each engine locate runs until the wait's deadline plus the slowest read the wait finished (the budget `IEngineSession.LocateAsync` gets), capped by the test timeout (`Screen.BudgetDeadline`, upstream's `budget.deadline`), so a screen that shows up during a slow last read passes. `Time.PollReadsAsync` is `pollReads`, and `PollConditionAsync` is `pollCondition` over a window with no cap. A read the deadline cut off is `ReadWindow.CutOff`: an `OPERATION_TIMEOUT` of a read that started with under one poll tick, or under the slowest read, left (it replaces `cutOffAtDeadline`). A negation counts as held only until the deadline. `Locator.ResolveExactlyOneAsync` and the other waiting resolves end a wait whose last read was cut off after a read that found nothing as `NOT_FOUND`, not as `ACTION_FAILED`, and a retry the deadline cut off ends on the failure it was retrying (upstream's `resolveCounted` and `resolveOnce`). An action whose deadline passed before it was sent fails with `ACTION_FAILED` (upstream's `lateAction`). A capture of `ToHaveScreenshotAsync` that outlives its budget is an `OPERATION_TIMEOUT` the poll judges as it judges any read; the message of a failure with no capture is `the last screenshot ran past the timeout`. Differences: the port polls every 50 ms in the locator and 100 ms in screenshots where upstream polls every 100 ms. An engine is handed the read's budget by `IEngineSession.LocateAsync`, `ObserveAsync(TimeSpan, CancellationToken)`, and `ScreenshotAsync(TimeSpan, CancellationToken)` (see the operation budget bullet in the mobile section); an engine that does not override the last two (the web and WinApp engines) drops it, so the window only times its read (a screenshot is cancelled at its budget). `Locator.WaitForAsync` keeps the port's `TIMEOUT` error where upstream's is `LOCATOR_NOT_FOUND`. Not ported: the positioned tap (`performAt`) `lateAction` check, which needs the locator engine's `performWithin`. Tests: `Time/PollConditionTests` (the five `pollReads` and `ReadWindow` its), `LocatorWaitFor/ASlowLastReadTests`, `LocatorEngine/LocatorEngineResolveRetryContractTests` (four its) and `LocatorEnginePerformContractTests` ('never sends an action whose target resolved past the deadline'), and `ScreenshotAssertions/ToHaveScreenshotTests` (the two capture-timeout its). - `screen.scrollUntilVisible` and `locator.scrollUntilVisible` (`locator/screen.ts`) are `Screen.ScrollUntilVisibleAsync` and `Locator.ScrollUntilVisibleAsync`, with `ScrollUntilVisibleOptions { Direction, Momentum, Timeout }` (down, slow, 30 s by default). `Momentum` (`None`, `Slow`, `Fast`; upstream's `Momentum`) moved from `E2E.Mobile` to `E2E.Engine` beside `ScrollDirection`, and `LocatorAction.Swipe` carries it. `IEngineSession.SwipeAsync(direction, momentum, ct)` is the viewport swipe with a stride; its default drops the momentum. `WebEngine` strides as upstream's `swipeDistance` (none half, slow three quarters, fast one and a half of the extent; a swipe naming none keeps three quarters), `MobileEngine` strides a node swipe by it and keeps the device client's default for the whole-screen swipe, and `DocumentEngine` and `WinAppEngine` have one stride. The port has no operation context, so a swipe's deadline is the cancellation of the token it runs under, and a swipe the deadline ends fails with an `OPERATION_TIMEOUT` cause. A locator swipe's engine error is wrapped in `ACTION_FAILED`, as any locator action's is (see Error codes that differ). `filter({ has })` now refuses a locator of another screen with `INVALID_LOCATOR`, as upstream's `ownLocator` does - `locator.waitFor` with `attached`, `detached`, `visible` (default), and `hidden`, as `WaitForAsync(new LocatorWaitForOptions { State, Timeout })`. The timeout defaults to the action timeout. `LocatorWaitForOptions` is named apart from `agent.waitFor`'s `WaitForOptions`. Differences: a timeout fails with `TIMEOUT` (upstream `LOCATOR_NOT_FOUND`, `locator did not become `), and a passing `waitFor` does not verify an earlier `act` (upstream it does). The timeout carries `Details` as upstream's does (the locator, what it asked for, and how long it waited; see [Trace page](#trace-page)) - Locator expectations: visible, hidden, attached, enabled, disabled, checked, selected, expanded, focused, text, contained text, value, attribute, accessible name, and count. They poll until the assertion timeout, or the matcher's own `timeout`. `Not` inverts a matcher, which then passes after 1000 ms of continuous truth (or the whole budget when it is shorter). Text matchers take the same `TextMatch` as queries, `ignoreCase`, and a list form. The boolean flags are `visible`, `attached`, `enabled`, and `isChecked` (`checked` is a C# keyword) @@ -304,7 +304,7 @@ The port installs the skill with `npx skills add hardkoded/e2e-dotnet` and regis - `DeviceSession.BackAsync` always settles (upstream's agent origin), since a test step has no origin here. - The core session has no attempt hooks, pixel observations, or artifact directory. So `DeviceSession` is one attempt (it registers on the engine until it closes), and `ScreenshotAsync` returns the masked, status-bar-normalized PNG for `toHaveScreenshot` instead of writing numbered files. A secure field that cannot be masked still withholds the image with `ENGINE_FAILURE`. - `DeviceSession.LocateAsync(LocatorExpression, TimeSpan timeout, CancellationToken)` is upstream's `locate` (`surface.ts`), so a locator or `Expect` on a device resolves in the engine. It snapshots with `interactiveOnly: false`, runs `DeviceLocator.ResolveExpression`, and binds the matches in a separate map capped at 2048 (the oldest goes first; an older ref is `NODE_STALE`). It hands the capture it matched against to `EngineStartOptions.Screen`, and tells `EngineStartOptions.Environment` the platform and device once, when the session starts. Differences: `timeout` is the budget left (upstream's `operation.timeoutMs`), since the core session has no operation context; the expression is one flat record (`LocatorExpression`) where upstream's is a union of `query`, `filter`, `index`, `union`, `selector`, and `frame`; and the port's refs carry no revision, so upstream's `rejectSupersededLocate` has no port (a device mints a new id for every projection, so a later locate never reuses a ref). Core retries a retryable `EngineException` of a locate until the wait's deadline, as `resolveOnce` does. -- Upstream's operation budget for reads (`fix(mobile): bound reads by the operation budget`, `57f9ed9`, tester-army/e2e#1464) is ported for the snapshots a locate and a keyboard dismissal take. `InputBudget.ReadAsync` (upstream's `read`) runs a snapshot with what is left of the budget, and `InputBudget.Remaining` is `remainingMs`. A locate runs under a budget of its own `timeout` (`DeviceSession.InputBudget(label, timeout)`, upstream's `inputBudget(label, timeoutMs)`), so a device that stops answering fails the locate with `OPERATION_TIMEOUT`, which a locator reports as `ACTION_FAILED`, where it held the step until the test timeout. A sparse capture's retry the budget cannot fit returns the sparse capture, and a capture the budget abandoned is not taken again. A relocation before an action, and the snapshots of the keyboard dismissal, run under the input's budget. `device.dismissKeyboard` now has a budget too (`actionTimeout`), as upstream's has. Differences: `DeviceSession.ObserveAsync`, the screenshots (`ScreenshotAsync`, the viewport probe, `ScreenshotArtifactAsync`), and the masked screenshot take no timeout in the core session, so they stay unbounded; upstream bounds them by `operation.timeoutMs` (the `bounds screenshot by operation budget`, `rethrows OPERATION_TIMEOUT from capturePixels during observe`, and `shares one budget across observe steps` its are not ported). The `InputBudget` parameter of `SnapshotAsync` and `SnapshotOrEmptyAsync` is nullable for that reason, where upstream's is required. Tests: in `Engine/ObservationTests`, 'bounds a snapshot by the operation budget', 'returns a sparse capture when the budget cannot fit a retry, instead of timing out', 'bounds the snapshots of a keyboard dismissal by its budget, and reports one cut off after the drag as possibly committed', 'lets a slow snapshot run for the whole timeout the locate gets', and 'takes no second snapshot for a capture the budget abandoned'; in `Support/SupportHelpersTests`, 'runs a read with what is left of its budget, and reports one cut off after an input as possibly committed'. These run on real timers of a few hundred milliseconds. +- Upstream's operation budget for reads (`fix(mobile): bound reads by the operation budget`, `57f9ed9`, tester-army/e2e#1464) is ported for the snapshots a locate and a keyboard dismissal take. `InputBudget.ReadAsync` (upstream's `read`) runs a snapshot with what is left of the budget, and `InputBudget.Remaining` is `remainingMs`. A locate runs under a budget of its own `timeout` (`DeviceSession.InputBudget(label, timeout)`, upstream's `inputBudget(label, timeoutMs)`), so a device that stops answering fails the locate with `OPERATION_TIMEOUT`, which a locator reports as `ACTION_FAILED`, where it held the step until the test timeout. A sparse capture's retry the budget cannot fit returns the sparse capture, and a capture the budget abandoned is not taken again. A relocation before an action, and the snapshots of the keyboard dismissal, run under the input's budget. `device.dismissKeyboard` now has a budget too (`actionTimeout`), as upstream's has. The observation and the screenshots are bounded too (#892). `IEngineSession.ObserveAsync(TimeSpan timeout, CancellationToken)` and `ScreenshotAsync(TimeSpan timeout, CancellationToken)` carry upstream's `operation.timeoutMs` (a public API change, like `LocateAsync`'s `timeout`): they have a default that drops the budget and calls the overload without it, so an engine that bounds nothing keeps compiling (.NET-only). `DeviceSession` overrides them. `ObserveAsync` runs its snapshot and its viewport probe under one `InputBudget('observe')`, `ScreenshotAsync` its snapshot and its capture under one `InputBudget('screenshot')`, so a device that stops answering fails with `OPERATION_TIMEOUT` and the screenshot cannot take a fresh budget. The core callers pass what they have: `Screen.TreeAsync` its timeout, a locator resolve over an observation and `ToHaveScreenshotAsync` the budget of the read window. Calls with no timeout (the agent's looks, failure evidence, `ObserveAsync(CancellationToken)`) get `ActionTimeout`. `IDeviceActions.ScreenshotAsync` takes the timeout too. `ScreenshotArtifactAsync` runs under `ActionTimeout`, where upstream's `artifacts.screenshot` takes `operation.timeoutMs`. A read the caller cancels is `CANCELLED` (`InputBudget.ReadAsync`), as the command reports it. The `InputBudget` parameter of `SnapshotAsync` and `SnapshotOrEmptyAsync` stays nullable, where upstream's is required. The port's observe takes no `pixels`, so upstream's `capturePixels` has no port: the screenshot an observation takes is the viewport probe, which rethrows `OPERATION_TIMEOUT` too. Tests, in `Engine/ObservationTests`: 'Partial_Bounds_screenshot_by_operation_budget' (the port's `ScreenshotAsync`, not `artifacts.screenshot`), 'Partial_Rethrows_OPERATION_TIMEOUT_from_capturePixels_during_observe' and 'Partial_Shares_one_budget_across_observe_steps_so_screenshot_cannot_take_a_fresh_budget' (the viewport probe, not the observation's pixels). Not ported: the positioned tap (`performAt`) `lateAction` check ('never sends a positioned tap whose target became visible past the deadline'), which needs `performWithin` (#861). Tests: in `Engine/ObservationTests`, 'bounds a snapshot by the operation budget', 'returns a sparse capture when the budget cannot fit a retry, instead of timing out', 'bounds the snapshots of a keyboard dismissal by its budget, and reports one cut off after the drag as possibly committed', 'lets a slow snapshot run for the whole timeout the locate gets', and 'takes no second snapshot for a capture the budget abandoned'; in `Support/SupportHelpersTests`, 'runs a read with what is left of its budget, and reports one cut off after an input as possibly committed'. These run on real timers of a few hundred milliseconds. - Upstream's Android locate (`fix(mobile): keep locator polling through unreadable android snapshots`, `4fe6baf`) is ported. A snapshot agent-device cannot read yet (`details.androidSnapshotHelperFailureReason` of `empty-helper-output`, `system-window-only`, or `content-poor-app-window`, the message `Android snapshot helper returned insufficient foreground app content`) matches nothing in a locate, so a locator or `Expect` keeps polling until its wait runs out, and a locate whose capture leaves no budget fails with `ENGINE_FAILURE` and that message. `DeviceErrors.IsUnreadableCapture` reads the reason off the `AgentDeviceError` that the translated `EngineException` keeps as its `InnerException`. An observation, such as the agent's next turn, still fails on it at once, as upstream's. - Upstream's iOS locate (`2b2b845`, tester-army/e2e#1281) is ported. Just after `OpenAppAsync`, `RestartAsync` or `OpenLinkAsync`, a snapshot that iOS refuses with `details.runnerErrorCode` of `APP_NOT_RUNNING` (`DeviceErrors.IsAppNotRunning`) matches nothing in a locate (locator waits and `Expect` only; an observation still fails at once), like an unreadable Android capture, until a snapshot succeeds (`DeviceSession._launching`, upstream's `launching`). A wait that runs out while the app is still down fails with `ENGINE_FAILURE` and that reason. An app that exits after its first readable screen still fails at once. - Upstream's Android test id match without the `:id/` prefix (`feat(mobile): match Android test ids without their package prefix`, `2791786`) is in `DeviceLocator`, which `DeviceSession.LocateAsync` now calls, so `Screen.GetByTestId("greet")` matches `:id/greet` on Android when nothing matches the full id. diff --git a/docs-site/src/content/docs/docs/index.mdx b/docs-site/src/content/docs/docs/index.mdx index a8ed06cf7..7279f6611 100644 --- a/docs-site/src/content/docs/docs/index.mdx +++ b/docs-site/src/content/docs/docs/index.mdx @@ -94,4 +94,4 @@ await Screen.GetByTestId("feed").ScrollUntilVisibleAsync(row, new ScrollUntilVis A locator reports an engine failure in the test's own codes, with the engine's error as `InnerException`: `NOT_ACTIONABLE`, `ACTION_MAY_HAVE_COMMITTED`, and `OPERATION_TIMEOUT` are `ACTION_FAILED`, a stale node or a missing frame is `NOT_FOUND`, an ambiguous frame is `STRICT_MODE`, and `INVALID_STATE` is `APP_NOT_OPEN`. `CANCELLED` and `ENGINE_FAILURE` keep their codes (category `infrastructure`), and `UNSUPPORTED_CAPABILITY` is a `ConfigurationException`. A stale node found by an action is looked up again, within the action timeout, before the action fails. -A read or navigation that times out fails with `OPERATION_TIMEOUT`. It is not retried, even when the call log mentions a detached element or a navigation race. On a device, an input (a tap, key, swipe, or `Back`) that the device does not answer within `actionTimeout` fails with `ACTION_MAY_HAVE_COMMITTED` and is never repeated, because it may have landed; an input the timeout ran out before is not sent (`OPERATION_TIMEOUT`), and a long press that holds at least as long as the time left is refused (`NOT_ACTIONABLE`). A locate on a device runs within the timeout of the call that takes it, so a device that stops answering fails the step with `ACTION_FAILED` instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. See `EngineErrorCodes` in the [API reference](/api/). +A read or navigation that times out fails with `OPERATION_TIMEOUT`. It is not retried, even when the call log mentions a detached element or a navigation race. On a device, an input (a tap, key, swipe, or `Back`) that the device does not answer within `actionTimeout` fails with `ACTION_MAY_HAVE_COMMITTED` and is never repeated, because it may have landed; an input the timeout ran out before is not sent (`OPERATION_TIMEOUT`), and a long press that holds at least as long as the time left is refused (`NOT_ACTIONABLE`). A locate, an observation, or a screenshot on a device runs within the timeout of the call that takes it, so a device that stops answering fails the step with `ACTION_FAILED` instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. See `EngineErrorCodes` in the [API reference](/api/). diff --git a/skills/e2e-dotnet/references/mobile.md b/skills/e2e-dotnet/references/mobile.md index fd1f8214d..a36185ad1 100644 --- a/skills/e2e-dotnet/references/mobile.md +++ b/skills/e2e-dotnet/references/mobile.md @@ -175,6 +175,6 @@ Limits of the preview: the tests attach to the running app, so `Device.InstallAp - Building the sample iOS app needs `-p:ValidateXcodeVersion=false` when the installed Xcode is newer than the one the .NET iOS pack expects. - Agent steps on a device are slower than in a browser. Raise `StepTimeout` to 90 seconds. The 30 second default timed out a two-field step. - A ref belongs to the agent-device capture it came from, and every action pins it there (`@e4~s7`), so agent-device refuses a ref once a newer capture supersedes it. An action on a node that a newer capture replaced never acts on a different element: a test's action waits out `TransitionMilliseconds` for a control that arrived with the last action, and a node the fresh capture no longer lists is a retryable `NODE_STALE`, so the locator resolves again within its timeout. A direct `Device` call (`DragToAsync`, `LongPressAsync`) does not retry and surfaces it. `Device.DragToAsync` between nodes located in separate captures takes both refs from the newest one. -- An input that outlives its timeout. A tap, fill, key, swipe, or `Back` gets the `actionTimeout` (30 seconds by default). If the device doesn't answer in time, the step fails with `ACTION_MAY_HAVE_COMMITTED` and isn't retried: the input may have landed. An input the timeout ran out before (its target was still settling) isn't sent and fails with `OPERATION_TIMEOUT`. A long press that holds at least as long as the time left is refused before it's sent, as `NOT_ACTIONABLE`: raise the `actionTimeout`. A locate (the snapshot a locator or `Expect` takes) runs within the timeout of the call that takes it, so a device that stops answering fails the step with `ACTION_FAILED` (the engine's `OPERATION_TIMEOUT` as its `InnerException`) instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. An element that never shows up still fails as a locator timeout. A command a timeout or a cancel abandoned keeps running on the device, so the next attempt on that device waits for it to settle before it starts (within the launch timeout, which fails the start as cancelled when it runs out). +- An input that outlives its timeout. A tap, fill, key, swipe, or `Back` gets the `actionTimeout` (30 seconds by default). If the device doesn't answer in time, the step fails with `ACTION_MAY_HAVE_COMMITTED` and isn't retried: the input may have landed. An input the timeout ran out before (its target was still settling) isn't sent and fails with `OPERATION_TIMEOUT`. A long press that holds at least as long as the time left is refused before it's sent, as `NOT_ACTIONABLE`: raise the `actionTimeout`. A locate (the snapshot a locator or `Expect` takes), an observation, and a screenshot run within the timeout of the call that takes them, so a device that stops answering fails the step with `ACTION_FAILED` (the engine's `OPERATION_TIMEOUT` as its `InnerException`) instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. An element that never shows up still fails as a locator timeout. A command a timeout or a cancel abandoned keeps running on the device, so the next attempt on that device waits for it to settle before it starts (within the launch timeout, which fails the start as cancelled when it runs out). - Device tests need a device. Keep them out of CI, for example with the `RealModel` category (`dotnet test --filter "TestCategory!=RealModel"`). - Android can report `Android snapshot helper returned insufficient foreground app content`, and iOS can report that the app is not running: agent-device could not read the screen yet, often an app still loading after a cold start. On iOS this only counts until the first readable screen after `OpenAppAsync`, `RestartAsync`, or `OpenLinkAsync`, as on a physical iPhone that is still launching the app; an app that exits later fails at once. A waiting locator action or `Expect` waits through it, as for an element not there yet. If the screen is still unreadable when the wait runs out, the step fails with this message. Immediate reads such as `CountAsync` and `IsVisibleAsync` see no matches instead, so `Not.ToBeVisibleAsync` can pass while the screen is unreadable, including on an app that crashed before it rendered. An observation, such as the agent's next turn, fails on it at once. diff --git a/src/E2E.Mobile/DeviceActions.cs b/src/E2E.Mobile/DeviceActions.cs index 1273db0ed..ec257c138 100644 --- a/src/E2E.Mobile/DeviceActions.cs +++ b/src/E2E.Mobile/DeviceActions.cs @@ -314,10 +314,10 @@ public Task SwipeAsync(DeviceSession session, ScrollDirection direction, Cancell return ScrollScreenAsync(session, direction, session.InputBudget("swipe"), cancellationToken); } - public Task ScreenshotAsync(DeviceSession session, CancellationToken cancellationToken) + public Task ScreenshotAsync(DeviceSession session, TimeSpan timeout, CancellationToken cancellationToken) { ArgumentNullException.ThrowIfNull(session); - return session.MaskedScreenshotAsync(cancellationToken); + return session.MaskedScreenshotAsync(session.InputBudget("screenshot", timeout), cancellationToken); } // --- what the contract has no verb for --- diff --git a/src/E2E.Mobile/DeviceSession.Screen.cs b/src/E2E.Mobile/DeviceSession.Screen.cs index 75388662c..76d52fea4 100644 --- a/src/E2E.Mobile/DeviceSession.Screen.cs +++ b/src/E2E.Mobile/DeviceSession.Screen.cs @@ -503,7 +503,7 @@ private async Task DismissKeyboardByGestureAsync(InputBudget input, Cancellation foreach (var horizontal in new[] { true, false }) { var projected = SnapshotProjection.ProjectSnapshot(raw.Nodes, () => "keyboard", raw.Viewport); - var viewport = await ViewportForAsync(projected, cancellationToken).ConfigureAwait(false); + var viewport = await ViewportForAsync(projected, input, cancellationToken).ConfigureAwait(false); var from = Keyboard.KeyboardDragStart(projected, viewport, KeyboardTop(raw, viewport)) ?? throw new EngineException( EngineErrorCodes.UnsupportedCapability, @@ -564,60 +564,61 @@ await CommandAsync( /// One screenshot into a temp directory that is removed once has taken what it needs from /// the response or the file, even when the caller abandons the capture. /// - private Task CaptureScreenshotAsync(Func> read, bool normalizeStatusBar, CancellationToken cancellationToken) => - CommandAsync( - "screenshot", - async (client, token) => - { - var directory = Directory.CreateTempSubdirectory("e2e-agent-device-"); - try - { - var file = Path.Combine(directory.FullName, "screenshot.png"); - var shot = await client.CallAsync( - DeviceMethods.CaptureScreenshot, - DeviceArgs.Of(("path", JsonValue.Create(file)), ("normalizeStatusBar", normalizeStatusBar ? JsonValue.Create(true) : null)), - token).ConfigureAwait(false); - return await read( - new RawScreenshotResult - { - Path = Text(shot?["path"]), - Width = Number(shot?["width"]), - Height = Number(shot?["height"]), - LogicalWidth = Number(shot?["logicalWidth"]), - LogicalHeight = Number(shot?["logicalHeight"]), - PixelDensity = Number(shot?["pixelDensity"]), - }, - file).ConfigureAwait(false); - } - finally - { - try - { - directory.Delete(recursive: true); - } - catch (IOException) - { - // The capture is best effort and so is its cleanup. - } - } - }, + private Task CaptureScreenshotAsync(Func> read, bool normalizeStatusBar, InputBudget budget, CancellationToken cancellationToken) => + budget.ReadAsync( + () => CommandAsync("screenshot", (client, token) => TakeScreenshotAsync(client, read, normalizeStatusBar, token), cancellationToken), cancellationToken); + private static async Task TakeScreenshotAsync(IDeviceClient client, Func> read, bool normalizeStatusBar, CancellationToken token) + { + var directory = Directory.CreateTempSubdirectory("e2e-agent-device-"); + try + { + var file = Path.Combine(directory.FullName, "screenshot.png"); + var shot = await client.CallAsync( + DeviceMethods.CaptureScreenshot, + DeviceArgs.Of(("path", JsonValue.Create(file)), ("normalizeStatusBar", normalizeStatusBar ? JsonValue.Create(true) : null)), + token).ConfigureAwait(false); + return await read( + new RawScreenshotResult + { + Path = Text(shot?["path"]), + Width = Number(shot?["width"]), + Height = Number(shot?["height"]), + LogicalWidth = Number(shot?["logicalWidth"]), + LogicalHeight = Number(shot?["logicalHeight"]), + PixelDensity = Number(shot?["pixelDensity"]), + }, + file).ConfigureAwait(false); + } + finally + { + try + { + directory.Delete(recursive: true); + } + catch (IOException) + { + // The capture is best effort and so is its cleanup. + } + } + } + /// /// The viewport as a screenshot for comparison: the status bar in a fixed state, and every secure node on the screen /// painted over. Observes first so the regions describe the screen the pixels show. Throws when a secure field /// cannot be covered, because an image that may hold a credential must not leave the engine. /// - internal Task MaskedScreenshotAsync(CancellationToken cancellationToken) => MaskedScreenshotAsync(normalizeStatusBar: true, cancellationToken); + internal Task MaskedScreenshotAsync(InputBudget budget, CancellationToken cancellationToken) => MaskedScreenshotAsync(normalizeStatusBar: true, budget, cancellationToken); /// - /// with the status bar left as the device shows it when + /// with the status bar left as the device shows it when /// is false, as an evidence screenshot is kept (upstream's maskedScreenshot). /// - internal async Task MaskedScreenshotAsync(bool normalizeStatusBar, CancellationToken cancellationToken) + internal async Task MaskedScreenshotAsync(bool normalizeStatusBar, InputBudget budget, CancellationToken cancellationToken) { - var projected = Project(await SnapshotOrEmptyAsync(false, budget: null, cancellationToken).ConfigureAwait(false)); - var data = await CaptureScreenshotAsync(async (shot, file) => await File.ReadAllBytesAsync(shot.Path ?? file, cancellationToken).ConfigureAwait(false), normalizeStatusBar, cancellationToken).ConfigureAwait(false); + var projected = Project(await SnapshotOrEmptyAsync(false, budget, cancellationToken).ConfigureAwait(false)); + var data = await CaptureScreenshotAsync(async (shot, file) => await File.ReadAllBytesAsync(shot.Path ?? file, cancellationToken).ConfigureAwait(false), normalizeStatusBar, budget, cancellationToken).ConfigureAwait(false); var viewport = _knownViewport; var (masked, regions) = RedactSecure(data, projected, viewport) ?? throw new EngineException(EngineErrorCodes.EngineFailure, "a secure field on screen could not be masked; screenshot withheld", retryable: false); diff --git a/src/E2E.Mobile/DeviceSession.cs b/src/E2E.Mobile/DeviceSession.cs index 6df60d0f7..6a7aec441 100644 --- a/src/E2E.Mobile/DeviceSession.cs +++ b/src/E2E.Mobile/DeviceSession.cs @@ -296,12 +296,19 @@ await CommandAsync( public bool CanLocate => true; - public async Task ObserveAsync(CancellationToken cancellationToken) + public Task ObserveAsync(CancellationToken cancellationToken) => ObserveAsync(ActionTimeout, cancellationToken); + + /// + /// Observes the screen within , the operation budget that bounds every device command of the observation: + /// the snapshot and the viewport probe share it (upstream's observe under inputBudget('observe', operation.timeoutMs)). + /// + public async Task ObserveAsync(TimeSpan timeout, CancellationToken cancellationToken) { - var raw = await SnapshotOrEmptyAsync(Options.Snapshot == SnapshotMode.Interactive, budget: null, cancellationToken).ConfigureAwait(false); + var budget = InputBudget("observe", timeout); + var raw = await SnapshotOrEmptyAsync(Options.Snapshot == SnapshotMode.Interactive, budget, cancellationToken).ConfigureAwait(false); var projected = Project(raw); _generation = projected.Index.ToDictionary(entry => entry.Id, entry => Bind(entry, projected.Index, projected.RefsGeneration)); - var viewport = await ViewportForAsync(projected, cancellationToken).ConfigureAwait(false); + var viewport = await ViewportForAsync(projected, budget, cancellationToken).ConfigureAwait(false); Route = DeviceSupport.ScreenLocation(raw.AppBundleId ?? raw.AppName ?? _appIdentity, SnapshotProjection.ScreenTitle(projected)) ?? ""; return new Observation { @@ -322,8 +329,11 @@ public Task PressAsync(string key, CancellationToken cancellationToken) => public Task SwipeAsync(ScrollDirection direction, CancellationToken cancellationToken) => _engine.Actions.SwipeAsync(this, direction, cancellationToken); - public Task ScreenshotAsync(CancellationToken cancellationToken) => - _engine.Actions.ScreenshotAsync(this, cancellationToken); + public Task ScreenshotAsync(CancellationToken cancellationToken) => ScreenshotAsync(ActionTimeout, cancellationToken); + + /// The masked screenshot within , the operation budget its snapshot and capture share. + public Task ScreenshotAsync(TimeSpan timeout, CancellationToken cancellationToken) => + _engine.Actions.ScreenshotAsync(this, timeout, cancellationToken); /// /// A redacted screenshot artifact. The device paints secure fields as dots, but the last typed character shows in clear, so every @@ -333,7 +343,7 @@ public Task ScreenshotAsync(CancellationToken cancellationToke public async Task ScreenshotArtifactAsync(string? label, CancellationToken cancellationToken) { var artifactsDir = _artifactsDir ?? throw DeviceSupport.InvalidState("screenshot outside an attempt"); - var masked = await MaskedScreenshotAsync(normalizeStatusBar: false, cancellationToken).ConfigureAwait(false); + var masked = await MaskedScreenshotAsync(normalizeStatusBar: false, InputBudget("screenshot"), cancellationToken).ConfigureAwait(false); var name = Interlocked.Increment(ref _screenshots).ToString("D3", System.Globalization.CultureInfo.InvariantCulture) + "-" + DeviceSupport.SanitizeFilename(label ?? "screenshot") + ".png"; var directory = Path.Combine(artifactsDir, "screenshots"); @@ -515,7 +525,7 @@ private ProjectedSnapshot Project(RawSnapshot raw) /// size; on a real device the tree's size is taken as it comes. A device whose size cannot be learned at all fails the /// observation: the harness measures every rect and point against the viewport, so an invented one would misplace every tap. /// - private async Task ViewportForAsync(ProjectedSnapshot projected, CancellationToken cancellationToken) + private async Task ViewportForAsync(ProjectedSnapshot projected, InputBudget budget, CancellationToken cancellationToken) { if (_knownViewport is { } known) { @@ -525,7 +535,7 @@ private async Task ViewportForAsync(ProjectedSnapshot projected, C } if (!SameScreen(read, known) - && await ProbeViewportAsync(cancellationToken).ConfigureAwait(false) is { } device + && await ProbeViewportAsync(budget, cancellationToken).ConfigureAwait(false) is { } device && !SameScreen(read, device)) { return known; @@ -535,7 +545,7 @@ private async Task ViewportForAsync(ProjectedSnapshot projected, C return read; } - var probed = await ProbeViewportAsync(cancellationToken).ConfigureAwait(false) + var probed = await ProbeViewportAsync(budget, cancellationToken).ConfigureAwait(false) ?? throw new EngineException( EngineErrorCodes.EngineFailure, "the device reported no screen geometry: the snapshot has no bounds and the screenshot no logical size, so the viewport is unknown", @@ -545,8 +555,8 @@ private async Task ViewportForAsync(ProjectedSnapshot projected, C } /// The device's logical screen size as its screenshot reports it. Null when it reports none. - private Task ProbeViewportAsync(CancellationToken cancellationToken) => - CaptureScreenshotAsync((shot, _) => Task.FromResult(DeviceSupport.LogicalScreenSize(shot)), normalizeStatusBar: false, cancellationToken); + private Task ProbeViewportAsync(InputBudget budget, CancellationToken cancellationToken) => + CaptureScreenshotAsync((shot, _) => Task.FromResult(DeviceSupport.LogicalScreenSize(shot)), normalizeStatusBar: false, budget, cancellationToken); private const string UninstalledBuild = "the build the target's `app.appPath` names installed first with `device.installApp()`"; diff --git a/src/E2E.Mobile/DeviceSupport.cs b/src/E2E.Mobile/DeviceSupport.cs index c8bc73674..2fc630680 100644 --- a/src/E2E.Mobile/DeviceSupport.cs +++ b/src/E2E.Mobile/DeviceSupport.cs @@ -234,7 +234,8 @@ public async Task ReadAsync(Func> dispatch, CancellationToken canc throw _sent ? MaybeCommitted() : new EngineException(EngineErrorCodes.OperationTimeout, $"{label} timed out before the device answered", retryable: false); } - throw; + // A read the caller cancelled is a cancel, as the command itself reports it. + throw new EngineException(EngineErrorCodes.Cancelled, $"{label} cancelled", retryable: false, ex); } } diff --git a/src/E2E.Mobile/MobileEngine.cs b/src/E2E.Mobile/MobileEngine.cs index efb9eb296..c5bc69bb7 100644 --- a/src/E2E.Mobile/MobileEngine.cs +++ b/src/E2E.Mobile/MobileEngine.cs @@ -19,7 +19,7 @@ public interface IDeviceActions Task SwipeAsync(DeviceSession session, ScrollDirection direction, CancellationToken cancellationToken); - Task ScreenshotAsync(DeviceSession session, CancellationToken cancellationToken); + Task ScreenshotAsync(DeviceSession session, TimeSpan timeout, CancellationToken cancellationToken); } /// What a worker's start is given: which target and worker slot it serves, and the environment prepare left. diff --git a/src/E2E/Agent/Agent.cs b/src/E2E/Agent/Agent.cs index 944768fa0..be725c5ce 100644 --- a/src/E2E/Agent/Agent.cs +++ b/src/E2E/Agent/Agent.cs @@ -1247,9 +1247,14 @@ public async Task Track(Task call) public ScreenshotContext? Screenshots { get; set; } /// One look at the screen, shown to the trace. Upstream's session.observe shows each observation it takes. - public async Task ObserveAsync(CancellationToken token) + public Task ObserveAsync(CancellationToken token) => ShowAsync(Session.ObserveAsync(token)); + + /// within , the operation budget the engine bounds its reads by. + public Task ObserveAsync(TimeSpan timeout, CancellationToken token) => ShowAsync(Session.ObserveAsync(timeout, token)); + + private async Task ShowAsync(Task look) { - var observation = await Session.ObserveAsync(token).ConfigureAwait(false); + var observation = await look.ConfigureAwait(false); // The adapter mints the revision, as upstream's session does: one per capture the attempt took, in order. observation.Revision = "b" + Interlocked.Increment(ref _observed).ToString(CultureInfo.InvariantCulture); diff --git a/src/E2E/Engine/IEngine.cs b/src/E2E/Engine/IEngine.cs index 4d8bf5cbe..403a7f330 100644 --- a/src/E2E/Engine/IEngine.cs +++ b/src/E2E/Engine/IEngine.cs @@ -20,7 +20,7 @@ public enum EngineCapabilities /// . History = 32, - /// , which toHaveScreenshot compares. + /// , which toHaveScreenshot compares. Screenshot = 64, /// The Modifiers of and : keys held for the click. @@ -191,6 +191,14 @@ public interface IEngineSession : IAsyncDisposable Task ObserveAsync(CancellationToken cancellationToken); + /// + /// within , the operation budget left for the read: upstream's + /// observe(operation), whose operation.timeoutMs bounds every device command of the observation. An engine that bounds + /// its reads overrides it and ends the read at the budget with OPERATION_TIMEOUT; the default drops the budget and observes + /// as does (.NET-only: it keeps the engines that bound nothing compiling). + /// + Task ObserveAsync(TimeSpan timeout, CancellationToken cancellationToken) => ObserveAsync(cancellationToken); + Task PerformAsync(SemanticNode node, LocatorAction action, CancellationToken cancellationToken); /// Sends a key to whatever holds focus. The harness has already validated the key. @@ -198,7 +206,7 @@ public interface IEngineSession : IAsyncDisposable /// /// Whether is implemented. A session that says false has its locators resolved in core over - /// . .NET-only and transitional: upstream's engines all locate, and this goes once the web and document + /// . .NET-only and transitional: upstream's engines all locate, and this goes once the web and document /// engines do. /// bool CanLocate => false; @@ -247,6 +255,13 @@ Task> LocateAsync(LocatorExpression expression, Time Task ScreenshotAsync(CancellationToken cancellationToken) => Task.FromException(new EngineException("UNSUPPORTED_CAPABILITY", "toHaveScreenshot is not supported by this engine.")); + /// + /// within , the operation budget left for the capture + /// (upstream's operation.timeoutMs). An engine that bounds its reads overrides it and ends the capture at the budget with + /// OPERATION_TIMEOUT; the default drops the budget (.NET-only, as ). + /// + Task ScreenshotAsync(TimeSpan timeout, CancellationToken cancellationToken) => ScreenshotAsync(cancellationToken); + /// /// Captures a redacted screenshot into the attempt's artifact directory and returns its artifact-relative path, screenshots/NNN-label.png. /// Every secure field is covered before the file is written. Upstream's session.artifacts.screenshot; an engine that keeps none diff --git a/src/E2E/Expect/ScreenshotCapture.cs b/src/E2E/Expect/ScreenshotCapture.cs index 4a8abcf6a..a0bd7880e 100644 --- a/src/E2E/Expect/ScreenshotCapture.cs +++ b/src/E2E/Expect/ScreenshotCapture.cs @@ -27,17 +27,17 @@ private sealed class Capturer(Screen screen, ScreenshotContext store, string api private bool _scrolled = !store.ScrollsIntoView; /// - /// One engine read on the window's budget. The port's observe and capture take no budget, so the read ends with it: the token - /// cancels at the budget, and a read that ends on that is the engine's OPERATION_TIMEOUT, reported as ACTION_FAILED. + /// One engine read on the window's budget, which the read is given and the token also ends: a read that ends on that is the + /// engine's OPERATION_TIMEOUT, reported as ACTION_FAILED. /// - private Task ReadAsync(Func> read, CancellationToken cancellationToken) => + private Task ReadAsync(Func> read, CancellationToken cancellationToken) => reads.ReadAsync(async budget => { using var bounded = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); bounded.CancelAfter(budget); try { - return await read(bounded.Token).ConfigureAwait(false); + return await read(budget, bounded.Token).ConfigureAwait(false); } catch (Exception ex) when (bounded.IsCancellationRequested && !cancellationToken.IsCancellationRequested && (ex is OperationCanceledException || ex is EngineException { Code: EngineErrorCodes.OperationTimeout or EngineErrorCodes.Cancelled } || ex.InnerException is OperationCanceledException)) diff --git a/src/E2E/Locator/Screen.cs b/src/E2E/Locator/Screen.cs index 29307a8f6..9a1d5ac7d 100644 --- a/src/E2E/Locator/Screen.cs +++ b/src/E2E/Locator/Screen.cs @@ -16,6 +16,7 @@ namespace E2E; public class Screen { private readonly Func> _observe; + private readonly Func>? _observeWithin; private readonly Func _perform; private readonly Func>>? _locate; private readonly Func _cancellation; @@ -43,9 +44,11 @@ internal Screen( SoftFailures? softFailures = null, TimeProvider? clock = null, Func>>? locate = null, - Func? swipe = null) + Func? swipe = null, + Func>? observeWithin = null) { _swipe = swipe; + _observeWithin = observeWithin; SoftFailures = softFailures ?? new SoftFailures(); _observe = observe; _perform = perform; @@ -67,6 +70,7 @@ internal Screen(Screen screen, Func wrap) _scopeOf = root; _wrap = wrap; _observe = root._observe; + _observeWithin = root._observeWithin; _perform = root._perform; _locate = root._locate; _cancellation = root._cancellation; @@ -323,7 +327,7 @@ async Task Read() try { // The wait ends with the timeout even when the engine ignores cancellation; the abandoned read's fault is observed. - var read = ObserveAsync(bounded.Token); + var read = ObserveAsync(timeout == System.Threading.Timeout.InfiniteTimeSpan ? ActionTimeout : timeout, bounded.Token); _ = read.ContinueWith(static finished => _ = finished.Exception, CancellationToken.None, TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default); var observation = await read.WaitAsync(bounded.Token).ConfigureAwait(false); return RenderTree?.Invoke(observation) ?? Redact(TreeText(observation, Redactor.None, null, E2EDefaults.MaxObservationBytes)); @@ -342,9 +346,16 @@ internal async Task ObserveAsync(CancellationToken cancellationToke return await AppNotOpen.GuardAsync(() => _observe(cancellationToken == default ? _cancellation() : cancellationToken)).ConfigureAwait(false); } + /// within , the budget the engine bounds its reads by (upstream's operation.timeoutMs). + internal async Task ObserveAsync(TimeSpan timeout, CancellationToken cancellationToken) + { + var token = cancellationToken == default ? _cancellation() : cancellationToken; + return await AppNotOpen.GuardAsync(() => _observeWithin is { } within ? within(timeout, token) : _observe(token)).ConfigureAwait(false); + } + /// /// Whether the engine resolves a locator itself. Upstream's engines all do; a port engine that does not has its locator resolved - /// here over , until it implements (.NET-only). + /// here over , until it implements (.NET-only). /// internal bool LocatesInEngine => _locate is not null; @@ -871,8 +882,7 @@ internal async Task> ResolveAsync(CancellationToken Observation observation; try { - // The observation takes no budget, so the window only times it. - observation = await reads.ReadAsync(_ => _screen.ObserveAsync(cancellationToken)).ConfigureAwait(false); + observation = await reads.ReadAsync(budget => _screen.ObserveAsync(budget, cancellationToken)).ConfigureAwait(false); } catch (EngineException ex) when (!cancellationToken.IsCancellationRequested) { diff --git a/src/E2E/Run/E2ESession.cs b/src/E2E/Run/E2ESession.cs index 4f621de5f..dabdae3a7 100644 --- a/src/E2E/Run/E2ESession.cs +++ b/src/E2E/Run/E2ESession.cs @@ -294,7 +294,8 @@ internal static async Task LaunchAsync(E2ESessionOptions options, IR options.AssertionTimeout, softFailures, locate: engine.CanLocate ? engine.LocateAsync : null, - swipe: engine.SwipeAsync); + swipe: engine.SwipeAsync, + observeWithin: scope.ObserveAsync); var fixtureContext = new EngineFixtureContext(screen); browser.Context = fixtureContext; (engine as IEngineFixtureHost)?.CreateFixtures(fixtureContext); @@ -316,7 +317,7 @@ internal static async Task LaunchAsync(E2ESessionOptions options, IR Ci = options.Ci ?? E2EConfig.IsCi(), ScrollsIntoView = (options.Engine.Capabilities & EngineCapabilities.Scroll) != 0, DirectoryOf = options.SnapshotDirectoryOf ?? (static file => file is null ? null : file + "-snapshots"), - Capture = token => engine.ScreenshotAsync(token), + Capture = (budget, token) => engine.ScreenshotAsync(budget, token), }; var context = new TestContext { diff --git a/src/E2E/Run/ScreenshotContext.cs b/src/E2E/Run/ScreenshotContext.cs index 182d0f568..eddce2384 100644 --- a/src/E2E/Run/ScreenshotContext.cs +++ b/src/E2E/Run/ScreenshotContext.cs @@ -45,7 +45,7 @@ internal sealed class ScreenshotContext /// Directory of the stored screenshots of the test file a call was made from. public Func DirectoryOf { get; init; } = static file => file is null ? null : file + "-snapshots"; - public required Func> Capture { get; init; } + public required Func> Capture { get; init; } public bool HasWritten(string file) => WrittenFiles.ContainsKey(file); diff --git a/tests/E2E.Mobile.Tests/Engine/ObservationTests.cs b/tests/E2E.Mobile.Tests/Engine/ObservationTests.cs index 1472723f5..1041a6b54 100644 --- a/tests/E2E.Mobile.Tests/Engine/ObservationTests.cs +++ b/tests/E2E.Mobile.Tests/Engine/ObservationTests.cs @@ -497,4 +497,55 @@ public async Task Takes_no_second_snapshot_for_a_capture_the_budget_abandoned() await Task.Delay(600); Assert.Equal(1, snapshots); } + + // Upstream's 'bounds screenshot by operation budget' runs artifacts.screenshot(label, operation). The port's ScreenshotArtifactAsync + // takes no timeout; the masked screenshot it shares with ScreenshotAsync does, and that is what this runs. + [Fact] + public async Task Partial_Bounds_screenshot_by_operation_budget() + { + var h = Harness.Create(); + var session = await Screens.OpenAttempt(h); + h.Fake.RespondAsync(DeviceMethods.CaptureScreenshot, (_, _) => Never()); + var failure = await Assert.ThrowsAsync(() => session.ScreenshotAsync(TimeSpan.FromMilliseconds(300), CancellationToken.None)); + Assert.Equal("OPERATION_TIMEOUT", failure.Code); + } + + // Upstream's observe takes `pixels` and its capturePixels rethrows the timeout. The port's observe takes no pixels; the screenshot it + // takes is the viewport probe of a snapshot whose size is not the known one, and that is what this runs. + [Fact] + public async Task Partial_Rethrows_OPERATION_TIMEOUT_from_capturePixels_during_observe() + { + var h = Harness.Create(); + var session = await OpenAttempt(h); + await session.ObserveAsync(CancellationToken.None); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => WithViewport(130, 281.33)); + h.Fake.RespondAsync(DeviceMethods.CaptureScreenshot, (_, _) => Never()); + var failure = await Assert.ThrowsAsync(() => session.ObserveAsync(TimeSpan.FromMilliseconds(300), CancellationToken.None)); + Assert.Equal("OPERATION_TIMEOUT", failure.Code); + } + + // As above: the screenshot that must not take a fresh budget is the viewport probe, not the observation's pixels. + [Fact] + public async Task Partial_Shares_one_budget_across_observe_steps_so_screenshot_cannot_take_a_fresh_budget() + { + var h = Harness.Create(); + var session = await OpenAttempt(h); + await session.ObserveAsync(CancellationToken.None); + h.Fake.RespondAsync(DeviceMethods.CaptureSnapshot, async (_, _) => + { + await Task.Delay(150); + return WithViewport(130, 281.33); + }); + var screenshots = 0; + h.Fake.RespondAsync(DeviceMethods.CaptureScreenshot, (_, _) => + { + Interlocked.Increment(ref screenshots); + return Never(); + }); + var started = DateTimeOffset.UtcNow; + var failure = await Assert.ThrowsAsync(() => session.ObserveAsync(TimeSpan.FromMilliseconds(500), CancellationToken.None)); + Assert.Equal("OPERATION_TIMEOUT", failure.Code); + Assert.Equal(1, screenshots); + Assert.True(DateTimeOffset.UtcNow - started < TimeSpan.FromSeconds(3)); + } } From fc5892b539229a073e6122b345feea4dbc1f7a31 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dar=C3=ADo=20Kondratiuk?= Date: Sun, 1 Nov 2026 10:32:19 -0300 Subject: [PATCH 2/2] Report a whole-screen read cut at its timeout as OPERATION_TIMEOUT whichever of the token and the budget ends first Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1yaZFgZ3FEJkYScfkK --- src/E2E/Locator/Screen.cs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/E2E/Locator/Screen.cs b/src/E2E/Locator/Screen.cs index 9a1d5ac7d..25ecf60a7 100644 --- a/src/E2E/Locator/Screen.cs +++ b/src/E2E/Locator/Screen.cs @@ -332,8 +332,10 @@ async Task Read() var observation = await read.WaitAsync(bounded.Token).ConfigureAwait(false); return RenderTree?.Invoke(observation) ?? Redact(TreeText(observation, Redactor.None, null, E2EDefaults.MaxObservationBytes)); } - catch (OperationCanceledException) when (bounded.IsCancellationRequested && !token.IsCancellationRequested) + catch (Exception ex) when (bounded.IsCancellationRequested && !token.IsCancellationRequested + && (ex is OperationCanceledException || ex is EngineException { Code: EngineErrorCodes.Cancelled })) { + // The engine's own budget ends with the token, so it may report the cancel first. throw new EngineException(EngineErrorCodes.OperationTimeout, api + " timed out", retryable: false); } }