From 99fcb5942e22d7b4700096bb2d64a6a24facb965 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dar=C3=ADo=20Kondratiuk?= Date: Sun, 1 Nov 2026 13:13:02 -0300 Subject: [PATCH] Port observe's pixels option and capturePixels to the engine session (upstream observe(operation, { pixels, comparable })) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1yaZFgZ3FEJkYScfkK --- COMPATIBILITY.md | 2 +- .../public/api/E2E.Engine.IEngineSession.html | 2 +- .../public/api/E2E.Engine.ObserveOptions.html | 13 ++ .../docs/api/E2E.Engine/IEngineSession.md | 2 +- .../docs/api/E2E.Engine/Observation.md | 19 +++ .../docs/api/E2E.Engine/ObserveOptions.md | 64 ++++++++ .../src/content/docs/api/E2E.Engine/index.md | 1 + skills/e2e-dotnet/SKILL.md | 8 +- src/E2E.Mobile/DeviceSession.Screen.cs | 28 ++++ src/E2E.Mobile/DeviceSession.cs | 17 ++- src/E2E/Engine/IEngine.cs | 15 ++ src/E2E/Engine/Observation.cs | 17 +++ .../Engine/ObservationTests.cs | 139 +++++++++++++----- 13 files changed, 284 insertions(+), 43 deletions(-) create mode 100644 docs-site/public/api/E2E.Engine.ObserveOptions.html create mode 100644 docs-site/src/content/docs/api/E2E.Engine/ObserveOptions.md diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 8e3684613..a9128d6eb 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -319,7 +319,7 @@ The port installs the skill with `npx skills add hardkoded/e2e-dotnet` and regis - `DeviceSession.BackAsync` always settles (upstream's agent origin), since a test step has no origin here. - The core session has no attempt hooks, pixel observations, or artifact directory. So `DeviceSession` is one attempt (it registers on the engine until it closes), and `ScreenshotAsync` returns the masked, status-bar-normalized PNG for `toHaveScreenshot` instead of writing numbered files. A secure field that cannot be masked still withholds the image with `ENGINE_FAILURE`. - `DeviceSession.LocateAsync(LocatorExpression, TimeSpan timeout, CancellationToken)` is upstream's `locate` (`surface.ts`), so a locator or `Expect` on a device resolves in the engine. It snapshots with `interactiveOnly: false`, runs `DeviceLocator.ResolveExpression`, and binds the matches in a separate map capped at 2048 (the oldest goes first; an older ref is `NODE_STALE`). It hands the capture it matched against to `EngineStartOptions.Screen`, and tells `EngineStartOptions.Environment` the platform and device once, when the session starts. Differences: `timeout` is the budget left (upstream's `operation.timeoutMs`), since the core session has no operation context; the expression is one flat record (`LocatorExpression`) where upstream's is a union of `query`, `filter`, `index`, `union`, `selector`, and `frame`; and the port's refs carry no revision, so upstream's `rejectSupersededLocate` has no port (a device mints a new id for every projection, so a later locate never reuses a ref). Core retries a retryable `EngineException` of a locate until the wait's deadline, as `resolveOnce` does. -- Upstream's operation budget for reads (`fix(mobile): bound reads by the operation budget`, `57f9ed9`, tester-army/e2e#1464) is ported for the snapshots a locate and a keyboard dismissal take. `InputBudget.ReadAsync` (upstream's `read`) runs a snapshot with what is left of the budget, and `InputBudget.Remaining` is `remainingMs`. A locate runs under a budget of its own `timeout` (`DeviceSession.InputBudget(label, timeout)`, upstream's `inputBudget(label, timeoutMs)`), so a device that stops answering fails the locate with `OPERATION_TIMEOUT`, which a locator reports as `ACTION_FAILED`, where it held the step until the test timeout. A sparse capture's retry the budget cannot fit returns the sparse capture, and a capture the budget abandoned is not taken again. A relocation before an action, and the snapshots of the keyboard dismissal, run under the input's budget. `device.dismissKeyboard` now has a budget too (`actionTimeout`), as upstream's has. The observation and the screenshots are bounded too (#892). `IEngineSession.ObserveAsync(TimeSpan timeout, CancellationToken)` and `ScreenshotAsync(TimeSpan timeout, CancellationToken)` carry upstream's `operation.timeoutMs` (a public API change, like `LocateAsync`'s `timeout`): they have a default that drops the budget and calls the overload without it, so an engine that bounds nothing keeps compiling (.NET-only). `DeviceSession` overrides them. `ObserveAsync` runs its snapshot and its viewport probe under one `InputBudget('observe')`, `ScreenshotAsync` its snapshot and its capture under one `InputBudget('screenshot')`, so a device that stops answering fails with `OPERATION_TIMEOUT` and the screenshot cannot take a fresh budget. The core callers pass what they have: `Screen.TreeAsync` its timeout, a locator resolve over an observation and `ToHaveScreenshotAsync` the budget of the read window. Calls with no timeout (the agent's looks, failure evidence, `ObserveAsync(CancellationToken)`) get `ActionTimeout`. `IDeviceActions.ScreenshotAsync(DeviceSession, TimeSpan, CancellationToken)` takes the timeout too; the overload without it stays, and the one with it defaults to it (.NET-only), so a caller-provided `MobileEngine.Actions` that defines only the old member keeps compiling. An infinite timeout (`Screen.TreeAsync` with none) gives an `InputBudget` that never expires and bounds no wait (.NET-only: upstream's budgets are numbers of milliseconds). `ScreenshotArtifactAsync` runs under `ActionTimeout`, where upstream's `artifacts.screenshot` takes `operation.timeoutMs`. A read the caller cancels is `CANCELLED` (`InputBudget.ReadAsync`), as the command reports it. The `InputBudget` parameter of `SnapshotAsync` and `SnapshotOrEmptyAsync` stays nullable, where upstream's is required. The port's observe takes no `pixels`, so upstream's `capturePixels` has no port: the screenshot an observation takes is the viewport probe, which rethrows `OPERATION_TIMEOUT` too. Tests, in `Engine/ObservationTests`: 'Partial_Bounds_screenshot_by_operation_budget' (the port's `ScreenshotAsync`, not `artifacts.screenshot`), 'Partial_Rethrows_OPERATION_TIMEOUT_from_capturePixels_during_observe' and 'Partial_Shares_one_budget_across_observe_steps_so_screenshot_cannot_take_a_fresh_budget' (the viewport probe, not the observation's pixels). Not ported: the positioned tap (`performAt`) `lateAction` check ('never sends a positioned tap whose target became visible past the deadline'), which needs `performWithin` (#861). Tests: in `Engine/ObservationTests`, 'bounds a snapshot by the operation budget', 'returns a sparse capture when the budget cannot fit a retry, instead of timing out', 'bounds the snapshots of a keyboard dismissal by its budget, and reports one cut off after the drag as possibly committed', 'lets a slow snapshot run for the whole timeout the locate gets', and 'takes no second snapshot for a capture the budget abandoned'; in `Support/SupportHelpersTests`, 'runs a read with what is left of its budget, and reports one cut off after an input as possibly committed'. These run on real timers of a few hundred milliseconds. +- Upstream's operation budget for reads (`fix(mobile): bound reads by the operation budget`, `57f9ed9`, tester-army/e2e#1464) is ported for the snapshots a locate and a keyboard dismissal take. `InputBudget.ReadAsync` (upstream's `read`) runs a snapshot with what is left of the budget, and `InputBudget.Remaining` is `remainingMs`. A locate runs under a budget of its own `timeout` (`DeviceSession.InputBudget(label, timeout)`, upstream's `inputBudget(label, timeoutMs)`), so a device that stops answering fails the locate with `OPERATION_TIMEOUT`, which a locator reports as `ACTION_FAILED`, where it held the step until the test timeout. A sparse capture's retry the budget cannot fit returns the sparse capture, and a capture the budget abandoned is not taken again. A relocation before an action, and the snapshots of the keyboard dismissal, run under the input's budget. `device.dismissKeyboard` now has a budget too (`actionTimeout`), as upstream's has. The observation and the screenshots are bounded too (#892). `IEngineSession.ObserveAsync(TimeSpan timeout, CancellationToken)` and `ScreenshotAsync(TimeSpan timeout, CancellationToken)` carry upstream's `operation.timeoutMs` (a public API change, like `LocateAsync`'s `timeout`): they have a default that drops the budget and calls the overload without it, so an engine that bounds nothing keeps compiling (.NET-only). `DeviceSession` overrides them. `ObserveAsync` runs its snapshot and its viewport probe under one `InputBudget('observe')`, `ScreenshotAsync` its snapshot and its capture under one `InputBudget('screenshot')`, so a device that stops answering fails with `OPERATION_TIMEOUT` and the screenshot cannot take a fresh budget. The core callers pass what they have: `Screen.TreeAsync` its timeout, a locator resolve over an observation and `ToHaveScreenshotAsync` the budget of the read window. Calls with no timeout (the agent's looks, failure evidence, `ObserveAsync(CancellationToken)`) get `ActionTimeout`. `IDeviceActions.ScreenshotAsync(DeviceSession, TimeSpan, CancellationToken)` takes the timeout too; the overload without it stays, and the one with it defaults to it (.NET-only), so a caller-provided `MobileEngine.Actions` that defines only the old member keeps compiling. An infinite timeout (`Screen.TreeAsync` with none) gives an `InputBudget` that never expires and bounds no wait (.NET-only: upstream's budgets are numbers of milliseconds). `ScreenshotArtifactAsync(label, TimeSpan timeout, ct)` is `artifacts.screenshot(label, operation)`; the overload without a timeout runs under `ActionTimeout`. `IEngineSession.ObserveAsync(ObserveOptions? options, TimeSpan timeout, ct)` is upstream's `observe(operation, { pixels, comparable })` (`ObserveOptions { Pixels, Comparable }` is `EngineObserveOptions`; a public API change). With `Pixels`, `Observation.Pixels` (an `EngineScreenshot`: upstream's `pixels` and `maskedRegionCount`) carries the screen's image, with every secure field covered and the status bar normalized when `Comparable`; the default drops the options and returns the tree alone (.NET-only), as an engine that takes none does. `DeviceSession.CapturePixelsAsync` is `capturePixels`: it takes the screenshot under the observation's budget, rethrows `OPERATION_TIMEOUT`, and costs the observation its image (not the step) for any other failure or a secure field it cannot prove covered. The agent still takes its screenshot through `Capture`, so `ObservationFeed` does not use `Observation.Pixels` yet. A read the caller cancels is `CANCELLED` (`InputBudget.ReadAsync`), as the command reports it. The `InputBudget` parameter of `SnapshotAsync` and `SnapshotOrEmptyAsync` stays nullable, where upstream's is required. Tests, in `Engine/ObservationTests`: 'bounds screenshot by operation budget', 'rethrows OPERATION_TIMEOUT from capturePixels during observe', 'shares one budget across observe steps so screenshot cannot take a fresh budget', 'captures pixels on request, masks every secure node, and degrades to tree-only when it cannot', and 'holds the status bar in a fixed state for a screenshot taken for comparison, and only then' and 'withholds the pixels of a screen with a secure field when the image cannot be read or decoded' (now run through `ObserveAsync` with `Pixels`). Not ported: the positioned tap (`performAt`) `lateAction` check ('never sends a positioned tap whose target became visible past the deadline'), which needs `performWithin` (#861). Tests: in `Engine/ObservationTests`, 'bounds a snapshot by the operation budget', 'returns a sparse capture when the budget cannot fit a retry, instead of timing out', 'bounds the snapshots of a keyboard dismissal by its budget, and reports one cut off after the drag as possibly committed', 'lets a slow snapshot run for the whole timeout the locate gets', and 'takes no second snapshot for a capture the budget abandoned'; in `Support/SupportHelpersTests`, 'runs a read with what is left of its budget, and reports one cut off after an input as possibly committed'. These run on real timers of a few hundred milliseconds. - Upstream's Android locate (`fix(mobile): keep locator polling through unreadable android snapshots`, `4fe6baf`) is ported. A snapshot agent-device cannot read yet (`details.androidSnapshotHelperFailureReason` of `empty-helper-output`, `system-window-only`, or `content-poor-app-window`, the message `Android snapshot helper returned insufficient foreground app content`) matches nothing in a locate, so a locator or `Expect` keeps polling until its wait runs out, and a locate whose capture leaves no budget fails with `ENGINE_FAILURE` and that message. `DeviceErrors.IsUnreadableCapture` reads the reason off the `AgentDeviceError` that the translated `EngineException` keeps as its `InnerException`. An observation, such as the agent's next turn, still fails on it at once, as upstream's. - Upstream's iOS locate (`2b2b845`, tester-army/e2e#1281) is ported. Just after `OpenAppAsync`, `RestartAsync` or `OpenLinkAsync`, a snapshot that iOS refuses with `details.runnerErrorCode` of `APP_NOT_RUNNING` (`DeviceErrors.IsAppNotRunning`) matches nothing in a locate (locator waits and `Expect` only; an observation still fails at once), like an unreadable Android capture, until a snapshot succeeds (`DeviceSession._launching`, upstream's `launching`). A wait that runs out while the app is still down fails with `ENGINE_FAILURE` and that reason. An app that exits after its first readable screen still fails at once. - Upstream's Android test id match without the `:id/` prefix (`feat(mobile): match Android test ids without their package prefix`, `2791786`) is in `DeviceLocator`, which `DeviceSession.LocateAsync` now calls, so `Screen.GetByTestId("greet")` matches `:id/greet` on Android when nothing matches the full id. diff --git a/docs-site/public/api/E2E.Engine.IEngineSession.html b/docs-site/public/api/E2E.Engine.IEngineSession.html index 395199090..75a821c76 100644 --- a/docs-site/public/api/E2E.Engine.IEngineSession.html +++ b/docs-site/public/api/E2E.Engine.IEngineSession.html @@ -2,7 +2,7 @@ - + Redirecting… diff --git a/docs-site/public/api/E2E.Engine.ObserveOptions.html b/docs-site/public/api/E2E.Engine.ObserveOptions.html new file mode 100644 index 000000000..b80800194 --- /dev/null +++ b/docs-site/public/api/E2E.Engine.ObserveOptions.html @@ -0,0 +1,13 @@ + + + + + + +Redirecting… + + + +

Continue

+ + diff --git a/docs-site/src/content/docs/api/E2E.Engine/IEngineSession.md b/docs-site/src/content/docs/api/E2E.Engine/IEngineSession.md index f89109cb8..c2c3f285d 100644 --- a/docs-site/src/content/docs/api/E2E.Engine/IEngineSession.md +++ b/docs-site/src/content/docs/api/E2E.Engine/IEngineSession.md @@ -21,4 +21,4 @@ public interface IEngineSession : IAsyncDisposable One attempt's session with the app. Disposed at the end of the attempt, after `afterEach`. -[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Engine/IEngine.cs#L179) +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Engine/IEngine.cs#L185) diff --git a/docs-site/src/content/docs/api/E2E.Engine/Observation.md b/docs-site/src/content/docs/api/E2E.Engine/Observation.md index e2c6bfd45..139753006 100644 --- a/docs-site/src/content/docs/api/E2E.Engine/Observation.md +++ b/docs-site/src/content/docs/api/E2E.Engine/Observation.md @@ -45,6 +45,25 @@ null only when it did not measure it. [View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Engine/Observation.cs#L128) + + + +### Pixels + + +```csharp +public EngineScreenshot? Pixels { get; init; } +``` + + +The screen's pixels, with every secure field covered, when [Pixels](/api/E2E.Engine/ObserveOptions/#E2E_Engine_ObserveOptions_Pixels) asked for them and the engine could +take them (upstream's `EngineSnapshot.pixels` and `maskedRegionCount`, as [EngineScreenshot](/api/E2E.Engine/EngineScreenshot/)). Null when it did +not ask, or when the image could not be taken or proven redacted: the observation is then the tree alone. + +**Value:** [EngineScreenshot](/api/E2E.Engine/EngineScreenshot/) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Engine/Observation.cs#L154) + diff --git a/docs-site/src/content/docs/api/E2E.Engine/ObserveOptions.md b/docs-site/src/content/docs/api/E2E.Engine/ObserveOptions.md new file mode 100644 index 000000000..e18cc5aca --- /dev/null +++ b/docs-site/src/content/docs/api/E2E.Engine/ObserveOptions.md @@ -0,0 +1,64 @@ +--- +title: "ObserveOptions" +description: "What an observation takes besides the tree. Upstream's EngineObserveOptions." +editUrl: false +prev: false +next: false +sidebar: + label: "ObserveOptions" + hidden: true +--- + + + +Namespace: [E2E.Engine](/api/E2E.Engine/) + + +```csharp +public sealed record ObserveOptions : IEquatable +``` + + +What an observation takes besides the tree. Upstream's `EngineObserveOptions`. + +**Inheritance:** [object](https://learn.microsoft.com/dotnet/api/system.object) + +**Implements:** [IEquatable<ObserveOptions>](https://learn.microsoft.com/dotnet/api/system.iequatable-1) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Engine/Observation.cs#L158) + +## Properties + + + + +### Comparable + + +```csharp +public bool Comparable { get; init; } +``` + + +Take the pixels in a state fit for comparing against a stored one: the status bar normalized. + +**Value:** [bool](https://learn.microsoft.com/dotnet/api/system.boolean) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Engine/Observation.cs#L164) + + + + +### Pixels + + +```csharp +public bool Pixels { get; init; } +``` + + +Take the screen's pixels with the tree ([Pixels](/api/E2E.Engine/Observation/#E2E_Engine_Observation_Pixels)). + +**Value:** [bool](https://learn.microsoft.com/dotnet/api/system.boolean) + +[View source](https://github.com/hardkoded/e2e-dotnet/blob/main/src/E2E/Engine/Observation.cs#L161) diff --git a/docs-site/src/content/docs/api/E2E.Engine/index.md b/docs-site/src/content/docs/api/E2E.Engine/index.md index e93cfa1d9..41f990d6d 100644 --- a/docs-site/src/content/docs/api/E2E.Engine/index.md +++ b/docs-site/src/content/docs/api/E2E.Engine/index.md @@ -60,6 +60,7 @@ Public types in the `E2E.Engine` namespace. | [NodeStates](/api/E2E.Engine/NodeStates/) | Boolean states a platform can report for one node. | | [Observation](/api/E2E.Engine/Observation/) | A redacted semantic snapshot of the current screen. | | [ObservationLimits](/api/E2E.Engine/ObservationLimits/) | Limits from the engine contract. Engines cut names and text before they reach the model. | +| [ObserveOptions](/api/E2E.Engine/ObserveOptions/) | What an observation takes besides the tree. Upstream's EngineObserveOptions. | | [ProviderRecordContext](/api/E2E.Engine/ProviderRecordContext/) | Handed to a provider's RecordAsync, once per attempt that records video. | | [ProviderRecording](/api/E2E.Engine/ProviderRecording/) | A recording a provider's service makes of the browser or device it leased, as RecordAsync started it. | | [ProviderRecordingResult](/api/E2E.Engine/ProviderRecordingResult/) | What a stopped recording became: a file the provider wrote into the directory it was given, by name (File), or a recording its service keeps, by http(s) URL and media type (Url and MediaType). | diff --git a/skills/e2e-dotnet/SKILL.md b/skills/e2e-dotnet/SKILL.md index ea4aa21d0..43aaa6260 100644 --- a/skills/e2e-dotnet/SKILL.md +++ b/skills/e2e-dotnet/SKILL.md @@ -339,7 +339,13 @@ lines you changed: with `ACTION_FAILED` (`InnerException` `OPERATION_TIMEOUT`) instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test - timeout, so a screen that shows up during it still passes. + timeout, so a screen that shows up during it still passes. An engine session + takes the screen's pixels with the tree through + `ObserveAsync(new ObserveOptions { Pixels = true, Comparable = ... }, timeout, ct)` + (`Observation.Pixels`, secure fields covered); the screenshot shares the + observation's budget, and a device that stops answering fails the call with + `OPERATION_TIMEOUT`. Any other failure of the screenshot leaves the + observation with the tree alone. - Secrets never appear in test code. Declare them under `secrets` in the config and read them with `Secrets.Get(name)`; accounts come from `Credentials.User(name)`. Hand the `Secret` only to `FillAsync` or diff --git a/src/E2E.Mobile/DeviceSession.Screen.cs b/src/E2E.Mobile/DeviceSession.Screen.cs index 60c51858e..4b68abdf1 100644 --- a/src/E2E.Mobile/DeviceSession.Screen.cs +++ b/src/E2E.Mobile/DeviceSession.Screen.cs @@ -627,6 +627,34 @@ internal async Task MaskedScreenshotAsync(bool normalizeStatus return new EngineScreenshot(masked, scale, regions); } + /// + /// The viewport's pixels for an observation (upstream's capturePixels). Best effort: a screenshot that cannot be taken, or whose + /// secure fields cannot be proven covered, costs the observation its image, not the step, unless the operation's budget ran out. + /// + private async Task CapturePixelsAsync(ProjectedSnapshot projected, ViewportSize viewport, bool comparable, InputBudget budget, CancellationToken cancellationToken) + { + byte[] data; + try + { + data = await CaptureScreenshotAsync(async (shot, file) => await File.ReadAllBytesAsync(shot.Path ?? file, cancellationToken).ConfigureAwait(false), comparable, budget, cancellationToken).ConfigureAwait(false); + } + catch (EngineException cause) when (cause.Code == EngineErrorCodes.OperationTimeout) + { + throw; + } + catch (Exception) when (!cancellationToken.IsCancellationRequested) + { + return null; + } + + if (RedactSecure(data, projected, viewport) is not { } redacted || DeviceSupport.ReadPngSize(redacted.Data) is not { } size) + { + return null; + } + + return new EngineScreenshot(redacted.Data, viewport.Width > 0 ? size.Width / viewport.Width : 1, redacted.Masked); + } + /// /// Paints every secure node's bounds black on a screenshot of the same screen. Returns the masked bytes, or null /// when a secure node has no bounds or the image format cannot be edited: the caller then withholds the image diff --git a/src/E2E.Mobile/DeviceSession.cs b/src/E2E.Mobile/DeviceSession.cs index 6a7aec441..57e92fadb 100644 --- a/src/E2E.Mobile/DeviceSession.cs +++ b/src/E2E.Mobile/DeviceSession.cs @@ -302,7 +302,13 @@ await CommandAsync( /// Observes the screen within , the operation budget that bounds every device command of the observation: /// the snapshot and the viewport probe share it (upstream's observe under inputBudget('observe', operation.timeoutMs)). /// - public async Task ObserveAsync(TimeSpan timeout, CancellationToken cancellationToken) + public Task ObserveAsync(TimeSpan timeout, CancellationToken cancellationToken) => ObserveAsync(null, timeout, cancellationToken); + + /// + /// with : with the + /// screenshot is taken under the same budget as the snapshot and the viewport probe (upstream's observe(operation, options)). + /// + public async Task ObserveAsync(ObserveOptions? options, TimeSpan timeout, CancellationToken cancellationToken) { var budget = InputBudget("observe", timeout); var raw = await SnapshotOrEmptyAsync(Options.Snapshot == SnapshotMode.Interactive, budget, cancellationToken).ConfigureAwait(false); @@ -310,9 +316,11 @@ public async Task ObserveAsync(TimeSpan timeout, CancellationToken _generation = projected.Index.ToDictionary(entry => entry.Id, entry => Bind(entry, projected.Index, projected.RefsGeneration)); var viewport = await ViewportForAsync(projected, budget, cancellationToken).ConfigureAwait(false); Route = DeviceSupport.ScreenLocation(raw.AppBundleId ?? raw.AppName ?? _appIdentity, SnapshotProjection.ScreenTitle(projected)) ?? ""; + var pixels = options?.Pixels == true ? await CapturePixelsAsync(projected, viewport, options.Comparable, budget, cancellationToken).ConfigureAwait(false) : null; return new Observation { Route = Route, + Pixels = pixels, Roots = [SnapshotProjection.ScreenRoot(projected.Roots, viewport)], Viewport = new TraceViewport((int)Math.Round(viewport.Width), (int)Math.Round(viewport.Height)), Truncated = raw.Truncated || raw.Sparse, @@ -340,10 +348,13 @@ public Task ScreenshotAsync(TimeSpan timeout, CancellationToke /// secure node's bounds are painted over before the file is kept. An image that cannot be redacted is not written at all (upstream's /// screenshot in mobile/src/surface.ts). /// - public async Task ScreenshotArtifactAsync(string? label, CancellationToken cancellationToken) + public Task ScreenshotArtifactAsync(string? label, CancellationToken cancellationToken) => ScreenshotArtifactAsync(label, ActionTimeout, cancellationToken); + + /// within , the operation budget its snapshot and capture share. + public async Task ScreenshotArtifactAsync(string? label, TimeSpan timeout, CancellationToken cancellationToken) { var artifactsDir = _artifactsDir ?? throw DeviceSupport.InvalidState("screenshot outside an attempt"); - var masked = await MaskedScreenshotAsync(normalizeStatusBar: false, InputBudget("screenshot"), cancellationToken).ConfigureAwait(false); + var masked = await MaskedScreenshotAsync(normalizeStatusBar: false, InputBudget("screenshot", timeout), cancellationToken).ConfigureAwait(false); var name = Interlocked.Increment(ref _screenshots).ToString("D3", System.Globalization.CultureInfo.InvariantCulture) + "-" + DeviceSupport.SanitizeFilename(label ?? "screenshot") + ".png"; var directory = Path.Combine(artifactsDir, "screenshots"); diff --git a/src/E2E/Engine/IEngine.cs b/src/E2E/Engine/IEngine.cs index 403a7f330..887daa9d3 100644 --- a/src/E2E/Engine/IEngine.cs +++ b/src/E2E/Engine/IEngine.cs @@ -199,6 +199,14 @@ public interface IEngineSession : IAsyncDisposable /// Task ObserveAsync(TimeSpan timeout, CancellationToken cancellationToken) => ObserveAsync(cancellationToken); + /// + /// with : upstream's + /// observe(operation, { pixels, comparable }). With the observation carries the screen's + /// , taken under the same budget as the tree. An engine that takes none returns the tree alone, and the + /// default does (.NET-only: it keeps the engines that capture no pixels compiling). + /// + Task ObserveAsync(ObserveOptions? options, TimeSpan timeout, CancellationToken cancellationToken) => ObserveAsync(timeout, cancellationToken); + Task PerformAsync(SemanticNode node, LocatorAction action, CancellationToken cancellationToken); /// Sends a key to whatever holds focus. The harness has already validated the key. @@ -270,6 +278,13 @@ Task ScreenshotAsync(CancellationToken cancellationToken) => Task ScreenshotArtifactAsync(string? label, CancellationToken cancellationToken) => Task.FromException(new EngineException("UNSUPPORTED_CAPABILITY", "screenshots is not supported by this engine.")); + /// + /// within , the operation budget its snapshot + /// and capture share (upstream's artifacts.screenshot(label, operation)). The default drops the budget (.NET-only, as + /// ). + /// + Task ScreenshotArtifactAsync(string? label, TimeSpan timeout, CancellationToken cancellationToken) => ScreenshotArtifactAsync(label, cancellationToken); + /// /// Starts recording the surface, once its page can be recorded. Engines that declare /// implement it. An attempt may start and stop more than once, never overlapping: the diff --git a/src/E2E/Engine/Observation.cs b/src/E2E/Engine/Observation.cs index 602423de7..79507032e 100644 --- a/src/E2E/Engine/Observation.cs +++ b/src/E2E/Engine/Observation.cs @@ -145,4 +145,21 @@ public sealed class Observation /// screen it last read. /// public string Revision { get; internal set; } = ""; + + /// + /// The screen's pixels, with every secure field covered, when asked for them and the engine could + /// take them (upstream's EngineSnapshot.pixels and maskedRegionCount, as ). Null when it did + /// not ask, or when the image could not be taken or proven redacted: the observation is then the tree alone. + /// + public EngineScreenshot? Pixels { get; init; } +} + +/// What an observation takes besides the tree. Upstream's EngineObserveOptions. +public sealed record ObserveOptions +{ + /// Take the screen's pixels with the tree (). + public bool Pixels { get; init; } + + /// Take the pixels in a state fit for comparing against a stored one: the status bar normalized. + public bool Comparable { get; init; } } diff --git a/tests/E2E.Mobile.Tests/Engine/ObservationTests.cs b/tests/E2E.Mobile.Tests/Engine/ObservationTests.cs index 1041a6b54..ca4bbe4be 100644 --- a/tests/E2E.Mobile.Tests/Engine/ObservationTests.cs +++ b/tests/E2E.Mobile.Tests/Engine/ObservationTests.cs @@ -4,6 +4,7 @@ using System.Text.Json.Nodes; using E2E.Engine; +using static E2E.Mobile.Tests.RawNodes; namespace E2E.Mobile.Tests.Engine; @@ -209,14 +210,11 @@ public async Task Holds_the_status_bar_in_a_fixed_state_for_a_screenshot_taken_f { asked.Add(options?["normalizeStatusBar"]?.GetValue()); File.WriteAllBytes(options!["path"]!.GetValue(), png); - return new JsonObject { ["path"] = options["path"]!.GetValue(), ["logicalWidth"] = 390, ["logicalHeight"] = 844 }; + return new JsonObject { ["path"] = options["path"]!.GetValue() }; }); - - // No geometry in the tree: the viewport probe is the one capture that is not a comparison. - h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => FakeDeviceClient.NodesOnly(RawNodes.SettingsNodes.Where(node => node.Password != true && node.Type != "secure-text-field").Select(node => node with { Rect = null }))); var session = await OpenAttempt(h); - await session.ObserveAsync(CancellationToken.None); - await session.ScreenshotAsync(CancellationToken.None); + await session.ObserveAsync(new ObserveOptions { Pixels = true }, TimeSpan.FromSeconds(30), CancellationToken.None); + await session.ObserveAsync(new ObserveOptions { Pixels = true, Comparable = true }, TimeSpan.FromSeconds(30), CancellationToken.None); Assert.Equal([null, true], asked); } @@ -241,24 +239,92 @@ public async Task Counts_the_regions_it_paints_over_secure_fields_in_a_screensho public async Task Withholds_the_pixels_of_a_screen_with_a_secure_field_when_the_image_cannot_be_read_or_decoded() { var h = Harness.Create(); + var artifactsDir = Directory.CreateTempSubdirectory("e2e-mobile-artifacts-"); + try + { + // A PNG signature and header naming the size, with no image data behind them: the size reads, the decode fails. + var headerOnly = PngCodec.Encode(PngCodec.Solid(195, 422, 4, 0))[..33]; + byte[] shot = [1, 2, 3]; + h.Fake.Respond(DeviceMethods.CaptureScreenshot, options => + { + File.WriteAllBytes(options!["path"]!.GetValue(), shot); + return new JsonObject { ["path"] = options["path"]!.GetValue() }; + }); + var session = await Screens.OpenAttempt(h, artifactsDir.FullName); + foreach (var bytes in new[] { new byte[] { 1, 2, 3 }, headerOnly }) + { + shot = bytes; + var observation = await session.ObserveAsync(new ObserveOptions { Pixels = true }, TimeSpan.FromSeconds(30), CancellationToken.None); + Assert.Null(observation.Pixels); + Assert.Single(observation.Roots[0].Children); + var withheld = await Assert.ThrowsAsync(() => session.ScreenshotArtifactAsync("password", TimeSpan.FromSeconds(30), CancellationToken.None)); + Assert.Equal("ENGINE_FAILURE", withheld.Code); + } - // A PNG signature and header naming the size, with no image data behind them: the size reads, the decode fails. - var headerOnly = PngCodec.Encode(PngCodec.Solid(195, 422, 4, 0))[..33]; - byte[] shot = [1, 2, 3]; + Assert.False(Directory.Exists(Path.Combine(artifactsDir.FullName, "screenshots"))); + } + finally + { + artifactsDir.Delete(recursive: true); + } + } + + [Fact] + public async Task Captures_pixels_on_request_masks_every_secure_node_and_degrades_to_tree_only_when_it_cannot() + { + var h = Harness.Create(); + // 390x844 viewport at half scale: a small white RGBA image keeps the encode cheap and still proves the bounds are scaled. + var png = PngCodec.Encode(PngCodec.Solid(195, 422, 4, 255)); h.Fake.Respond(DeviceMethods.CaptureScreenshot, options => { - File.WriteAllBytes(options!["path"]!.GetValue(), shot); + File.WriteAllBytes(options!["path"]!.GetValue(), png); return new JsonObject { ["path"] = options["path"]!.GetValue() }; }); var session = await OpenAttempt(h); - foreach (var bytes in new[] { new byte[] { 1, 2, 3 }, headerOnly }) - { - shot = bytes; - var observation = await session.ObserveAsync(CancellationToken.None); - Assert.Single(observation.Roots[0].Children); - var withheld = await Assert.ThrowsAsync(() => session.ScreenshotAsync(CancellationToken.None)); - Assert.Equal("ENGINE_FAILURE", withheld.Code); - } + var timeout = TimeSpan.FromSeconds(30); + var observation = await session.ObserveAsync(new ObserveOptions { Pixels = true }, timeout, CancellationToken.None); + var pixels = observation.Pixels!; + Assert.Equal(0.5, pixels.Scale); + // One secure node on the Settings fixture: the Password field at y=270, 44 tall. + Assert.Equal(1, pixels.MaskedRegionCount); + var decoded = PngCodec.Decode(pixels.Png); + Assert.Equal((195, 422), (decoded.Width, decoded.Height)); + int[] At(DecodedPng image, int x, int y) => [image.Pixels[(((y * 195) + x) * 4)], image.Pixels[(((y * 195) + x) * 4) + 1], image.Pixels[(((y * 195) + x) * 4) + 2]]; + Assert.Equal([0, 0, 0], At(decoded, 100, 290 / 2)); + Assert.Equal([255, 255, 255], At(decoded, 100, 240 / 2)); + + // Without the option the observation is the tree alone. + Assert.Null((await session.ObserveAsync(timeout, CancellationToken.None)).Pixels); + + // Android flags a password EditText by attribute, not by class; it is painted over the same way. + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => FakeDeviceClient.NodesOnly( + [ + N("e1", index: 0, depth: 0, type: "android.widget.FrameLayout", rect: new Rect(0, 0, 390, 844)), + N("e2", index: 1, parentIndex: 0, depth: 1, type: "android.widget.EditText", label: "Password", value: "hunter2", password: true, rect: new Rect(0, 270, 390, 44)), + ])); + var android = await session.ObserveAsync(new ObserveOptions { Pixels = true }, timeout, CancellationToken.None); + Assert.Equal(1, android.Pixels!.MaskedRegionCount); + var password = Named(android.Roots[0], "Password"); + Assert.Equal("textbox", password.Role); + Assert.True(password.States.Secure); + Assert.Null(password.Value); + var androidPixels = PngCodec.Decode(android.Pixels.Png); + Assert.Equal([0, 0, 0], At(androidPixels, 100, 290 / 2)); + Assert.Equal([255, 255, 255], At(androidPixels, 100, 240 / 2)); + + // A secure node without bounds cannot be masked: the tree ships, the image does not. + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => FakeDeviceClient.NodesOnly([N("e1", type: "SecureTextField", label: "PIN", value: "1234")])); + var unmaskable = await session.ObserveAsync(new ObserveOptions { Pixels = true }, timeout, CancellationToken.None); + Assert.Null(unmaskable.Pixels); + Assert.Single(unmaskable.Roots[0].Children); + // A snapshot without geometry keeps the viewport the session last learned. + Assert.Equal(new TraceViewport(390, 844), unmaskable.Viewport); + + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => FakeDeviceClient.SettingsSnapshot()); + h.Fake.Respond(DeviceMethods.CaptureScreenshot, _ => throw new AgentDeviceError("COMMAND_FAILED", "screenshot failed")); + var treeOnly = await session.ObserveAsync(new ObserveOptions { Pixels = true }, timeout, CancellationToken.None); + Assert.Null(treeOnly.Pixels); + Assert.NotEmpty(treeOnly.Roots[0].Children); } [Fact] @@ -498,43 +564,44 @@ public async Task Takes_no_second_snapshot_for_a_capture_the_budget_abandoned() Assert.Equal(1, snapshots); } - // Upstream's 'bounds screenshot by operation budget' runs artifacts.screenshot(label, operation). The port's ScreenshotArtifactAsync - // takes no timeout; the masked screenshot it shares with ScreenshotAsync does, and that is what this runs. [Fact] - public async Task Partial_Bounds_screenshot_by_operation_budget() + public async Task Bounds_screenshot_by_operation_budget() { var h = Harness.Create(); - var session = await Screens.OpenAttempt(h); - h.Fake.RespondAsync(DeviceMethods.CaptureScreenshot, (_, _) => Never()); - var failure = await Assert.ThrowsAsync(() => session.ScreenshotAsync(TimeSpan.FromMilliseconds(300), CancellationToken.None)); - Assert.Equal("OPERATION_TIMEOUT", failure.Code); + var artifactsDir = Directory.CreateTempSubdirectory("e2e-mobile-artifacts-"); + try + { + var session = await Screens.OpenAttempt(h, artifactsDir.FullName); + h.Fake.RespondAsync(DeviceMethods.CaptureScreenshot, (_, _) => Never()); + var failure = await Assert.ThrowsAsync(() => session.ScreenshotArtifactAsync("hung shot", TimeSpan.FromMilliseconds(300), CancellationToken.None)); + Assert.Equal("OPERATION_TIMEOUT", failure.Code); + } + finally + { + artifactsDir.Delete(recursive: true); + } } - // Upstream's observe takes `pixels` and its capturePixels rethrows the timeout. The port's observe takes no pixels; the screenshot it - // takes is the viewport probe of a snapshot whose size is not the known one, and that is what this runs. [Fact] - public async Task Partial_Rethrows_OPERATION_TIMEOUT_from_capturePixels_during_observe() + public async Task Rethrows_OPERATION_TIMEOUT_from_capturePixels_during_observe() { var h = Harness.Create(); var session = await OpenAttempt(h); - await session.ObserveAsync(CancellationToken.None); - h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => WithViewport(130, 281.33)); h.Fake.RespondAsync(DeviceMethods.CaptureScreenshot, (_, _) => Never()); - var failure = await Assert.ThrowsAsync(() => session.ObserveAsync(TimeSpan.FromMilliseconds(300), CancellationToken.None)); + var failure = await Assert.ThrowsAsync(() => session.ObserveAsync(new ObserveOptions { Pixels = true }, TimeSpan.FromMilliseconds(300), CancellationToken.None)); Assert.Equal("OPERATION_TIMEOUT", failure.Code); } - // As above: the screenshot that must not take a fresh budget is the viewport probe, not the observation's pixels. [Fact] - public async Task Partial_Shares_one_budget_across_observe_steps_so_screenshot_cannot_take_a_fresh_budget() + public async Task Shares_one_budget_across_observe_steps_so_screenshot_cannot_take_a_fresh_budget() { var h = Harness.Create(); var session = await OpenAttempt(h); - await session.ObserveAsync(CancellationToken.None); + // The snapshot carries its geometry, so the only screenshot is the observation's pixels. h.Fake.RespondAsync(DeviceMethods.CaptureSnapshot, async (_, _) => { await Task.Delay(150); - return WithViewport(130, 281.33); + return WithViewport(402, 874); }); var screenshots = 0; h.Fake.RespondAsync(DeviceMethods.CaptureScreenshot, (_, _) => @@ -543,7 +610,7 @@ public async Task Partial_Shares_one_budget_across_observe_steps_so_screenshot_c return Never(); }); var started = DateTimeOffset.UtcNow; - var failure = await Assert.ThrowsAsync(() => session.ObserveAsync(TimeSpan.FromMilliseconds(500), CancellationToken.None)); + var failure = await Assert.ThrowsAsync(() => session.ObserveAsync(new ObserveOptions { Pixels = true }, TimeSpan.FromMilliseconds(500), CancellationToken.None)); Assert.Equal("OPERATION_TIMEOUT", failure.Code); Assert.Equal(1, screenshots); Assert.True(DateTimeOffset.UtcNow - started < TimeSpan.FromSeconds(3));