Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions COMPATIBILITY.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs-site/src/content/docs/docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,4 @@ await Screen.GetByTestId("feed").ScrollUntilVisibleAsync(row, new ScrollUntilVis

A locator reports an engine failure in the test's own codes, with the engine's error as `InnerException`: `NOT_ACTIONABLE`, `ACTION_MAY_HAVE_COMMITTED`, and `OPERATION_TIMEOUT` are `ACTION_FAILED`, a stale node or a missing frame is `LOCATOR_NOT_FOUND`, an ambiguous frame is `LOCATOR_AMBIGUOUS`, and `INVALID_STATE` is `APP_NOT_OPEN`. `CANCELLED` and `ENGINE_FAILURE` keep their codes (category `infrastructure`), and `UNSUPPORTED_CAPABILITY` is a `ConfigurationException`. A stale node found by an action is looked up again, within the action timeout, before the action fails.

A read or navigation that times out fails with `OPERATION_TIMEOUT`. It is not retried, even when the call log mentions a detached element or a navigation race. On a device, an input (a tap, key, swipe, or `Back`) that the device does not answer within `actionTimeout` fails with `ACTION_MAY_HAVE_COMMITTED` and is never repeated, because it may have landed; an input the timeout ran out before is not sent (`OPERATION_TIMEOUT`), and a long press that holds at least as long as the time left is refused (`NOT_ACTIONABLE`). A locate on a device runs within the timeout of the call that takes it, so a device that stops answering fails the step with `ACTION_FAILED` instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. See `EngineErrorCodes` in the [API reference](/api/).
A read or navigation that times out fails with `OPERATION_TIMEOUT`. It is not retried, even when the call log mentions a detached element or a navigation race. On a device, an input (a tap, key, swipe, or `Back`) that the device does not answer within `actionTimeout` fails with `ACTION_MAY_HAVE_COMMITTED` and is never repeated, because it may have landed; an input the timeout ran out before is not sent (`OPERATION_TIMEOUT`), and a long press that holds at least as long as the time left is refused (`NOT_ACTIONABLE`). A locate, an observation, or a screenshot on a device runs within the timeout of the call that takes it, so a device that stops answering fails the step with `ACTION_FAILED` instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. See `EngineErrorCodes` in the [API reference](/api/).
11 changes: 7 additions & 4 deletions skills/e2e-dotnet/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -330,10 +330,13 @@ lines you changed:
answer within `actionTimeout` fails with `ACTION_MAY_HAVE_COMMITTED` and is
never repeated; one the timeout ran out before is not sent and fails with
`OPERATION_TIMEOUT`; a long press that holds at least as long as the time left
is refused with `NOT_ACTIONABLE`. A locate on a device (a locator, `Expect`)
runs within the timeout of the call that takes it, so a device that stops
answering fails the step with `ACTION_FAILED` instead of holding it until the
test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their
is refused with `NOT_ACTIONABLE`. A locate, an observation, or a screenshot
on a device (a locator, `Expect`, `ToHaveScreenshotAsync`, `Screen.TreeAsync`)
runs within the timeout of the call that takes it, and its snapshot and its
capture or viewport probe share that one operation budget (a call with no
timeout gets `actionTimeout`). A device that stops answering fails the step
with `ACTION_FAILED` (`InnerException` `OPERATION_TIMEOUT`) instead of holding
it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their
last read extra time, up to their slowest read so far and within the test
timeout, so a screen that shows up during it still passes.
- Secrets never appear in test code. Declare them under `secrets` in the
Expand Down
2 changes: 1 addition & 1 deletion skills/e2e-dotnet/references/mobile.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,6 @@ Limits of the preview: the tests attach to the running app, so `Device.InstallAp
- Building the sample iOS app needs `-p:ValidateXcodeVersion=false` when the installed Xcode is newer than the one the .NET iOS pack expects.
- Agent steps on a device are slower than in a browser. Raise `StepTimeout` to 90 seconds. The 30 second default timed out a two-field step.
- A ref belongs to the agent-device capture it came from, and every action pins it there (`@e4~s7`), so agent-device refuses a ref once a newer capture supersedes it. An action on a node that a newer capture replaced never acts on a different element: a test's action waits out `TransitionMilliseconds` for a control that arrived with the last action, and a node the fresh capture no longer lists is a retryable `NODE_STALE`, so the locator resolves again within its timeout. A direct `Device` call (`DragToAsync`, `LongPressAsync`) does not retry and surfaces it. `Device.DragToAsync` between nodes located in separate captures takes both refs from the newest one.
- An input that outlives its timeout. A tap, fill, key, swipe, or `Back` gets the `actionTimeout` (30 seconds by default). If the device doesn't answer in time, the step fails with `ACTION_MAY_HAVE_COMMITTED` and isn't retried: the input may have landed. An input the timeout ran out before (its target was still settling) isn't sent and fails with `OPERATION_TIMEOUT`. A long press that holds at least as long as the time left is refused before it's sent, as `NOT_ACTIONABLE`: raise the `actionTimeout`. A locate (the snapshot a locator or `Expect` takes) runs within the timeout of the call that takes it, so a device that stops answering fails the step with `ACTION_FAILED` (the engine's `OPERATION_TIMEOUT` as its `InnerException`) instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. An element that never shows up still fails as a locator timeout. A command a timeout or a cancel abandoned keeps running on the device, so the next attempt on that device waits for it to settle before it starts (within the launch timeout, which fails the start as cancelled when it runs out).
- An input that outlives its timeout. A tap, fill, key, swipe, or `Back` gets the `actionTimeout` (30 seconds by default). If the device doesn't answer in time, the step fails with `ACTION_MAY_HAVE_COMMITTED` and isn't retried: the input may have landed. An input the timeout ran out before (its target was still settling) isn't sent and fails with `OPERATION_TIMEOUT`. A long press that holds at least as long as the time left is refused before it's sent, as `NOT_ACTIONABLE`: raise the `actionTimeout`. A locate (the snapshot a locator or `Expect` takes), an observation, and a screenshot run within the timeout of the call that takes them, so a device that stops answering fails the step with `ACTION_FAILED` (the engine's `OPERATION_TIMEOUT` as its `InnerException`) instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. An element that never shows up still fails as a locator timeout. A command a timeout or a cancel abandoned keeps running on the device, so the next attempt on that device waits for it to settle before it starts (within the launch timeout, which fails the start as cancelled when it runs out).
- Device tests need a device. Keep them out of CI, for example with the `RealModel` category (`dotnet test --filter "TestCategory!=RealModel"`).
- Android can report `Android snapshot helper returned insufficient foreground app content`, and iOS can report that the app is not running: agent-device could not read the screen yet, often an app still loading after a cold start. On iOS this only counts until the first readable screen after `OpenAppAsync`, `RestartAsync`, or `OpenLinkAsync`, as on a physical iPhone that is still launching the app; an app that exits later fails at once. A waiting locator action or `Expect` waits through it, as for an element not there yet. If the screen is still unreadable when the wait runs out, the step fails with this message. Immediate reads such as `CountAsync` and `IsVisibleAsync` see no matches instead, so `Not.ToBeVisibleAsync` can pass while the screen is unreadable, including on an app that crashed before it rendered. An observation, such as the agent's next turn, fails on it at once.
8 changes: 7 additions & 1 deletion src/E2E.Mobile/DeviceActions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,13 @@ public Task SwipeAsync(DeviceSession session, ScrollDirection direction, Cancell
public Task<EngineScreenshot> ScreenshotAsync(DeviceSession session, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(session);
return session.MaskedScreenshotAsync(cancellationToken);
return session.MaskedScreenshotAsync(session.InputBudget("screenshot"), cancellationToken);
}

public Task<EngineScreenshot> ScreenshotAsync(DeviceSession session, TimeSpan timeout, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(session);
return session.MaskedScreenshotAsync(session.InputBudget("screenshot", timeout), cancellationToken);
}

// --- what the contract has no verb for ---
Expand Down
76 changes: 39 additions & 37 deletions src/E2E.Mobile/DeviceSession.Screen.cs
Original file line number Diff line number Diff line change
Expand Up @@ -503,7 +503,7 @@ private async Task DismissKeyboardByGestureAsync(InputBudget input, Cancellation
foreach (var horizontal in new[] { true, false })
{
var projected = SnapshotProjection.ProjectSnapshot(raw.Nodes, () => "keyboard", raw.Viewport);
var viewport = await ViewportForAsync(projected, cancellationToken).ConfigureAwait(false);
var viewport = await ViewportForAsync(projected, input, cancellationToken).ConfigureAwait(false);
var from = Keyboard.KeyboardDragStart(projected, viewport, KeyboardTop(raw, viewport))
?? throw new EngineException(
EngineErrorCodes.UnsupportedCapability,
Expand Down Expand Up @@ -564,60 +564,62 @@ await CommandAsync(
/// One screenshot into a temp directory that is removed once <paramref name="read"/> has taken what it needs from
/// the response or the file, even when the caller abandons the capture.
/// </summary>
private Task<T> CaptureScreenshotAsync<T>(Func<RawScreenshotResult, string, Task<T>> read, bool normalizeStatusBar, CancellationToken cancellationToken) =>
CommandAsync(
"screenshot",
async (client, token) =>
{
var directory = Directory.CreateTempSubdirectory("e2e-agent-device-");
try
{
var file = Path.Combine(directory.FullName, "screenshot.png");
var shot = await client.CallAsync(
DeviceMethods.CaptureScreenshot,
DeviceArgs.Of(("path", JsonValue.Create(file)), ("normalizeStatusBar", normalizeStatusBar ? JsonValue.Create(true) : null)),
token).ConfigureAwait(false);
return await read(
new RawScreenshotResult
{
Path = Text(shot?["path"]),
Width = Number(shot?["width"]),
Height = Number(shot?["height"]),
LogicalWidth = Number(shot?["logicalWidth"]),
LogicalHeight = Number(shot?["logicalHeight"]),
PixelDensity = Number(shot?["pixelDensity"]),
},
file).ConfigureAwait(false);
}
finally
private Task<T> CaptureScreenshotAsync<T>(Func<RawScreenshotResult, string, Task<T>> read, bool normalizeStatusBar, InputBudget budget, CancellationToken cancellationToken) =>
budget.ReadAsync(
() => CommandAsync(
"screenshot",
async (client, token) =>
{
var directory = Directory.CreateTempSubdirectory("e2e-agent-device-");
try
{
directory.Delete(recursive: true);
var file = Path.Combine(directory.FullName, "screenshot.png");
var shot = await client.CallAsync(
DeviceMethods.CaptureScreenshot,
DeviceArgs.Of(("path", JsonValue.Create(file)), ("normalizeStatusBar", normalizeStatusBar ? JsonValue.Create(true) : null)),
token).ConfigureAwait(false);
return await read(
new RawScreenshotResult
{
Path = Text(shot?["path"]),
Width = Number(shot?["width"]),
Height = Number(shot?["height"]),
LogicalWidth = Number(shot?["logicalWidth"]),
LogicalHeight = Number(shot?["logicalHeight"]),
PixelDensity = Number(shot?["pixelDensity"]),
},
file).ConfigureAwait(false);
}
catch (IOException)
finally
{
// The capture is best effort and so is its cleanup.
try
{
directory.Delete(recursive: true);
}
catch (IOException)
{
// The capture is best effort and so is its cleanup.
}
}
}
},
},
cancellationToken),
cancellationToken);

/// <summary>
/// The viewport as a screenshot for comparison: the status bar in a fixed state, and every secure node on the screen
/// painted over. Observes first so the regions describe the screen the pixels show. Throws when a secure field
/// cannot be covered, because an image that may hold a credential must not leave the engine.
/// </summary>
internal Task<EngineScreenshot> MaskedScreenshotAsync(CancellationToken cancellationToken) => MaskedScreenshotAsync(normalizeStatusBar: true, cancellationToken);
internal Task<EngineScreenshot> MaskedScreenshotAsync(InputBudget budget, CancellationToken cancellationToken) => MaskedScreenshotAsync(normalizeStatusBar: true, budget, cancellationToken);

/// <summary>
/// <see cref="MaskedScreenshotAsync(CancellationToken)"/> with the status bar left as the device shows it when
/// <see cref="MaskedScreenshotAsync(InputBudget, CancellationToken)"/> with the status bar left as the device shows it when
/// <paramref name="normalizeStatusBar"/> is false, as an evidence screenshot is kept (upstream's <c>maskedScreenshot</c>).
/// </summary>
internal async Task<EngineScreenshot> MaskedScreenshotAsync(bool normalizeStatusBar, CancellationToken cancellationToken)
internal async Task<EngineScreenshot> MaskedScreenshotAsync(bool normalizeStatusBar, InputBudget budget, CancellationToken cancellationToken)
{
var projected = Project(await SnapshotOrEmptyAsync(false, budget: null, cancellationToken).ConfigureAwait(false));
var data = await CaptureScreenshotAsync(async (shot, file) => await File.ReadAllBytesAsync(shot.Path ?? file, cancellationToken).ConfigureAwait(false), normalizeStatusBar, cancellationToken).ConfigureAwait(false);
var projected = Project(await SnapshotOrEmptyAsync(false, budget, cancellationToken).ConfigureAwait(false));
var data = await CaptureScreenshotAsync(async (shot, file) => await File.ReadAllBytesAsync(shot.Path ?? file, cancellationToken).ConfigureAwait(false), normalizeStatusBar, budget, cancellationToken).ConfigureAwait(false);
var viewport = _knownViewport;
var (masked, regions) = RedactSecure(data, projected, viewport)
?? throw new EngineException(EngineErrorCodes.EngineFailure, "a secure field on screen could not be masked; screenshot withheld", retryable: false);
Expand Down
Loading
Loading