Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions COMPATIBILITY.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs-site/src/content/docs/docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,4 @@ await Screen.GetByTestId("feed").ScrollUntilVisibleAsync(row, new ScrollUntilVis

A locator reports an engine failure in the test's own codes, with the engine's error as `InnerException`: `NOT_ACTIONABLE`, `ACTION_MAY_HAVE_COMMITTED`, and `OPERATION_TIMEOUT` are `ACTION_FAILED`, a stale node or a missing frame is `NOT_FOUND`, an ambiguous frame is `STRICT_MODE`, and `INVALID_STATE` is `APP_NOT_OPEN`. `CANCELLED`, `UNSUPPORTED_CAPABILITY`, and `ENGINE_FAILURE` stay as the engine threw them.

A read or navigation that times out fails with `OPERATION_TIMEOUT`. It is not retried, even when the call log mentions a detached element or a navigation race. On a device, an input (a tap, key, swipe, or `Back`) that the device does not answer within `actionTimeout` fails with `ACTION_MAY_HAVE_COMMITTED` and is never repeated, because it may have landed; an input the timeout ran out before is not sent (`OPERATION_TIMEOUT`), and a long press that holds at least as long as the time left is refused (`NOT_ACTIONABLE`). See `EngineErrorCodes` in the [API reference](/api/).
A read or navigation that times out fails with `OPERATION_TIMEOUT`. It is not retried, even when the call log mentions a detached element or a navigation race. On a device, an input (a tap, key, swipe, or `Back`) that the device does not answer within `actionTimeout` fails with `ACTION_MAY_HAVE_COMMITTED` and is never repeated, because it may have landed; an input the timeout ran out before is not sent (`OPERATION_TIMEOUT`), and a long press that holds at least as long as the time left is refused (`NOT_ACTIONABLE`). A locate on a device runs within the timeout of the call that takes it, so a device that stops answering fails the step with `ACTION_FAILED` instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. See `EngineErrorCodes` in the [API reference](/api/).
7 changes: 6 additions & 1 deletion skills/e2e-dotnet/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,12 @@ lines you changed:
answer within `actionTimeout` fails with `ACTION_MAY_HAVE_COMMITTED` and is
never repeated; one the timeout ran out before is not sent and fails with
`OPERATION_TIMEOUT`; a long press that holds at least as long as the time left
is refused with `NOT_ACTIONABLE`. Reading the screen is not cut off.
is refused with `NOT_ACTIONABLE`. A locate on a device (a locator, `Expect`)
runs within the timeout of the call that takes it, so a device that stops
answering fails the step with `ACTION_FAILED` instead of holding it until the
test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their
last read extra time, up to their slowest read so far and within the test
timeout, so a screen that shows up during it still passes.
- Secrets never appear in test code. Declare them under `secrets` in the
config and read them with `Secrets.Get(name)`; accounts come from
`Credentials.User(name)`. Hand the `Secret` only to `FillAsync` or
Expand Down
2 changes: 1 addition & 1 deletion skills/e2e-dotnet/references/mobile.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,6 @@ Limits of the preview: the tests attach to the running app, so `Device.InstallAp
- Building the sample iOS app needs `-p:ValidateXcodeVersion=false` when the installed Xcode is newer than the one the .NET iOS pack expects.
- Agent steps on a device are slower than in a browser. Raise `StepTimeout` to 90 seconds. The 30 second default timed out a two-field step.
- A ref belongs to the agent-device capture it came from, and every action pins it there (`@e4~s7`), so agent-device refuses a ref once a newer capture supersedes it. An action on a node that a newer capture replaced never acts on a different element: a test's action waits out `TransitionMilliseconds` for a control that arrived with the last action, and a node the fresh capture no longer lists is a retryable `NODE_STALE`, so the locator resolves again within its timeout. A direct `Device` call (`DragToAsync`, `LongPressAsync`) does not retry and surfaces it. `Device.DragToAsync` between nodes located in separate captures takes both refs from the newest one.
- An input that outlives its timeout. A tap, fill, key, swipe, or `Back` gets the `actionTimeout` (30 seconds by default). If the device doesn't answer in time, the step fails with `ACTION_MAY_HAVE_COMMITTED` and isn't retried: the input may have landed. An input the timeout ran out before (its target was still settling) isn't sent and fails with `OPERATION_TIMEOUT`. A long press that holds at least as long as the time left is refused before it's sent, as `NOT_ACTIONABLE`: raise the `actionTimeout`. Finding the element isn't cut off, so a missing one still fails as a locator timeout. A command a timeout or a cancel abandoned keeps running on the device, so the next attempt on that device waits for it to settle before it starts (within the launch timeout, which fails the start as cancelled when it runs out).
- An input that outlives its timeout. A tap, fill, key, swipe, or `Back` gets the `actionTimeout` (30 seconds by default). If the device doesn't answer in time, the step fails with `ACTION_MAY_HAVE_COMMITTED` and isn't retried: the input may have landed. An input the timeout ran out before (its target was still settling) isn't sent and fails with `OPERATION_TIMEOUT`. A long press that holds at least as long as the time left is refused before it's sent, as `NOT_ACTIONABLE`: raise the `actionTimeout`. A locate (the snapshot a locator or `Expect` takes) runs within the timeout of the call that takes it, so a device that stops answering fails the step with `ACTION_FAILED` (the engine's `OPERATION_TIMEOUT` as its `InnerException`) instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. An element that never shows up still fails as a locator timeout. A command a timeout or a cancel abandoned keeps running on the device, so the next attempt on that device waits for it to settle before it starts (within the launch timeout, which fails the start as cancelled when it runs out).
- Device tests need a device. Keep them out of CI, for example with the `RealModel` category (`dotnet test --filter "TestCategory!=RealModel"`).
- Android can report `Android snapshot helper returned insufficient foreground app content`, and iOS can report that the app is not running: agent-device could not read the screen yet, often an app still loading after a cold start. On iOS this only counts until the first readable screen after `OpenAppAsync`, `RestartAsync`, or `OpenLinkAsync`, as on a physical iPhone that is still launching the app; an app that exits later fails at once. A waiting locator action or `Expect` waits through it, as for an element not there yet. If the screen is still unreadable when the wait runs out, the step fails with this message. Immediate reads such as `CountAsync` and `IsVisibleAsync` see no matches instead, so `Not.ToBeVisibleAsync` can pass while the screen is unreadable, including on an app that crashed before it rendered. An observation, such as the agent's next turn, fails on it at once.
6 changes: 3 additions & 3 deletions src/E2E.Mobile/DeviceActions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -123,15 +123,15 @@ await session.ActAsync(
await session.WaitForTransitionAsync(entry, token).ConfigureAwait(false);
}

target = await session.RelocatedAsync(entry, token).ConfigureAwait(false);
target = await session.RelocatedAsync(entry, input, token).ConfigureAwait(false);
if (!target.TakesText)
{
throw RefuseFill(action, target);
}
}
else
{
target = session.Origin == ActionOrigin.Test ? await session.SettledAsync(entry, token).ConfigureAwait(false) : entry;
target = session.Origin == ActionOrigin.Test ? await session.SettledAsync(entry, input, token).ConfigureAwait(false) : entry;
}

await RunAsync(session, client, target, action, settle, input, token).ConfigureAwait(false);
Expand Down Expand Up @@ -384,7 +384,7 @@ await session.ActAsync(
async (client, token) =>
{
var settle = session.ActionSettle();
var target = session.Origin == ActionOrigin.Test ? await session.SettledAsync(entry, token).ConfigureAwait(false) : entry;
var target = session.Origin == ActionOrigin.Test ? await session.SettledAsync(entry, input, token).ConfigureAwait(false) : entry;
await run(client, target, settle, input, token).ConfigureAwait(false);
},
cancellationToken).ConfigureAwait(false);
Expand Down
41 changes: 21 additions & 20 deletions src/E2E.Mobile/DeviceSession.Screen.cs
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ private void ClearLocated()
public async Task<IReadOnlyList<SemanticNode>> LocateAsync(LocatorExpression expression, TimeSpan timeout, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(expression);
var raw = await LocateSnapshotAsync(timeout, cancellationToken).ConfigureAwait(false);
var raw = await LocateSnapshotAsync(InputBudget("locate", timeout), cancellationToken).ConfigureAwait(false);
if (raw is null)
{
return [];
Expand Down Expand Up @@ -152,17 +152,18 @@ public async Task<IReadOnlyList<SemanticNode>> LocateAsync(LocatorExpression exp
/// <summary>
/// Snapshot for a locate; null for a screen agent-device could not read yet (an app still on its splash, or one iOS does not
/// report running yet after a launch). The harness polls a locate until its deadline, so that screen matches nothing and the
/// poll waits for it to render. A capture that leaves no budget for another poll fails with the reason instead, so a wait that
/// ends on an unreadable screen says so.
/// poll waits for it to render. When what is left of the budget cannot fit another capture as long as this one, the locate fails
/// with the reason instead: the budget bounds every capture, so an unreadable one never outlasts it, and the wait that ends on
/// an unreadable screen says so.
/// </summary>
private async Task<RawSnapshot?> LocateSnapshotAsync(TimeSpan timeout, CancellationToken cancellationToken)
private async Task<RawSnapshot?> LocateSnapshotAsync(InputBudget budget, CancellationToken cancellationToken)
{
var started = _engine.Clock.GetUtcNow();
try
{
return await SnapshotOrEmptyAsync(false, cancellationToken).ConfigureAwait(false);
return await SnapshotOrEmptyAsync(false, budget, cancellationToken).ConfigureAwait(false);
}
catch (EngineException cause) when (!cancellationToken.IsCancellationRequested && (DeviceErrors.IsUnreadableCapture(cause) || (_launching && DeviceErrors.IsAppNotRunning(cause))) && _engine.Clock.GetUtcNow() - started < timeout)
catch (EngineException cause) when (!cancellationToken.IsCancellationRequested && (DeviceErrors.IsUnreadableCapture(cause) || (_launching && DeviceErrors.IsAppNotRunning(cause))) && budget.Remaining >= _engine.Clock.GetUtcNow() - started)
{
return null;
}
Expand Down Expand Up @@ -349,9 +350,9 @@ a is not null && b is not null
/// the control landed: Android reports a window's frames while it slides in, so a ref resolved mid-flight would
/// send the action where the control was.
/// </summary>
internal async Task<NodeBinding> SettledAsync(NodeBinding entry, CancellationToken cancellationToken) =>
internal async Task<NodeBinding> SettledAsync(NodeBinding entry, InputBudget input, CancellationToken cancellationToken) =>
await WaitForTransitionAsync(entry, cancellationToken).ConfigureAwait(false)
? await RelocatedAsync(entry, cancellationToken).ConfigureAwait(false)
? await RelocatedAsync(entry, input, cancellationToken).ConfigureAwait(false)
: entry;

/// <summary>
Expand Down Expand Up @@ -392,9 +393,9 @@ internal async Task<bool> WaitForTransitionAsync(NodeBinding entry, Cancellation
}

/// <summary>The same control in a fresh snapshot. A snapshot that no longer lists it is a retryable <c>NODE_STALE</c>, so the locator resolves again.</summary>
internal async Task<NodeBinding> RelocatedAsync(NodeBinding entry, CancellationToken cancellationToken)
internal async Task<NodeBinding> RelocatedAsync(NodeBinding entry, InputBudget input, CancellationToken cancellationToken)
{
var projected = Project(await SnapshotOrEmptyAsync(false, cancellationToken).ConfigureAwait(false));
var projected = Project(await SnapshotOrEmptyAsync(false, input, cancellationToken).ConfigureAwait(false));
return Refind(entry, projected.Index) is { } found
? Bind(found, projected.Index, projected.RefsGeneration)
: throw new EngineException(EngineErrorCodes.NodeStale, $"node {entry.Id} changed during transition; re-resolve", retryable: true);
Expand Down Expand Up @@ -453,13 +454,13 @@ raw.KeyboardKind is "visible" or "derived" && raw.KeyboardFrame is { } frame
/// about the screen it never makes on a caller's behalf. The engine makes that guess, the way a user does and
/// Maestro's <c>hideKeyboard</c> does, in <see cref="DismissKeyboardByGestureAsync"/>.
/// </summary>
public Task DismissKeyboardAsync(CancellationToken cancellationToken) => DismissKeyboardAsync(cancellationToken, input: null);
public Task DismissKeyboardAsync(CancellationToken cancellationToken) => DismissKeyboardAsync(cancellationToken, InputBudget("device.dismissKeyboard"));

/// <summary>
/// <see cref="DismissKeyboardAsync(CancellationToken)"/> under an input's budget, or directly with none: the
/// <c>device.dismissKeyboard</c> fixture has no budget. Upstream's <c>dismissKeyboard(signal, input?)</c>.
/// <see cref="DismissKeyboardAsync(CancellationToken)"/> under an input's budget: the
/// <c>device.dismissKeyboard</c> fixture gets one of its own. Upstream's <c>dismissKeyboard(signal, input?)</c>.
/// </summary>
internal async Task DismissKeyboardAsync(CancellationToken cancellationToken, InputBudget? input)
internal async Task DismissKeyboardAsync(CancellationToken cancellationToken, InputBudget input)
{
var before = _latestIndex;
try
Expand Down Expand Up @@ -491,9 +492,9 @@ await CommandAsync(
/// drags fails naming what does work. Each look is the observation's own capture, sparse-tree retries included: a
/// one-node tree has no keys and would pass for a dismissed keyboard.
/// </summary>
private async Task DismissKeyboardByGestureAsync(InputBudget? input, CancellationToken cancellationToken)
private async Task DismissKeyboardByGestureAsync(InputBudget input, CancellationToken cancellationToken)
{
var raw = await DismissKeyboardTipAsync(await SnapshotAsync(false, cancellationToken).ConfigureAwait(false), input, cancellationToken).ConfigureAwait(false);
var raw = await DismissKeyboardTipAsync(await SnapshotAsync(false, input, cancellationToken).ConfigureAwait(false), input, cancellationToken).ConfigureAwait(false);
if (!KeyboardShowing(raw))
{
return;
Expand All @@ -515,7 +516,7 @@ await CommandAsync(
"keyboard.dismiss",
(client, token) => SendInputAsync(input, () => client.CallAsync(DeviceMethods.InteractionsSwipe, DeviceArgs.Of(("from", PointNode(from)), ("to", PointNode(to))), token), token),
cancellationToken).ConfigureAwait(false);
raw = await SnapshotAsync(false, cancellationToken).ConfigureAwait(false);
raw = await SnapshotAsync(false, input, cancellationToken).ConfigureAwait(false);
if (!KeyboardShowing(raw))
{
return;
Expand All @@ -533,7 +534,7 @@ await CommandAsync(
/// button is the only way past it, and Maestro's <c>hideKeyboard</c> presses it too. Answers the capture after the
/// press, or the one given when no tip is showing.
/// </summary>
private async Task<RawSnapshot> DismissKeyboardTipAsync(RawSnapshot raw, InputBudget? input, CancellationToken cancellationToken)
private async Task<RawSnapshot> DismissKeyboardTipAsync(RawSnapshot raw, InputBudget input, CancellationToken cancellationToken)
{
var tip = raw.Nodes.FirstOrDefault(node => (node.Label ?? node.Value ?? "").StartsWith("Speed up your typing", StringComparison.Ordinal));
if (tip?.Rect is not { } tipRect)
Expand All @@ -552,7 +553,7 @@ await CommandAsync(
"keyboard.dismiss",
(client, token) => SendInputAsync(input, () => client.CallAsync(DeviceMethods.InteractionsPress, DeviceArgs.Of(("x", JsonValue.Create(centre.X)), ("y", JsonValue.Create(centre.Y))), token), token),
cancellationToken).ConfigureAwait(false);
return await SnapshotAsync(false, cancellationToken).ConfigureAwait(false);
return await SnapshotAsync(false, input, cancellationToken).ConfigureAwait(false);
}

internal static JsonObject PointNode(Point point) => new() { ["x"] = point.X, ["y"] = point.Y };
Expand Down Expand Up @@ -615,7 +616,7 @@ private Task<T> CaptureScreenshotAsync<T>(Func<RawScreenshotResult, string, Task
/// </summary>
internal async Task<EngineScreenshot> MaskedScreenshotAsync(bool normalizeStatusBar, CancellationToken cancellationToken)
{
var projected = Project(await SnapshotOrEmptyAsync(false, cancellationToken).ConfigureAwait(false));
var projected = Project(await SnapshotOrEmptyAsync(false, budget: null, cancellationToken).ConfigureAwait(false));
var data = await CaptureScreenshotAsync(async (shot, file) => await File.ReadAllBytesAsync(shot.Path ?? file, cancellationToken).ConfigureAwait(false), normalizeStatusBar, cancellationToken).ConfigureAwait(false);
var viewport = _knownViewport;
var (masked, regions) = RedactSecure(data, projected, viewport)
Expand Down
Loading
Loading