Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions COMPATIBILITY.md

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion docs-site/src/content/docs/get-started/mobile.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,9 @@ await Expect.That(Device.Locator("role=navigation-bar id=About")).ToBeVisibleAsy
await Device.Locator("id=SW_VERSION_SPECIFIER role=button").TapAsync();
```

Every action on a node pins its ref to the capture that issued it, so a ref a newer capture replaced is never sent to a different element. If the control changed during the `TransitionMilliseconds` wait (500 ms by default), the action fails with a retryable `NODE_STALE` and a locator resolves it again within its timeout. A direct `Device` call such as `DragToAsync` does not retry, so it surfaces the `NODE_STALE`.
Every action on a node pins its ref to the capture that issued it, so a ref a newer capture replaced is never sent to a different element. If the control changed during the `TransitionMilliseconds` wait (500 ms by default), the action fails with a retryable `NODE_STALE` and a locator resolves it again within its timeout. A direct `Device` call such as `DragToAsync` does not retry, so it surfaces the `NODE_STALE`. Unmoved controls are acted on immediately only when keyboard visibility is unchanged: opening or closing the keyboard also uses the remaining `TransitionMilliseconds` budget. A `target_still_moving` refusal from agent-device that reports `dispatched: no` is a retryable `NODE_STALE` as well.

`FillAsync` and `ClearAsync` read the field again before typing. A field the fresh capture no longer lists, a changed input purpose or secure, disabled, or hidden state, or an incomplete capture fails with a retryable `NODE_STALE` before anything is typed, and the locator resolves it again. A node that cannot take text fails with `UNSUPPORTED_CAPABILITY`.

For anything `Device` does not cover, `Device.UnstableRaw` has an [agent-device](https://github.com/callstack/agent-device) client bound to the test's session, plus the session name and the device's id and name. It is unstable and may change or go away in any release:

Expand Down
2 changes: 1 addition & 1 deletion skills/e2e-dotnet/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ trip. The installed tool prints the same text with `e2e guide <topic>`
| Handling a failed run: the trace page, `report.json` for scripts, `E2ESession.Report()` for a session you run yourself, error codes and their fixes, a visible browser, replay misses, phase timings, the agent step table, every model call | `debugging` | [references/debugging.md](references/debugging.md) |
| Exploring an app toward a goal without a test file, with `e2e explore` or `Explore.ExploreAsync` | `explore` | [references/explore.md](references/explore.md) |
| Running isolated per-test sessions from your own runner: `E2EHost.OpenAsync`, `OpenHostOptions` (config value or file, target, `EngineFactory`), `host.OpenSessionAsync`, `E2ESession.CloseAsync` and the failures it returns, `Steps()` | `host` | [references/host.md](references/host.md) |
| Testing an iOS or Android app: `MobileEngine`, the `Device` fixture, picking a simulator (without `MobileOptions.Device`, `MobileEngine` skips booted devices another agent-device session holds, and `prepare` fails with `ENGINE_FAILURE` when every booted one is held), video of the device screen, `Device.UnstableRaw` (the test's agent-device session: name, device id and name, selection, and a raw `IDeviceClient` whose calls are not steps, replayed, or redacted; unstable; `INVALID_STATE` after the session is disposed), Limrun hosted devices, a MAUI app through the DevFlow agent, known limits, `Locator.PressAsync("Enter")` tapping an unfocused field first, and the agent's `open_app` returning to the app under test when it names no app, a Compose Multiplatform field on iOS (for an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text, that text is the field's `Placeholder`, the child is left out, and the name is the label before `, `; a non-empty `ContentDescription` stays the name, and that label is then the `Text`; any other field keeps its label and children), and `FillAsync` and `ClearAsync` refusing a node that is not an editable field with `UNSUPPORTED_CAPABILITY` (an iOS `other` that holds a value or only text is typed into) | `mobile` | [references/mobile.md](references/mobile.md) |
| Testing an iOS or Android app: `MobileEngine`, the `Device` fixture, picking a simulator (without `MobileOptions.Device`, `MobileEngine` skips booted devices another agent-device session holds, and `prepare` fails with `ENGINE_FAILURE` when every booted one is held), video of the device screen, `Device.UnstableRaw` (the test's agent-device session: name, device id and name, selection, and a raw `IDeviceClient` whose calls are not steps, replayed, or redacted; unstable; `INVALID_STATE` after the session is disposed), Limrun hosted devices, a MAUI app through the DevFlow agent, known limits, `Locator.PressAsync("Enter")` tapping an unfocused field first, and the agent's `open_app` returning to the app under test when it names no app, a Compose Multiplatform field on iOS (for an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text, that text is the field's `Placeholder`, the child is left out, and the name is the label before `, `; a non-empty `ContentDescription` stays the name, and that label is then the `Text`; any other field keeps its label and children), and `FillAsync` and `ClearAsync` refusing a node that is not an editable field with `UNSUPPORTED_CAPABILITY` (an iOS `other` that holds a value or only text is typed into), reading the field again before typing (a field the new capture no longer lists, an incomplete capture, or a changed input purpose or secure, disabled, or hidden state is a retryable `NODE_STALE` before anything is typed), looking at the screen again after the keyboard opens or closes (an unmoved control is acted on at once only when keyboard visibility is unchanged, else it waits out `TransitionMilliseconds`), and a `target_still_moving` refusal with `dispatched: no` being a retryable `NODE_STALE` | `mobile` | [references/mobile.md](references/mobile.md) |
| Testing a Windows desktop app: `WinAppEngine`, `WinAppTest`, the `Window` fixture, the `windows` target in `e2e explore` and `e2e mcp`, the interactive-desktop requirement, known limits | `windows` | [references/windows.md](references/windows.md) |
| Asked to bug bash or hunt for bugs: proving each bug with a repro test | `bug-bash` | [references/bug-bash.md](references/bug-bash.md) |

Expand Down
2 changes: 1 addition & 1 deletion skills/e2e-dotnet/references/mobile.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ Limits of the preview: the tests attach to the running app, so `Device.InstallAp
- Android needs `adb` from the Android SDK Platform-Tools on `PATH`, locally and for the Limrun tunnel.
- Building the sample iOS app needs `-p:ValidateXcodeVersion=false` when the installed Xcode is newer than the one the .NET iOS pack expects.
- Agent steps on a device are slower than in a browser. Raise `StepTimeout` to 90 seconds. The 30 second default timed out a two-field step.
- A ref belongs to the agent-device capture it came from, and every action pins it there (`@e4~s7`), so agent-device refuses a ref once a newer capture supersedes it. An action on a node that a newer capture replaced never acts on a different element: a test's action waits out `TransitionMilliseconds` for a control that arrived with the last action, and a node the fresh capture no longer lists is a retryable `NODE_STALE`, so the locator resolves again within its timeout. A direct `Device` call (`DragToAsync`, `LongPressAsync`) does not retry and surfaces it. `Device.DragToAsync` between nodes located in separate captures takes both refs from the newest one.
- A ref belongs to the agent-device capture it came from, and every action pins it there (`@e4~s7`), so agent-device refuses a ref once a newer capture supersedes it. An action on a node that a newer capture replaced never acts on a different element: a test's action waits out `TransitionMilliseconds` for a control that arrived with the last action, and a node the fresh capture no longer lists is a retryable `NODE_STALE`, so the locator resolves again within its timeout. A direct `Device` call (`DragToAsync`, `LongPressAsync`) does not retry and surfaces it. Unmoved controls are acted on immediately only when keyboard visibility is unchanged: opening or closing the keyboard also uses the remaining `TransitionMilliseconds` budget. A `target_still_moving` refusal that reports `dispatched: no` is a retryable `NODE_STALE`. `FillAsync` and `ClearAsync` read the field again before typing: a field the fresh capture no longer lists, a changed input purpose or secure, disabled, or hidden state, or an incomplete capture is a retryable `NODE_STALE` before anything is typed; a node that cannot take text is `UNSUPPORTED_CAPABILITY`. `Device.DragToAsync` between nodes located in separate captures takes both refs from the newest one.
- An input that outlives its timeout. A tap, fill, key, swipe, or `Back` gets the `actionTimeout` (30 seconds by default). If the device doesn't answer in time, the step fails with `ACTION_MAY_HAVE_COMMITTED` and isn't retried: the input may have landed. An input the timeout ran out before (its target was still settling) isn't sent and fails with `OPERATION_TIMEOUT`. A long press that holds at least as long as the time left is refused before it's sent, as `NOT_ACTIONABLE`: raise the `actionTimeout`. A locate (the snapshot a locator or `Expect` takes), an observation, and a screenshot run within the timeout of the call that takes them, so a device that stops answering fails the step with `ACTION_FAILED` (the engine's `OPERATION_TIMEOUT` as its `InnerException`) instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. An element that never shows up still fails as a locator timeout. A command a timeout or a cancel abandoned keeps running on the device, so the next attempt on that device waits for it to settle before it starts (within the launch timeout, which fails the start as cancelled when it runs out).
- Device tests need a device. Keep them out of CI, for example with the `RealModel` category (`dotnet test --filter "TestCategory!=RealModel"`).
- Android can report `Android snapshot helper returned insufficient foreground app content`, and iOS can report that the app is not running: agent-device could not read the screen yet, often an app still loading after a cold start. On iOS this only counts until the first readable screen after `OpenAppAsync`, `RestartAsync`, or `OpenLinkAsync`, as on a physical iPhone that is still launching the app; an app that exits later fails at once. A waiting locator action or `Expect` waits through it, as for an element not there yet. If the screen is still unreadable when the wait runs out, the step fails with this message. Immediate reads such as `CountAsync` and `IsVisibleAsync` see no matches instead, so `Not.ToBeVisibleAsync` can pass while the screen is unreadable, including on an app that crashed before it rendered. An observation, such as the agent's next turn, fails on it at once.
10 changes: 5 additions & 5 deletions src/E2E.Mobile/DeviceActions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ public async Task PerformAsync(DeviceSession session, SemanticNode node, Locator
return;
}

var before = session.LatestIndex;
var before = session.LatestScreen;
await session.ActAsync(
label,
async (client, token) =>
Expand All @@ -123,7 +123,7 @@ await session.ActAsync(
await session.WaitForTransitionAsync(entry, token).ConfigureAwait(false);
}

target = await session.RelocatedAsync(entry, input, token).ConfigureAwait(false);
target = await session.RelocatedAsync(entry, input, token, forInput: true).ConfigureAwait(false);
if (!target.TakesText)
{
throw RefuseFill(action, target);
Expand Down Expand Up @@ -265,7 +265,7 @@ private static async Task SwipeNodeAsync(IDeviceClient client, NodeBinding targe

private static async Task ScrollScreenAsync(DeviceSession session, ScrollDirection direction, InputBudget? input, CancellationToken cancellationToken)
{
var before = session.LatestIndex;
var before = session.LatestScreen;
await session.CommandAsync(
"swipe",
(client, token) => DeviceSession.SendInputAsync(
Expand All @@ -289,7 +289,7 @@ public Task PressAsync(DeviceSession session, string key, CancellationToken canc
private static async Task PressFocusedAsync(DeviceSession session, string key, InputBudget input, CancellationToken cancellationToken)
{
var typed = SoftKeyboardKey(key);
var before = session.LatestIndex;
var before = session.LatestScreen;
if (typed is null)
{
await session.CommandAsync(
Expand Down Expand Up @@ -384,7 +384,7 @@ private static async Task RunNodeAsync(
var input = session.InputBudget(kind);
var label = $"perform {kind}";
var entry = session.Resolve(node);
var before = session.LatestIndex;
var before = session.LatestScreen;
await session.ActAsync(
label,
async (client, token) =>
Expand Down
11 changes: 7 additions & 4 deletions src/E2E.Mobile/DeviceErrors.cs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ namespace E2E.Mobile;
/// <item><term>SESSION_NOT_FOUND, or a session-or-device-selector refusal</term><description>INVALID_STATE</description></item>
/// <item><term>UNSUPPORTED_OPERATION, NOT_IMPLEMENTED, UNSUPPORTED_PLATFORM, or "not supported on this device" under any code</term><description>UNSUPPORTED_CAPABILITY</description></item>
/// <item><term>a stale-ref refusal, by <c>details.reason</c> (action paths only)</term><description>NODE_STALE (retryable)</description></item>
/// <item><term><c>target_still_moving</c> with <c>details.dispatched: no</c></term><description>NODE_STALE (retryable)</description></item>
/// <item><term>the iOS runner busy, wedged, or past its watchdog; a snapshot the runner acquired but could not present</term><description>ENGINE_FAILURE, naming the runner and the recovery</description></item>
/// <item><term>anything else, agent-device's own timeouts included</term><description>ENGINE_FAILURE with the text</description></item>
/// </list>
Expand Down Expand Up @@ -202,9 +203,9 @@ public static async Task<T> RunCommandAsync<T>(string label, Func<CancellationTo
/// <summary>
/// Like <see cref="TranslateError"/>, but a ref agent-device refused as stale becomes retryable
/// <c>NODE_STALE</c>: nothing was dispatched, so the harness may re-observe and re-resolve the node
/// instead of failing the action. Keyed on the reason alone: agent-device marks a drag's stale ref
/// <c>dispatched: 'unknown'</c> after Android blocking-dialog recovery, although it refuses it before
/// the gesture.
/// instead of failing the action. Existing stale-ref reasons are sufficient on their own: Android dialog
/// recovery can mark a refused drag as <c>dispatched: unknown</c>. A <c>target_still_moving</c> refusal
/// additionally requires <c>dispatched: no</c>; missing or unknown dispatch is not retryable.
/// </summary>
public static Exception StaleOr(Exception cause, string operation, string? where = null)
{
Expand All @@ -213,7 +214,9 @@ public static Exception StaleOr(Exception cause, string operation, string? where
return cause;
}

if (Details(cause).GetValueOrDefault("reason") is string reason && StaleRefReasons.Contains(reason))
var details = Details(cause);
if (details.GetValueOrDefault("reason") is string reason
&& (StaleRefReasons.Contains(reason) || (reason == "target_still_moving" && details.GetValueOrDefault("dispatched") is "no")))
{
return new EngineException(EngineErrorCodes.NodeStale, $"{operation}: {Message(cause)}", retryable: true, cause);
}
Expand Down
4 changes: 2 additions & 2 deletions src/E2E.Mobile/DeviceSession.Input.cs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ public async Task PerformAtAsync(Point point, PointerAction action, Cancellation
PointerAction.SwipeTo swipe => swipe.Duration,
_ => (TimeSpan?)null,
};
var before = _latestIndex;
var before = _latestScreen;
await CommandAsync(
$"{PointerLabel(action)} at point",
(client, token) => SendInputAsync(input, () => DeviceActions.PointerInteractionAsync(client, point, action, ActionSettle(), token), token, hold),
Expand All @@ -45,7 +45,7 @@ public async Task TypeTextAsync(string text, bool replace = false, CancellationT
}

var input = InputBudget("type");
var before = _latestIndex;
var before = _latestScreen;
await CommandAsync(
"keyboard.type",
(client, token) => SendInputAsync(input, () => client.CallAsync(DeviceMethods.InteractionsType, DeviceArgs.Of(("text", JsonValue.Create(text))), token), token),
Expand Down
Loading
Loading