From 84ee95c09741adf02a26ba00464c2bb320186267 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dar=C3=ADo=20Kondratiuk?= Date: Sun, 1 Nov 2026 18:49:57 -0300 Subject: [PATCH 1/2] Revalidate mobile input bindings before fill and clear, and reobserve after keyboard transitions (upstream #1410, #1297) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1yaZFgZ3FEJkYScfkK --- COMPATIBILITY.md | 5 +- .../src/content/docs/get-started/mobile.mdx | 4 +- skills/e2e-dotnet/references/mobile.md | 2 +- src/E2E.Mobile/DeviceActions.cs | 10 +- src/E2E.Mobile/DeviceErrors.cs | 11 +- src/E2E.Mobile/DeviceSession.Input.cs | 4 +- src/E2E.Mobile/DeviceSession.Screen.cs | 58 ++-- src/E2E.Mobile/DeviceSession.cs | 7 +- src/E2E.Mobile/MobileOptions.cs | 4 +- src/E2E.Mobile/RawNode.cs | 6 + .../Engine/DeterministicActionsTests.cs | 132 ++++++++ .../Engine/InputFreshnessTests.cs | 284 +++++++++++++++++- .../Errors/ErrorTranslationTests.cs | 16 + 13 files changed, 501 insertions(+), 42 deletions(-) diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index dd713b9ba..059eb9da5 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -326,7 +326,7 @@ The port installs the skill with `npx skills add hardkoded/e2e-dotnet` and regis - Upstream's iOS locate (`2b2b845`, tester-army/e2e#1281) is ported. Just after `OpenAppAsync`, `RestartAsync` or `OpenLinkAsync`, a snapshot that iOS refuses with `details.runnerErrorCode` of `APP_NOT_RUNNING` (`DeviceErrors.IsAppNotRunning`) matches nothing in a locate (locator waits and `Expect` only; an observation still fails at once), like an unreadable Android capture, until a snapshot succeeds (`DeviceSession._launching`, upstream's `launching`). A wait that runs out while the app is still down fails with `ENGINE_FAILURE` and that reason. An app that exits after its first readable screen still fails at once. - Upstream's Android test id match without the `:id/` prefix (`feat(mobile): match Android test ids without their package prefix`, `2791786`) is in `DeviceLocator`, which `DeviceSession.LocateAsync` now calls, so `Screen.GetByTestId("greet")` matches `:id/greet` on Android when nothing matches the full id. - The core does not say who calls `PerformAsync`, so `DeviceSession.Origin` does. `ActionOrigin.Agent` (the default) settles after each action. `ActionOrigin.Test` never settles and waits out a transition (`TransitionMilliseconds`) for a control that arrived with the last action. The core `LocatorAction` has no long press, hover, drag, or swipe momentum, so `LongPressAsync`, `HoverAsync`, `DragToAsync`, `SwipeAsync(node, direction, momentum)`, and `PerformAtAsync(point, PointerAction)` are methods of `DeviceSession`. `ScrollIntoView`, `Select`, and `PressSequentially` are `UNSUPPORTED_CAPABILITY`. -- Upstream's mobile ref generations (`fix(mobile): pin each action's ref to the snapshot it came from`, `eeacc8a`, tester-army/e2e#1413, and `fix(mobile): reject stale refs after transition refresh`, `4af1aa8`, tester-army/e2e#1300) are ported. `RawSnapshot.RefsGeneration` (upstream `RawSnapshot.refsGeneration`) travels through `ProjectedSnapshot.RefsGeneration` into `NodeBinding.RefsGeneration`, and `DeviceSession.ActionTarget` (`actionTarget`) sends `@~s` when the capture reported one, so agent-device refuses a superseded ref (`ref_generation_mismatch`, a retryable `NODE_STALE`). `DeviceSession.Current` is `current`: `DragToAsync` takes both refs from the newest capture, finding a binding again where it was (`SamePlace`) and failing with `NODE_STALE` before sending when no node is there. `RelocatedAsync` throws `NODE_STALE` when the fresh snapshot lists no such control, where it used to act on the binding as it was. The port keeps `_latestScreen` beside `_latestIndex` for the newest capture's generation (upstream's `latestScreen` holds the projected screen). Upstream's `input` flag of `relocated` (a truncated snapshot or changed security facts refuse an input) has no counterpart in the port. +- Upstream's mobile ref generations (`fix(mobile): pin each action's ref to the snapshot it came from`, `eeacc8a`, tester-army/e2e#1413, and `fix(mobile): reject stale refs after transition refresh`, `4af1aa8`, tester-army/e2e#1300) are ported. `RawSnapshot.RefsGeneration` (upstream `RawSnapshot.refsGeneration`) travels through `ProjectedSnapshot.RefsGeneration` into `NodeBinding.RefsGeneration`, and `DeviceSession.ActionTarget` (`actionTarget`) sends `@~s` when the capture reported one, so agent-device refuses a superseded ref (`ref_generation_mismatch`, a retryable `NODE_STALE`). `DeviceSession.Current` is `current`: `DragToAsync` takes both refs from the newest capture, finding a binding again where it was (`SamePlace`) and failing with `NODE_STALE` before sending when no node is there. `RelocatedAsync` throws `NODE_STALE` when the fresh snapshot lists no such control, where it used to act on the binding as it was. `_latestScreen` is upstream's `latestScreen`, the projected screen of the newest capture. - `Device` is a plain class (`session.Device`) with `...Async` methods that take a cancellation token per call. Each `Device` call is a `resource` step named `device.`, with upstream's label (the state, mode, permission, orientation, pose, or alert action; the build path or the link with its query and fragment cut; ` chars` for clipboard text), through `EngineFixtureContext.RunOperationAsync` (.NET-only, upstream's `FixtureRecorder.run`); a `Device` on a session no test harness started runs unrecorded. The call is not wrapped in the action timeout, as upstream's is. `appearance`, `network`, `alert`, and biometric results are enums. A permission name or state cannot be unknown, so the `INVALID_ARGUMENT` check for one is not needed. - The sample (`samples/MobileApp.E2ETests`) derives from `MobileAppTest` (see [MobileAppTest](#mobileapptest-not-in-upstream)). It launches the app with `Device.OpenAppAsync`, because the core `App.OpenAsync` needs a URL, and reads the `Device` fixture from `MobileEngine.ActiveSessions`, because the core session does not expose the engine's session. It was run against an iPhone 17 simulator. The Android path is not run. - `MobileTools.Create` returns `AgentTool`s by name (the port's `defineTool`), with `Platforms` as the strings the engines report (`ios`, `android`). Pass `MobileTools.Create(engine)` as the agent's tools (`AgentTools` on `E2ETest`), and the agent offers `open_app`, `swipe`, and `alert` on the platforms of the engines it was built from. `open_app` without an `app` returns to the target's app, and refuses with `INVALID_STATE` when the target pins none. `ai-imports` has no counterpart: there is no `ai` package. @@ -338,7 +338,8 @@ The port installs the skill with `npx skills add hardkoded/e2e-dotnet` and regis - Upstream's `device.unstable_raw` (`feat(mobile): device.unstable_raw exposes the test's agent-device session`, `c3c4238`, tester-army/e2e#1380) is `Device.UnstableRaw`. Upstream's `DeviceSession` interface (`device.ts`) is `RawDeviceSession`, a .NET-only name, because the port's `DeviceSession` is the surface (upstream's `AgentDeviceSurface`) and a public type of that name exists. Its `client` is the session's `IDeviceClient` (the port's stand-in for `AgentDeviceClient`), `selection` is `DeviceSelection`, and `platform` is `MobilePlatform`. `BoundDevice` (`surface.ts`) is internal, set by the boot from the `id` and `device` of the `devices.boot` answer (`DeviceSession.BoundDevice`, upstream's `boundDevice`), and cleared when the session closes. Reading `UnstableRaw`, or `Client` of a handle taken earlier, after the session closed is `INVALID_STATE`. The client is `DeviceCommands.Translate`, which refuses a method it does not list, so `DeviceMethods.AppsList` (`apps.list`, the daemon's `apps` command) is added for upstream's reference example; the other methods of upstream's client stay unsupported until code calls them. Tests: the three `it` of `describe('unstable_raw')` in `Engine/DeviceFixtureUnstableRawTests`; the first is `Partial_`, since the port's fixture has no map of declared operations for `expect(declared).not.toHaveProperty('unstable_raw')`. Not ported: `unstable_raw reaches the test session through agent-device` (`apps/mobile-benchmark/tests/device-fixture.e2e.ts`), which needs the mobile benchmark app, a React Native app the port lacks. - Upstream's known-viewport fix (`fix(mobile): keep the known viewport when the app reads smaller than the screen`, `7674a2c`, tester-army/e2e#1291) is ported. `DeviceSession.Project` learns a snapshot's viewport only when it is plainly the screen's: the first one the session reads, the one it knows, or that one rotated (`SameScreen`, upstream's `sameScreen`; two sizes within a point are one screen, since rects from the tree carry fractions). `ViewportForAsync` (upstream's `viewportFor`) checks a tree of any other size against the device's screenshot, in either orientation, before it replaces the known one: a screenshot reports the portrait size even in landscape, so it confirms a size but cannot pick the orientation. The size it checks against is `DeviceSupport.LogicalScreenSize` (upstream's `logicalScreenSize`): the screenshot's logical dimensions, else its pixel dimensions over its density. A simulator's screenshot reports those; a real device's reports neither, so there the tree's size is taken as it comes. A screenshot that fails during that check fails the observation, as upstream's does. `TraceScreen` and `MaskedScreenshotAsync` size the trace screen and the secure-field masks by the known viewport, not the tree's. On iOS, a swipe from a sheet's grabber can pull down Notification Center, and the app's tree behind it reported a third of the screen (130x281 on a 390x844 device), which misplaced every rect and tap after it. Upstream records this only in its changeset; its skill references and docs say nothing about it, so `SKILL.md` and `docs/` have nothing to mirror. Tests: in `Engine/ObservationTests`, 'keeps the known viewport for an app drawn smaller than the screen, and follows a rotation' (new) and 'measures the screen in the viewport the snapshot publishes, and in its tree when it publishes none' (split into two harnesses, as upstream's); in `Engine/DeviceFixtureDismissingAKeyboardThatHasNoDismissKeyTests`, 'drags at the centre of a screen rotated since the last observation' (new), since the keyboard drag measures its start through `ViewportForAsync` too. - Upstream's input budget (`fix(mobile): bound each input by its budget`, `14b77d3`, tester-army/e2e#1073) is ported. An input (a tap or other node action, a point action, typing, a key, `back`, a keyboard dismissal, and the node `LongPressAsync`, `HoverAsync`, `DragToAsync`, and `SwipeAsync` the port has in place of upstream's `longPress`, `hover`, `dragTo`, and `swipe` actions) runs its device commands under `InputBudget` (`DeviceSupport.cs`, upstream's `support.ts`): with no budget left the input is not sent and fails with `OPERATION_TIMEOUT`, an input that does not answer in time fails with `ACTION_MAY_HAVE_COMMITTED`, one sent with almost no budget left still gets 250 ms to answer, a hold (a long press or timed swipe) at least as long as the budget left is refused with `NOT_ACTIONABLE`, and each command of a multi-command input (a key on an unfocused field) is sent on its own. Launches and artifacts are unbounded. Snapshots are bounded since #869 (see the next bullet). Differences: upstream reads the budget per call, from `operation.timeoutMs`, which is the call's `timeout` or `actionTimeout`. The port's session calls carry no timeout, so every input gets `DeviceSession.ActionTimeout`, which `MobileStartInfo.ActionTimeout` sets (`EngineStartOptions.ActionTimeout` fills it), as the web engine's operations get the action timeout. `DeviceSession.InputBudget` is upstream's `inputBudget` and `DeviceSession.SendInputAsync` its `sendInput`. `InputBudget`'s `track` is an `Action` that registers the task (upstream's returns the promise it is given); see [In-flight commands](#in-flight-commands). The `OperationContext.signal` doc change in `docs/reference/engine.mdx` and `contract.ts` has no counterpart: the port has no `OperationContext`. Tests: in `Engine/ObservationTests`, 'ends %s, hung on %s, at its budget as possibly committed' (a theory of the same seven entries; the screen swipe is a `perform` on the root), 'sends nothing once a settling target spent the budget, and no keystroke once a focus tap used it up', and 'refuses a long press that holds longer than its timeout leaves, before sending it' (also on the node `LongPressAsync`); in `Support/SupportHelpersTests`, 'sends an input with what is left of its budget, and reports one cut off as possibly committed'. These run on real timers of a few hundred milliseconds, where upstream's fake timers do not wait. 'lets a locate outlive its budget' was replaced in #869. -- Upstream's Compose Multiplatform text fields on iOS (`mobile reads Compose Multiplatform text fields on iOS`, `75ae2ab`, tester-army/e2e#1427) are ported in part. `SnapshotProjection.ProjectSnapshot` reads the placeholder an empty iOS field folds into its label (upstream's `foldedPlaceholder` in `nodes.ts`, `FoldedPlaceholder` here): for an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text, that text is the field's `Placeholder`, the child is left out of the tree, and the name is the label before `, ` (a non-empty `ContentDescription` stays the name, and that label is then the `Text`). A field of any other shape keeps its label and children as reported. The private record `Folded` is a .NET-only name for upstream's inline `{ label, placeholder, child }` object. Test: 'reads the placeholder an empty Compose Multiplatform field folds into its label' in `Nodes/SnapshotProjectionTests`. `mayTakeText` and `NodeBinding.takesText` (`nodes.ts`, `surface.ts`) are ported (#850): `SnapshotProjection.MayTakeText` and `NodeBinding.TakesText`, set in `Bind` from the node with its children. `DeviceActions.PerformAsync` refuses a `fill` or `clear` whose node does not take text with `UNSUPPORTED_CAPABILITY` (the message of upstream's `refuseFill`, `RefuseFill` here), before the action and again on the binding found by a fresh snapshot taken just before typing (upstream's `relocated`; `DeviceSession.RelocatedAsync`, with the transition wait split out as `WaitForTransitionAsync`, upstream's `waitForTransition`). Tests: 'Acts_on_an_unclassified_iOS_field', 'Rejects_an_unclassified_iOS_element', and 'Rejects_an_unclassified_iOS_element_that_gained_a_control_before_typing' in `Engine/InputFreshnessTests`. Differences: a fill or clear now reads the field again, so it costs one more `capture.snapshot` than before. Upstream's `relocated` also refuses, as `NODE_STALE`, a truncated capture or a changed input purpose, disabled, secure, or hidden state before typing; the port has no such check yet, so only the `takesText` re-check of this change is ported. The other `it`s of upstream's `describe('input freshness')` are not part of this change. +- Upstream's Compose Multiplatform text fields on iOS (`mobile reads Compose Multiplatform text fields on iOS`, `75ae2ab`, tester-army/e2e#1427) are ported in part. `SnapshotProjection.ProjectSnapshot` reads the placeholder an empty iOS field folds into its label (upstream's `foldedPlaceholder` in `nodes.ts`, `FoldedPlaceholder` here): for an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text, that text is the field's `Placeholder`, the child is left out of the tree, and the name is the label before `, ` (a non-empty `ContentDescription` stays the name, and that label is then the `Text`). A field of any other shape keeps its label and children as reported. The private record `Folded` is a .NET-only name for upstream's inline `{ label, placeholder, child }` object. Test: 'reads the placeholder an empty Compose Multiplatform field folds into its label' in `Nodes/SnapshotProjectionTests`. `mayTakeText` and `NodeBinding.takesText` (`nodes.ts`, `surface.ts`) are ported (#850): `SnapshotProjection.MayTakeText` and `NodeBinding.TakesText`, set in `Bind` from the node with its children. `DeviceActions.PerformAsync` refuses a `fill` or `clear` whose node does not take text with `UNSUPPORTED_CAPABILITY` (the message of upstream's `refuseFill`, `RefuseFill` here), before the action and again on the binding found by a fresh snapshot taken just before typing (upstream's `relocated`; `DeviceSession.RelocatedAsync`, with the transition wait split out as `WaitForTransitionAsync`, upstream's `waitForTransition`). Tests: 'Acts_on_an_unclassified_iOS_field', 'Rejects_an_unclassified_iOS_element', and 'Rejects_an_unclassified_iOS_element_that_gained_a_control_before_typing' in `Engine/InputFreshnessTests`. Differences: a fill or clear now reads the field again, so it costs one more `capture.snapshot` than before. Upstream's `relocated` `input` flag is ported with #751. +- Upstream's mobile input revalidation (`fix(mobile): revalidate input bindings before fill and clear`, `b309bcd`, tester-army/e2e#1410) and keyboard transitions (`fix(mobile): reobserve targets after keyboard transitions`, `bdd7315`, tester-army/e2e#1297) are ported. `DeviceSession.RelocatedAsync` takes `forInput` (upstream's `input` flag of `relocated`; `input` is the `InputBudget` here). With it, a truncated or sparse capture, a node the capture no longer lists, or a changed `InputPurpose`, `Disabled`, `Secure`, or `Hidden` is a retryable `NODE_STALE` before anything is typed, and `DeviceActions.PerformAsync` sets it for `fill` and `clear`. `ProjectedSnapshot.KeyboardVisible` (set in `Project` from `KeyboardShowing`) and `NodeBinding.KeyboardVisible` stand for upstream's `ProjectedScreen` type (`ProjectedSnapshot & { keyboardVisible }`), which C# has no intersection type for. `LatestScreen` and `MarkAction(ProjectedSnapshot?)` replace `LatestIndex` and the index-based `MarkAction` (upstream's `latestScreen` and `markAction(before: ProjectedScreen)`), and `WaitForTransitionAsync` acts at once on an unmoved control only when the keyboard state of the screen before the action equals the control's. `DeviceErrors.StaleOr` also maps `target_still_moving` with `details.dispatched: no` to a retryable `NODE_STALE`. The `agent-input-stale.test.ts` file of #1410 is already ported in part (`AgentInputStale/SecretFillStaleRetryTests`); its replay half still needs a project runner, `credentials`, and the `secret.fill` policy event, which the port lacks. Tests: the `input freshness` `it`s in `Engine/InputFreshnessTests`, the three keyboard `it`s in `Engine/DeterministicActionsTests`, and 'retries a moving target only when no input was dispatched' in `Errors/ErrorTranslationTests`. - Upstream's WebDriver element-type fix (`fix(mobile): read XCUIElementType class names as their XCTest types`, `3f9a7d8`, tester-army/e2e#1133) is ported. `SnapshotProjection.NormalizeKind` is upstream's `normalizeKind` in `nodes.ts`: it drops `^XCUIElementType(?=[A-Z])` (`XcuiElementTypePrefix`, a .NET `[GeneratedRegex]` for upstream's inline regex) before it maps the type, so the `XCUIElementTypeButton` that agent-device's WebDriver runtime reports on hosted Appium clouds reads as `Button`. `DeviceSession.KeyboardShowing` is upstream's `keyboardShowing` in `surface.ts`: it routes node types through `NormalizeKind`, so an `XCUIElementTypeKeyboard` counts as a shown keyboard. Upstream records this only in its changeset; its skill references and docs say nothing about it, so `SKILL.md` and `docs/` have nothing to mirror. Tests: in `Nodes/SnapshotProjectionTests`, 'reads the XCUIElementType class names a WebDriver session reports as the native XCTest types'; in `Engine/DeviceFixtureDismissingAKeyboardThatHasNoDismissKeyTests`, 'reads an XCUIElementTypeKeyboard from a WebDriver session as a shown keyboard'. - Upstream's held-device skip (`mobile: skip booted devices another agent-device session holds`, `9e53c40`, tester-army/e2e#1379) is ported. `BootedDevices` (`DevicePool.cs`, upstream's class in `pool.ts`) skips a booted device whose inventory entry has a `claimedBy.session` that is not one of this target's own slot sessions, logs the skipped ones, and fails `prepare` with a non-retryable `ENGINE_FAILURE` when every booted device is held. `BootedDevice` is upstream's interface of that name, as a record. `LifecycleTests.Skips_a_booted_device_another_agent_device_session_holds_keeps_one_its_own_slot_session_holds_and_fails_when_every_booted_one_is_held` ports the `lifecycle` `it` of `engine.test.ts`, whole. - Not ported: the tool-pixels test (needs the full runner and a scripted model), and the `device.openApp` check of an unknown permission name. diff --git a/docs-site/src/content/docs/get-started/mobile.mdx b/docs-site/src/content/docs/get-started/mobile.mdx index e9f6fc737..6c1b5efb5 100644 --- a/docs-site/src/content/docs/get-started/mobile.mdx +++ b/docs-site/src/content/docs/get-started/mobile.mdx @@ -96,7 +96,9 @@ await Expect.That(Device.Locator("role=navigation-bar id=About")).ToBeVisibleAsy await Device.Locator("id=SW_VERSION_SPECIFIER role=button").TapAsync(); ``` -Every action on a node pins its ref to the capture that issued it, so a ref a newer capture replaced is never sent to a different element. If the control changed during the `TransitionMilliseconds` wait (500 ms by default), the action fails with a retryable `NODE_STALE` and a locator resolves it again within its timeout. A direct `Device` call such as `DragToAsync` does not retry, so it surfaces the `NODE_STALE`. +Every action on a node pins its ref to the capture that issued it, so a ref a newer capture replaced is never sent to a different element. If the control changed during the `TransitionMilliseconds` wait (500 ms by default), the action fails with a retryable `NODE_STALE` and a locator resolves it again within its timeout. A direct `Device` call such as `DragToAsync` does not retry, so it surfaces the `NODE_STALE`. Unmoved controls are acted on immediately only when keyboard visibility is unchanged: opening or closing the keyboard also uses the remaining `TransitionMilliseconds` budget. A `target_still_moving` refusal from agent-device that reports `dispatched: no` is a retryable `NODE_STALE` as well. + +`FillAsync` and `ClearAsync` read the field again before typing. A field the fresh capture no longer lists, a changed input purpose or secure, disabled, or hidden state, or an incomplete capture fails with a retryable `NODE_STALE` before anything is typed, and the locator resolves it again. A node that cannot take text fails with `UNSUPPORTED_CAPABILITY`. For anything `Device` does not cover, `Device.UnstableRaw` has an [agent-device](https://github.com/callstack/agent-device) client bound to the test's session, plus the session name and the device's id and name. It is unstable and may change or go away in any release: diff --git a/skills/e2e-dotnet/references/mobile.md b/skills/e2e-dotnet/references/mobile.md index a36185ad1..d2dee2dbb 100644 --- a/skills/e2e-dotnet/references/mobile.md +++ b/skills/e2e-dotnet/references/mobile.md @@ -174,7 +174,7 @@ Limits of the preview: the tests attach to the running app, so `Device.InstallAp - Android needs `adb` from the Android SDK Platform-Tools on `PATH`, locally and for the Limrun tunnel. - Building the sample iOS app needs `-p:ValidateXcodeVersion=false` when the installed Xcode is newer than the one the .NET iOS pack expects. - Agent steps on a device are slower than in a browser. Raise `StepTimeout` to 90 seconds. The 30 second default timed out a two-field step. -- A ref belongs to the agent-device capture it came from, and every action pins it there (`@e4~s7`), so agent-device refuses a ref once a newer capture supersedes it. An action on a node that a newer capture replaced never acts on a different element: a test's action waits out `TransitionMilliseconds` for a control that arrived with the last action, and a node the fresh capture no longer lists is a retryable `NODE_STALE`, so the locator resolves again within its timeout. A direct `Device` call (`DragToAsync`, `LongPressAsync`) does not retry and surfaces it. `Device.DragToAsync` between nodes located in separate captures takes both refs from the newest one. +- A ref belongs to the agent-device capture it came from, and every action pins it there (`@e4~s7`), so agent-device refuses a ref once a newer capture supersedes it. An action on a node that a newer capture replaced never acts on a different element: a test's action waits out `TransitionMilliseconds` for a control that arrived with the last action, and a node the fresh capture no longer lists is a retryable `NODE_STALE`, so the locator resolves again within its timeout. A direct `Device` call (`DragToAsync`, `LongPressAsync`) does not retry and surfaces it. Unmoved controls are acted on immediately only when keyboard visibility is unchanged: opening or closing the keyboard also uses the remaining `TransitionMilliseconds` budget. A `target_still_moving` refusal that reports `dispatched: no` is a retryable `NODE_STALE`. `FillAsync` and `ClearAsync` read the field again before typing: a field the fresh capture no longer lists, a changed input purpose or secure, disabled, or hidden state, or an incomplete capture is a retryable `NODE_STALE` before anything is typed; a node that cannot take text is `UNSUPPORTED_CAPABILITY`. `Device.DragToAsync` between nodes located in separate captures takes both refs from the newest one. - An input that outlives its timeout. A tap, fill, key, swipe, or `Back` gets the `actionTimeout` (30 seconds by default). If the device doesn't answer in time, the step fails with `ACTION_MAY_HAVE_COMMITTED` and isn't retried: the input may have landed. An input the timeout ran out before (its target was still settling) isn't sent and fails with `OPERATION_TIMEOUT`. A long press that holds at least as long as the time left is refused before it's sent, as `NOT_ACTIONABLE`: raise the `actionTimeout`. A locate (the snapshot a locator or `Expect` takes), an observation, and a screenshot run within the timeout of the call that takes them, so a device that stops answering fails the step with `ACTION_FAILED` (the engine's `OPERATION_TIMEOUT` as its `InnerException`) instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. An element that never shows up still fails as a locator timeout. A command a timeout or a cancel abandoned keeps running on the device, so the next attempt on that device waits for it to settle before it starts (within the launch timeout, which fails the start as cancelled when it runs out). - Device tests need a device. Keep them out of CI, for example with the `RealModel` category (`dotnet test --filter "TestCategory!=RealModel"`). - Android can report `Android snapshot helper returned insufficient foreground app content`, and iOS can report that the app is not running: agent-device could not read the screen yet, often an app still loading after a cold start. On iOS this only counts until the first readable screen after `OpenAppAsync`, `RestartAsync`, or `OpenLinkAsync`, as on a physical iPhone that is still launching the app; an app that exits later fails at once. A waiting locator action or `Expect` waits through it, as for an element not there yet. If the screen is still unreadable when the wait runs out, the step fails with this message. Immediate reads such as `CountAsync` and `IsVisibleAsync` see no matches instead, so `Not.ToBeVisibleAsync` can pass while the screen is unreadable, including on an app that crashed before it rendered. An observation, such as the agent's next turn, fails on it at once. diff --git a/src/E2E.Mobile/DeviceActions.cs b/src/E2E.Mobile/DeviceActions.cs index 78e303443..df4f44910 100644 --- a/src/E2E.Mobile/DeviceActions.cs +++ b/src/E2E.Mobile/DeviceActions.cs @@ -108,7 +108,7 @@ public async Task PerformAsync(DeviceSession session, SemanticNode node, Locator return; } - var before = session.LatestIndex; + var before = session.LatestScreen; await session.ActAsync( label, async (client, token) => @@ -123,7 +123,7 @@ await session.ActAsync( await session.WaitForTransitionAsync(entry, token).ConfigureAwait(false); } - target = await session.RelocatedAsync(entry, input, token).ConfigureAwait(false); + target = await session.RelocatedAsync(entry, input, token, forInput: true).ConfigureAwait(false); if (!target.TakesText) { throw RefuseFill(action, target); @@ -265,7 +265,7 @@ private static async Task SwipeNodeAsync(IDeviceClient client, NodeBinding targe private static async Task ScrollScreenAsync(DeviceSession session, ScrollDirection direction, InputBudget? input, CancellationToken cancellationToken) { - var before = session.LatestIndex; + var before = session.LatestScreen; await session.CommandAsync( "swipe", (client, token) => DeviceSession.SendInputAsync( @@ -289,7 +289,7 @@ public Task PressAsync(DeviceSession session, string key, CancellationToken canc private static async Task PressFocusedAsync(DeviceSession session, string key, InputBudget input, CancellationToken cancellationToken) { var typed = SoftKeyboardKey(key); - var before = session.LatestIndex; + var before = session.LatestScreen; if (typed is null) { await session.CommandAsync( @@ -384,7 +384,7 @@ private static async Task RunNodeAsync( var input = session.InputBudget(kind); var label = $"perform {kind}"; var entry = session.Resolve(node); - var before = session.LatestIndex; + var before = session.LatestScreen; await session.ActAsync( label, async (client, token) => diff --git a/src/E2E.Mobile/DeviceErrors.cs b/src/E2E.Mobile/DeviceErrors.cs index b5460ee1f..8252da43f 100644 --- a/src/E2E.Mobile/DeviceErrors.cs +++ b/src/E2E.Mobile/DeviceErrors.cs @@ -16,6 +16,7 @@ namespace E2E.Mobile; /// SESSION_NOT_FOUND, or a session-or-device-selector refusalINVALID_STATE /// UNSUPPORTED_OPERATION, NOT_IMPLEMENTED, UNSUPPORTED_PLATFORM, or "not supported on this device" under any codeUNSUPPORTED_CAPABILITY /// a stale-ref refusal, by details.reason (action paths only)NODE_STALE (retryable) +/// target_still_moving with details.dispatched: noNODE_STALE (retryable) /// the iOS runner busy, wedged, or past its watchdog; a snapshot the runner acquired but could not presentENGINE_FAILURE, naming the runner and the recovery /// anything else, agent-device's own timeouts includedENGINE_FAILURE with the text /// @@ -202,9 +203,9 @@ public static async Task RunCommandAsync(string label, Func /// Like , but a ref agent-device refused as stale becomes retryable /// NODE_STALE: nothing was dispatched, so the harness may re-observe and re-resolve the node - /// instead of failing the action. Keyed on the reason alone: agent-device marks a drag's stale ref - /// dispatched: 'unknown' after Android blocking-dialog recovery, although it refuses it before - /// the gesture. + /// instead of failing the action. Existing stale-ref reasons are sufficient on their own: Android dialog + /// recovery can mark a refused drag as dispatched: unknown. A target_still_moving refusal + /// additionally requires dispatched: no; missing or unknown dispatch is not retryable. /// public static Exception StaleOr(Exception cause, string operation, string? where = null) { @@ -213,7 +214,9 @@ public static Exception StaleOr(Exception cause, string operation, string? where return cause; } - if (Details(cause).GetValueOrDefault("reason") is string reason && StaleRefReasons.Contains(reason)) + var details = Details(cause); + if (details.GetValueOrDefault("reason") is string reason + && (StaleRefReasons.Contains(reason) || (reason == "target_still_moving" && details.GetValueOrDefault("dispatched") is "no"))) { return new EngineException(EngineErrorCodes.NodeStale, $"{operation}: {Message(cause)}", retryable: true, cause); } diff --git a/src/E2E.Mobile/DeviceSession.Input.cs b/src/E2E.Mobile/DeviceSession.Input.cs index 9c5ef6d73..087611707 100644 --- a/src/E2E.Mobile/DeviceSession.Input.cs +++ b/src/E2E.Mobile/DeviceSession.Input.cs @@ -23,7 +23,7 @@ public async Task PerformAtAsync(Point point, PointerAction action, Cancellation PointerAction.SwipeTo swipe => swipe.Duration, _ => (TimeSpan?)null, }; - var before = _latestIndex; + var before = _latestScreen; await CommandAsync( $"{PointerLabel(action)} at point", (client, token) => SendInputAsync(input, () => DeviceActions.PointerInteractionAsync(client, point, action, ActionSettle(), token), token, hold), @@ -45,7 +45,7 @@ public async Task TypeTextAsync(string text, bool replace = false, CancellationT } var input = InputBudget("type"); - var before = _latestIndex; + var before = _latestScreen; await CommandAsync( "keyboard.type", (client, token) => SendInputAsync(input, () => client.CallAsync(DeviceMethods.InteractionsType, DeviceArgs.Of(("text", JsonValue.Create(text))), token), token), diff --git a/src/E2E.Mobile/DeviceSession.Screen.cs b/src/E2E.Mobile/DeviceSession.Screen.cs index 4b68abdf1..d209d447e 100644 --- a/src/E2E.Mobile/DeviceSession.Screen.cs +++ b/src/E2E.Mobile/DeviceSession.Screen.cs @@ -23,7 +23,7 @@ public enum ActionOrigin /// capture Ref and ControlRef came from, pinned on every action so agent-device refuses them once a newer capture supersedes it. /// TakesText is whether fill and clear may type into the node, judged with the children Node leaves out. /// -internal sealed record NodeBinding(string Id, string Ref, SemanticNode Node, string ControlRef, IReadOnlyDictionary States, long? RefsGeneration, bool TakesText); +internal sealed record NodeBinding(string Id, string Ref, SemanticNode Node, string ControlRef, IReadOnlyDictionary States, long? RefsGeneration, bool TakesText, bool KeyboardVisible); public sealed partial class DeviceSession { @@ -50,19 +50,18 @@ public sealed partial class DeviceSession /// Locators of one test may resolve at once, and an action reads the map while a locate fills it. private readonly Lock _locatedLock = new(); - private IReadOnlyList? _latestIndex; /// The newest projected screen, whose capture generation the refs of an action must come from. private ProjectedSnapshot? _latestScreen; - /// The screen as last projected before the last action, to tell controls that were already there from ones that came with it. - private IReadOnlyList? _indexBeforeAction; + /// The screen as last projected before the last action, with its keyboard state, to tell controls that were already there from ones that came with it. + private ProjectedSnapshot? _screenBeforeAction; /// When this session last acted on the device (an input or an app launch). The screen may be in transition for a while after. private DateTimeOffset _lastActionAt = DateTimeOffset.MinValue; /// The screen as the last look projected it. An action records it as the screen before the action. - internal IReadOnlyList? LatestIndex => _latestIndex; + internal ProjectedSnapshot? LatestScreen => _latestScreen; /// What agent-device is told about settling after an action: the quiet window, or nothing when Settle is off. internal JsonObject SettleArgs() => @@ -81,7 +80,7 @@ internal JsonObject SettleArgs() => /// internal async Task ScreenCommandAsync(string label, Func> run, CancellationToken cancellationToken) { - var before = _latestIndex; + var before = _latestScreen; var result = await CommandAsync(label, run, cancellationToken).ConfigureAwait(false); MarkAction(before); return result; @@ -91,9 +90,9 @@ internal async Task ScreenCommandAsync(string label, Func - internal void MarkAction(IReadOnlyList? before) + internal void MarkAction(ProjectedSnapshot? before) { - _indexBeforeAction = before; + _screenBeforeAction = before; _lastActionAt = _engine.Clock.GetUtcNow(); } @@ -137,7 +136,7 @@ public async Task> LocateAsync(LocatorExpression exp { foreach (var entry in matches) { - _located[entry.Id] = Bind(entry, projected.Index, projected.RefsGeneration); + _located[entry.Id] = Bind(entry, projected); } while (_located.Count > MaxLocatedRefs) @@ -277,8 +276,8 @@ private static int DepthBelow(ProjectedNode entry, ProjectedNode ancestor) } /// Copies the fields actions use and pre-resolves a toggle's inner control. - private static NodeBinding Bind(ProjectedNode entry, IReadOnlyList snapshot, long? refsGeneration) => - new(entry.Id, entry.Ref, entry.Node, ControlOf(entry, snapshot).Ref, entry.States, refsGeneration, SnapshotProjection.MayTakeText(entry.Node)); + private static NodeBinding Bind(ProjectedNode entry, ProjectedSnapshot snapshot) => + new(entry.Id, entry.Ref, entry.Node, ControlOf(entry, snapshot.Index).Ref, entry.States, snapshot.RefsGeneration, SnapshotProjection.MayTakeText(entry.Node), snapshot.KeyboardVisible); /// The same control in the newest capture: the binding itself when it came from that capture, else found there again where it was. internal NodeBinding Current(NodeBinding entry) @@ -289,7 +288,7 @@ internal NodeBinding Current(NodeBinding entry) } return SamePlace(entry, latest.Index) is { } found - ? Bind(found, latest.Index, latest.RefsGeneration) + ? Bind(found, latest) : throw new EngineException(EngineErrorCodes.NodeStale, $"node {entry.Id} changed since it was located; re-resolve", retryable: true); } @@ -344,8 +343,9 @@ a is not null && b is not null /// /// Lets a control that came with the last action finish arriving before a test acts on it, and answers the /// binding to act on. A control already present at the same place in the snapshot the last action was resolved - /// from is not in transition and is acted on at once. Anything else waits out the remainder of the transition - /// budget since that action and is then found again in a fresh snapshot. The budget is the only signal that the + /// from is acted on at once only when keyboard state also stayed the same: a keyboard change can precede the app's + /// relayout. Anything else waits out the remainder of the transition budget since that action and is then found + /// again in a fresh snapshot. The budget is the only signal that the /// transition is over, since the tree cannot tell a sliding control from a landed one. The fresh snapshot is where /// the control landed: Android reports a window's frames while it slides in, so a ref resolved mid-flight would /// send the action where the control was. @@ -368,7 +368,7 @@ internal async Task WaitForTransitionAsync(NodeBinding entry, Cancellation return false; } - if (_indexBeforeAction is { } before && SamePlace(entry, before) is not null) + if (_screenBeforeAction is { } before && before.KeyboardVisible == entry.KeyboardVisible && SamePlace(entry, before.Index) is not null) { return false; } @@ -392,15 +392,31 @@ internal async Task WaitForTransitionAsync(NodeBinding entry, Cancellation return true; } - /// The same control in a fresh snapshot. A snapshot that no longer lists it is a retryable NODE_STALE, so the locator resolves again. - internal async Task RelocatedAsync(NodeBinding entry, InputBudget input, CancellationToken cancellationToken) + /// + /// The same control in a fresh snapshot. A snapshot that no longer lists it is a retryable NODE_STALE, so the locator resolves again. + /// An input mutation () also refuses an incomplete capture and changed security facts, so the field is observed and authorized again. + /// + internal async Task RelocatedAsync(NodeBinding entry, InputBudget input, CancellationToken cancellationToken, bool forInput = false) { - var projected = Project(await SnapshotOrEmptyAsync(false, input, cancellationToken).ConfigureAwait(false)); - return Refind(entry, projected.Index) is { } found - ? Bind(found, projected.Index, projected.RefsGeneration) + var raw = await SnapshotOrEmptyAsync(false, input, cancellationToken).ConfigureAwait(false); + var projected = Project(raw); + var found = Refind(entry, projected.Index); + if (forInput && (raw.Truncated || raw.Sparse || found is null || SecurityFactsChanged(entry.Node, found.Node))) + { + throw new EngineException(EngineErrorCodes.NodeStale, $"input node {entry.Id} changed before mutation; observe and authorize it again", retryable: true); + } + + return found is not null + ? Bind(found, projected) : throw new EngineException(EngineErrorCodes.NodeStale, $"node {entry.Id} changed during transition; re-resolve", retryable: true); } + private static bool SecurityFactsChanged(SemanticNode was, SemanticNode now) => + was.InputPurpose != now.InputPurpose + || was.States.Disabled != now.States.Disabled + || was.States.Secure != now.States.Secure + || was.States.Hidden != now.States.Hidden; + /// /// Runs one action's device command, abandoned when the caller's token fires. A ref agent-device refuses as stale /// becomes a retryable NODE_STALE. @@ -462,7 +478,7 @@ raw.KeyboardKind is "visible" or "derived" && raw.KeyboardFrame is { } frame /// internal async Task DismissKeyboardAsync(CancellationToken cancellationToken, InputBudget input) { - var before = _latestIndex; + var before = _latestScreen; try { await CommandAsync( diff --git a/src/E2E.Mobile/DeviceSession.cs b/src/E2E.Mobile/DeviceSession.cs index 57e92fadb..cc401d4e1 100644 --- a/src/E2E.Mobile/DeviceSession.cs +++ b/src/E2E.Mobile/DeviceSession.cs @@ -286,7 +286,7 @@ await CommandAsync( public async Task BackAsync(CancellationToken cancellationToken) { var input = InputBudget("back"); - var before = _latestIndex; + var before = _latestScreen; await CommandAsync( "back", (client, token) => SendInputAsync(input, () => client.CallAsync(DeviceMethods.CommandBack, ActionSettle(), token), token), @@ -313,7 +313,7 @@ public async Task ObserveAsync(ObserveOptions? options, TimeSpan ti var budget = InputBudget("observe", timeout); var raw = await SnapshotOrEmptyAsync(Options.Snapshot == SnapshotMode.Interactive, budget, cancellationToken).ConfigureAwait(false); var projected = Project(raw); - _generation = projected.Index.ToDictionary(entry => entry.Id, entry => Bind(entry, projected.Index, projected.RefsGeneration)); + _generation = projected.Index.ToDictionary(entry => entry.Id, entry => Bind(entry, projected)); var viewport = await ViewportForAsync(projected, budget, cancellationToken).ConfigureAwait(false); Route = DeviceSupport.ScreenLocation(raw.AppBundleId ?? raw.AppName ?? _appIdentity, SnapshotProjection.ScreenTitle(projected)) ?? ""; var pixels = options?.Pixels == true ? await CapturePixelsAsync(projected, viewport, options.Comparable, budget, cancellationToken).ConfigureAwait(false) : null; @@ -389,7 +389,6 @@ internal async Task CloseAsync(TimeSpan timeout, CancellationToken cancellationT _appIdentity = null; InstalledApp = null; _knownViewport = null; - _latestIndex = null; _latestScreen = null; _generation = []; ClearLocated(); @@ -518,8 +517,8 @@ private static bool SameScreen(ViewportSize a, ViewportSize b) private ProjectedSnapshot Project(RawSnapshot raw) { var projected = SnapshotProjection.ProjectSnapshot(raw.Nodes, () => $"n{++_idCounter}", raw.Viewport, raw.RefsGeneration); + projected.KeyboardVisible = KeyboardShowing(raw); _latestScreen = projected; - _latestIndex = projected.Index; if (projected.Viewport is { } read && (_knownViewport is not { } known || SameScreen(read, known))) { _knownViewport = read; diff --git a/src/E2E.Mobile/MobileOptions.cs b/src/E2E.Mobile/MobileOptions.cs index 9a2db4a05..e780cebfc 100644 --- a/src/E2E.Mobile/MobileOptions.cs +++ b/src/E2E.Mobile/MobileOptions.cs @@ -143,7 +143,9 @@ public sealed record MobileOptions /// /// How long a control that appeared or moved with the last action is given to finish arriving before a test /// acts on it, in milliseconds. Default 500, the length of a modal or screen transition. If the control changes during the - /// wait, the action fails with a retryable NODE_STALE and a locator resolves it again. + /// wait, the action fails with a retryable NODE_STALE and a locator resolves it again. Unmoved controls are acted + /// on immediately only when keyboard visibility is unchanged. Opening or closing the keyboard also uses the remaining + /// budget. Raise it for slower transitions. /// public int TransitionMilliseconds { get; init; } = DefaultTransitionMilliseconds; diff --git a/src/E2E.Mobile/RawNode.cs b/src/E2E.Mobile/RawNode.cs index 6205342fc..7fe673d08 100644 --- a/src/E2E.Mobile/RawNode.cs +++ b/src/E2E.Mobile/RawNode.cs @@ -119,4 +119,10 @@ public sealed class ProjectedSnapshot /// The agent-device capture the refs were issued from; null from one that reports none. public long? RefsGeneration { get; init; } + + /// + /// Whether the soft keyboard showed in the capture, kept with the projection across an action. Upstream's + /// ProjectedScreen (ProjectedSnapshot & { keyboardVisible }), which C# has no intersection type for. + /// + internal bool KeyboardVisible { get; set; } } diff --git a/tests/E2E.Mobile.Tests/Engine/DeterministicActionsTests.cs b/tests/E2E.Mobile.Tests/Engine/DeterministicActionsTests.cs index a348319f5..756e30239 100644 --- a/tests/E2E.Mobile.Tests/Engine/DeterministicActionsTests.cs +++ b/tests/E2E.Mobile.Tests/Engine/DeterministicActionsTests.cs @@ -126,6 +126,138 @@ public async Task Does_not_tap_a_recycled_ref_when_a_native_button_becomes_a_tab JsonAssert.Args("""{"ref":"@e12"}""", h.Fake.LastArgs("interactions.press")); } + public static TheoryData KeyboardOpens() + { + var data = new TheoryData(); + foreach (var action in new[] { "tap", "focus", "fill", "clear", "type", "press character", "focused character" }) + { + foreach (var signal in new[] { "measured", "tree" }) + { + data.Add(action, signal); + } + } + + return data; + } + + private static readonly JsonObject VisibleKeyboard = new() { ["kind"] = "visible", ["frame"] = new JsonObject { ["x"] = 0, ["y"] = 580, ["width"] = 390, ["height"] = 264 } }; + + private static readonly JsonObject AbsentKeyboard = new() { ["kind"] = "absent" }; + + /// The Settings screen with the About cell moved up to a new ref, as the app lays it out once the keyboard transition ends. + private static IEnumerable AboutLanded(IEnumerable nodes) => + nodes.Select(node => node.Identifier == "ABOUT" || node.Label == "About" ? node with { Ref = "@e42", Rect = new Rect(0, 200, 390, 44) } : node); + + [Theory] + [MemberData(nameof(KeyboardOpens))] + public async Task Reobserves_after_action_when_signal_keyboard_state_changes_before_the_control_moves(string action, string signal) + { + var h = WithTransition(120); + var session = await OpenTestAttempt(h); + h.Clock.Advance(TimeSpan.FromMilliseconds(150)); + var keyboardNode = N("@keyboard", index: 11, type: "Keyboard", rect: new Rect(0, 580, 390, 264)); + JsonObject Visible(IEnumerable nodes) => signal == "measured" + ? FakeDeviceClient.SnapshotOf(nodes, ("keyboard", VisibleKeyboard.DeepClone())) + : FakeDeviceClient.SnapshotOf([.. nodes, keyboardNode]); + var search = await Observed(session, "Search"); + switch (action) + { + case "type": + await session.TypeTextAsync("blue", replace: false); + break; + case "focused character": + await session.PressAsync("a", CancellationToken.None); + break; + case "press character": + await session.PerformAsync(search, new LocatorAction.Press("a"), CancellationToken.None); + break; + case "fill": + await session.PerformAsync(search, new LocatorAction.Fill("blue"), CancellationToken.None); + break; + case "clear": + await session.PerformAsync(search, new LocatorAction.Clear(), CancellationToken.None); + break; + case "focus": + await session.PerformAsync(search, new LocatorAction.Focus(), CancellationToken.None); + break; + default: + await session.PerformAsync(search, new LocatorAction.Tap(), CancellationToken.None); + break; + } + + // The first stale layout already reports the keyboard, as in the live probe. + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Visible(SettingsNodes)); + var stale = await Observed(session, "About"); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Visible(AboutLanded(SettingsNodes))); + + await Tap(session, stale); + + JsonAssert.Args("""{"ref":"@e42"}""", h.Fake.LastArgs("interactions.press")); + } + + [Theory] + [InlineData("Enter")] + [InlineData("focused Enter")] + [InlineData("dismiss")] + public async Task Reobserves_after_action_hides_the_keyboard_before_the_control_moves(string action) + { + var h = WithTransition(120); + var session = await OpenTestAttempt(h); + h.Clock.Advance(TimeSpan.FromMilliseconds(150)); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => FakeDeviceClient.SnapshotOf(SettingsNodes, ("keyboard", VisibleKeyboard.DeepClone()))); + var search = await Observed(session, "Search"); + switch (action) + { + case "Enter": + await session.PerformAsync(search, new LocatorAction.Press("Enter"), CancellationToken.None); + break; + case "focused Enter": + await session.PressAsync("Enter", CancellationToken.None); + break; + default: + await session.DismissKeyboardAsync(CancellationToken.None); + break; + } + + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => FakeDeviceClient.SnapshotOf(SettingsNodes, ("keyboard", AbsentKeyboard.DeepClone()))); + var stale = await Observed(session, "About"); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => FakeDeviceClient.SnapshotOf(AboutLanded(SettingsNodes), ("keyboard", AbsentKeyboard.DeepClone()))); + + await Tap(session, stale); + + JsonAssert.Args("""{"ref":"@e42"}""", h.Fake.LastArgs("interactions.press")); + } + + [Theory] + [InlineData("type")] + [InlineData("fill")] + [InlineData("clear")] + public async Task Keeps_unchanged_controls_immediate_after_action_when_keyboard_state_is_unchanged(string action) + { + foreach (var visible in new[] { false, true }) + { + var h = WithTransition(120); + var session = await OpenTestAttempt(h); + h.Clock.Advance(TimeSpan.FromMilliseconds(150)); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => FakeDeviceClient.SnapshotOf(SettingsNodes, ("keyboard", (visible ? VisibleKeyboard : AbsentKeyboard).DeepClone()))); + var search = await Observed(session, "Search"); + switch (action) + { + case "type": + await session.TypeTextAsync("blue", replace: false); + break; + case "fill": + await session.PerformAsync(search, new LocatorAction.Fill("blue"), CancellationToken.None); + break; + default: + await session.PerformAsync(search, new LocatorAction.Clear(), CancellationToken.None); + break; + } + + await TapsAtOnce(h, session, await Observed(session, "About"), """{"ref":"@e4"}"""); + } + } + [Fact] public async Task Acts_on_the_duplicate_nearest_where_the_control_was_when_a_fresh_snapshot_lists_several_alike() { diff --git a/tests/E2E.Mobile.Tests/Engine/InputFreshnessTests.cs b/tests/E2E.Mobile.Tests/Engine/InputFreshnessTests.cs index 427115158..e7f370b78 100644 --- a/tests/E2E.Mobile.Tests/Engine/InputFreshnessTests.cs +++ b/tests/E2E.Mobile.Tests/Engine/InputFreshnessTests.cs @@ -9,9 +9,42 @@ namespace E2E.Mobile.Tests.Engine; -/// Upstream's describe('input freshness') of engine.test.ts; only the commit's three its are here. +/// Upstream's describe('input freshness') of engine.test.ts, as of tester-army/e2e#1410 and #1427. public sealed class InputFreshnessTests { + /// Upstream's form(y, changes): the secure field Account and its neighbor, after a slow accessibility capture. + private static JsonObject Account(double y = 237, Func? change = null, params (string Key, JsonNode? Value)[] more) + { + var field = N("@e20", index: 1, parentIndex: 0, depth: 1, type: "secure-text-field", label: "Account", identifier: "account", rect: R(40, y, 322, 49)); + List nodes = + [ + SettingsNodes[0], + change?.Invoke(field) ?? field, + N("@e21", index: 2, parentIndex: 0, depth: 1, type: "text-field", label: "Username", identifier: "username", rect: R(40, 218, 322, 49)), + ]; + return FakeDeviceClient.SnapshotOf(nodes, more); + } + + private static Harness WithTransition(int milliseconds) => + Harness.Create(options => options with { TransitionMilliseconds = milliseconds }); + + private static async Task OpenAs(Harness h, ActionOrigin origin, TimeSpan? actionTimeout = null) + { + var session = await OpenAttempt(h, actionTimeout: actionTimeout); + session.Origin = origin; + return session; + } + + private static int Snapshots(Harness h) => h.Fake.Methods().Count(method => method == "capture.snapshot"); + + private static int Fills(Harness h) => h.Fake.Methods().Count(method => method == "interactions.fill"); + + private static double Waited(Harness h, int since) => h.Delays.Skip(since).Sum(delay => delay.TotalMilliseconds); + + private static LocatorAction Typing(string kind, bool sensitive) => kind == "fill" ? new LocatorAction.Fill("example", sensitive) : new LocatorAction.Clear(); + + private static string Fill(string @ref, string text) => $$"""{"ref":"{{@ref}}","text":"{{text}}"}"""; + /// A uniquely identified field and its neighbor, as upstream's form, with the field's type and what it holds named. private static JsonObject Form(RawNode field, RawNode? inside = null) { @@ -127,6 +160,255 @@ public static TheoryData GainedCases() return data; } + [Theory] + [MemberData(nameof(GainedCases))] + public async Task Uses_one_fresh_binding_after_a_slow_capture_for_origin_and_kind(ActionOrigin origin, string kind) + { + var h = Harness.Create(); + var session = await OpenAs(h, origin); + h.Fake.RespondAsync(DeviceMethods.CaptureSnapshot, (_, _) => + { + h.Clock.Spend(TimeSpan.FromMilliseconds(600)); + return Task.FromResult(Account()); + }); + var old = await Observed(session, "Account"); + var captures = Snapshots(h); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account(126, field => field with { Ref = "@e42" })); + + await session.PerformAsync(old, Typing(kind, sensitive: true), CancellationToken.None); + + JsonAssert.Args(Fill("@e42", kind == "fill" ? "example" : ""), Strip(h.Fake.LastArgs("interactions.fill")!)); + Assert.Equal(captures + 1, Snapshots(h)); + Assert.Equal(1, Fills(h)); + } + + [Theory] + [MemberData(nameof(GainedCases))] + public async Task Acts_on_a_sliding_input_with_one_capture_for_origin_and_kind(ActionOrigin origin, string kind) + { + var h = Harness.Create(); + var session = await OpenAs(h, origin); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account(2000)); + var sliding = await Observed(session, "Account"); + var captures = Snapshots(h); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account(126, field => field with { Ref = "@e42" })); + var delays = h.Delays.Count; + + await session.PerformAsync(sliding, Typing(kind, sensitive: false), CancellationToken.None); + + JsonAssert.Args(Fill("@e42", kind == "fill" ? "example" : ""), Strip(h.Fake.LastArgs("interactions.fill")!)); + Assert.Equal(captures + 1, Snapshots(h)); + if (origin == ActionOrigin.Test) + { + Assert.True(Waited(h, delays) >= 500); + } + } + + [Theory] + [MemberData(nameof(GainedCases))] + public async Task Rejects_an_originally_non_input_target_before_waiting_or_capturing_for_origin_and_kind(ActionOrigin origin, string kind) + { + var h = Harness.Create(); + var session = await OpenAs(h, origin); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account(237, field => field with { Type = "button" })); + var button = await Observed(session, "Account"); + var captures = Snapshots(h); + var delays = h.Delays.Count; + + var failure = await Assert.ThrowsAsync(() => session.PerformAsync(button, Typing(kind, sensitive: false), CancellationToken.None)); + + Assert.Equal("UNSUPPORTED_CAPABILITY", failure.Code); + Assert.False(failure.Retryable); + Assert.Equal(delays, h.Delays.Count); + Assert.Equal(captures, Snapshots(h)); + Assert.DoesNotContain("interactions.fill", h.Fake.Methods()); + } + + [Fact] + public async Task Refreshes_an_unchanged_input_even_when_the_transition_shortcut_skips_the_wait() + { + var h = Harness.Create(); + var session = await OpenAs(h, ActionOrigin.Test); + await session.PerformAsync(await Observed(session, "Back"), new LocatorAction.Tap(), CancellationToken.None); + var search = await Observed(session, "Search"); + var captures = Snapshots(h); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => FakeDeviceClient.SnapshotOf(SettingsNodes.Select(node => node.Label == "Search" ? node with { Ref = "@e42" } : node))); + var delays = h.Delays.Count; + + await session.PerformAsync(search, new LocatorAction.Clear(), CancellationToken.None); + + Assert.Equal(delays, h.Delays.Count); + JsonAssert.Args(Fill("@e42", ""), Strip(h.Fake.LastArgs("interactions.fill")!)); + Assert.Equal(captures + 1, Snapshots(h)); + } + + public static TheoryData ChangedTargets() => new("missing", "renamed", "different identifier", "truncated", "non-editable", "no longer secure", "disabled", "hidden"); + + private static JsonObject Changed(string name) => name switch + { + "missing" => FakeDeviceClient.SnapshotOf([SettingsNodes[0]]), + "renamed" => Account(237, field => field with { Label = "Other account" }), + "different identifier" => Account(237, field => field with { Identifier = "other" }), + "truncated" => Account(237, null, ("truncated", true)), + "non-editable" => Account(237, field => field with { Type = "button" }), + "no longer secure" => Account(237, field => field with { Type = "text-field" }), + "disabled" => Account(237, field => field with { Enabled = false }), + _ => Account(237, field => field with { VisibleToUser = false }), + }; + + [Theory] + [MemberData(nameof(ChangedTargets))] + public async Task Rejects_a_target_without_dispatching_or_exposing_the_value(string name) + { + var h = WithTransition(0); + var session = await OpenAttempt(h); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account()); + var old = await Observed(session, "Account"); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Changed(name)); + + var failure = await Assert.ThrowsAsync(() => session.PerformAsync(old, new LocatorAction.Fill("not-for-diagnostics", Sensitive: true), CancellationToken.None)); + + Assert.Equal("NODE_STALE", failure.Code); + Assert.True(failure.Retryable); + Assert.DoesNotContain("not-for-diagnostics", failure.Message, StringComparison.Ordinal); + Assert.DoesNotContain("interactions.fill", h.Fake.Methods()); + } + + public static TheoryData OtpCases() => new() { { "fill", 0 }, { "fill", 1 }, { "clear", 0 }, { "clear", 1 } }; + + [Theory] + [MemberData(nameof(OtpCases))] + public async Task Can_act_on_identical_OTP_input_with_the_existing_nearest_node_matching(string kind, int selected) + { + var h = Harness.Create(); + var session = await OpenAs(h, ActionOrigin.Test); + static JsonObject Digits(string[] refs, double shift) => FakeDeviceClient.SnapshotOf( + [ + SettingsNodes[0], + .. refs.Select((@ref, index) => N(@ref, index: index + 1, parentIndex: 0, depth: 1, type: "text-field", label: "Digit", rect: R(40, 100 + (index * 100) + shift, 49, 49))), + ]); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Digits(["@e20", "@e21"], 0)); + var target = Walk(await RootOf(session)).Where(node => node.Name == "Digit").ElementAt(selected); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Digits(["@e41", "@e42"], 10)); + var captures = Snapshots(h); + + await session.PerformAsync(target, Typing(kind, sensitive: false), CancellationToken.None); + + Assert.Equal(selected == 0 ? "@e41" : "@e42", h.Fake.LastArgs("interactions.fill")!["ref"]!.GetValue()); + Assert.Equal(captures + 1, Snapshots(h)); + } + + [Fact] + public async Task Recovers_from_a_truncated_capture_after_re_observation() + { + var h = WithTransition(0); + var session = await OpenAttempt(h); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account()); + var old = await Observed(session, "Account"); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account(237, null, ("truncated", true))); + + var failure = await Assert.ThrowsAsync(() => session.PerformAsync(old, new LocatorAction.Clear(), CancellationToken.None)); + + Assert.Equal("NODE_STALE", failure.Code); + Assert.True(failure.Retryable); + Assert.DoesNotContain("interactions.fill", h.Fake.Methods()); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account()); + await session.PerformAsync(await Observed(session, "Account"), new LocatorAction.Clear(), CancellationToken.None); + Assert.Equal(1, Fills(h)); + } + + [Fact] + public async Task Uses_the_newest_backend_ref_for_an_unchanged_uniquely_identified_input_without_taking_pixels() + { + var h = WithTransition(0); + var session = await OpenAttempt(h); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account()); + var old = await Observed(session, "Account"); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account(237, field => field with { Ref = "@e42", Focused = true })); + + await session.PerformAsync(old, new LocatorAction.Fill("example", Sensitive: true), CancellationToken.None); + + JsonAssert.Args(Fill("@e42", "example"), Strip(h.Fake.LastArgs("interactions.fill")!)); + Assert.DoesNotContain("capture.screenshot", h.Fake.Methods()); + JsonAssert.Args("""{"interactiveOnly":false}""", h.Fake.LastArgs("capture.snapshot")); + } + + [Fact] + public async Task Does_not_silently_accept_changed_security_facts_during_a_transition() + { + var h = Harness.Create(); + var session = await OpenAs(h, ActionOrigin.Test); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account()); + var old = await Observed(session, "Account"); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account(126, field => field with { Enabled = false })); + var captures = Snapshots(h); + + var failure = await Assert.ThrowsAsync(() => session.PerformAsync(old, new LocatorAction.Fill("example", Sensitive: true), CancellationToken.None)); + + Assert.Equal("NODE_STALE", failure.Code); + Assert.DoesNotContain("interactions.fill", h.Fake.Methods()); + Assert.Equal(captures + 1, Snapshots(h)); + } + + [Fact] + public async Task Does_not_send_input_if_the_preflight_consumed_the_input_budget() + { + var h = WithTransition(0); + var session = await OpenAttempt(h, actionTimeout: TimeSpan.FromMilliseconds(500)); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account()); + var old = await Observed(session, "Account"); + h.Fake.RespondAsync(DeviceMethods.CaptureSnapshot, (_, _) => + { + h.Clock.Spend(TimeSpan.FromMilliseconds(600)); + return Task.FromResult(Account()); + }); + + var failure = await Assert.ThrowsAsync(() => session.PerformAsync(old, new LocatorAction.Clear(), CancellationToken.None)); + + Assert.Equal("OPERATION_TIMEOUT", failure.Code); + Assert.DoesNotContain("interactions.fill", h.Fake.Methods()); + } + + [Theory] + [InlineData("fill")] + [InlineData("clear")] + public async Task Does_not_capture_or_send_after_its_transition_wait_is_cancelled(string kind) + { + var h = WithTransition(300); + h.Waits.Wait = (_, token) => Task.Delay(Timeout.Infinite, token); + var session = await OpenAs(h, ActionOrigin.Test); + var search = await Observed(session, "Search"); + var captures = Snapshots(h); + using var controller = new CancellationTokenSource(); + controller.CancelAfter(TimeSpan.FromMilliseconds(20)); + + var cancelled = await Assert.ThrowsAsync(() => session.PerformAsync(search, Typing(kind, sensitive: false), controller.Token)); + + Assert.Equal("CANCELLED", cancelled.Code); + Assert.Equal(captures, Snapshots(h)); + Assert.DoesNotContain("interactions.fill", h.Fake.Methods()); + } + + [Fact] + public async Task Does_not_dispatch_when_aborted_during_the_preflight_capture() + { + var h = WithTransition(0); + var session = await OpenAttempt(h); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => Account()); + var old = await Observed(session, "Account"); + using var controller = new CancellationTokenSource(); + h.Fake.Respond(DeviceMethods.CaptureSnapshot, _ => + { + controller.Cancel(); + return Account(); + }); + + var cancelled = await Assert.ThrowsAsync(() => session.PerformAsync(old, new LocatorAction.Clear(), controller.Token)); + + Assert.Equal("CANCELLED", cancelled.Code); + Assert.DoesNotContain("interactions.fill", h.Fake.Methods()); + } + /// The ref and text of a fill call, the way upstream's toMatchObject reads them: settle options are not part of the assertion. private static JsonObject Strip(JsonObject args) => new() { ["ref"] = args["ref"]?.DeepClone(), ["text"] = args["text"]?.DeepClone() }; } diff --git a/tests/E2E.Mobile.Tests/Errors/ErrorTranslationTests.cs b/tests/E2E.Mobile.Tests/Errors/ErrorTranslationTests.cs index 518681b15..9ba3e9072 100644 --- a/tests/E2E.Mobile.Tests/Errors/ErrorTranslationTests.cs +++ b/tests/E2E.Mobile.Tests/Errors/ErrorTranslationTests.cs @@ -100,6 +100,22 @@ public void Maps_the_stale_ref_refusal_to_retryable_NODE_STALE(string reason, st Assert.Equal("ENGINE_FAILURE", Engine(DeviceErrors.TranslateError(refused, "observe")).Code); } + [Theory] + [InlineData("no")] + [InlineData("unknown")] + [InlineData(null)] + public void Retries_a_moving_target_only_when_no_input_was_dispatched(string? dispatched) + { + var cause = new AgentDeviceError( + "COMMAND_FAILED", + "Target position was not confirmed", + dispatched is null ? Details(("reason", "target_still_moving")) : Details(("reason", "target_still_moving"), ("dispatched", dispatched))); + var translated = Engine(DeviceErrors.StaleOr(cause, "perform tap")); + Assert.Equal(dispatched == "no" ? "NODE_STALE" : "ENGINE_FAILURE", translated.Code); + Assert.Equal(dispatched == "no", translated.Retryable); + Assert.Same(cause, translated.InnerException); + } + [Fact] public void Maps_the_rest_to_ENGINE_FAILURE_with_the_text_a_ref_named_in_a_message_without_a_stale_reason_included() { From 09039ff237be0450236417181f56707ccf6e75a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dar=C3=ADo=20Kondratiuk?= Date: Sun, 1 Nov 2026 19:05:39 -0300 Subject: [PATCH 2/2] Document mobile input revalidation and keyboard reobservation in SKILL.md Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1yaZFgZ3FEJkYScfkK --- skills/e2e-dotnet/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/e2e-dotnet/SKILL.md b/skills/e2e-dotnet/SKILL.md index 7149c221c..1e7926ac9 100644 --- a/skills/e2e-dotnet/SKILL.md +++ b/skills/e2e-dotnet/SKILL.md @@ -36,7 +36,7 @@ trip. The installed tool prints the same text with `e2e guide ` | Handling a failed run: the trace page, `report.json` for scripts, `E2ESession.Report()` for a session you run yourself, error codes and their fixes, a visible browser, replay misses, phase timings, the agent step table, every model call | `debugging` | [references/debugging.md](references/debugging.md) | | Exploring an app toward a goal without a test file, with `e2e explore` or `Explore.ExploreAsync` | `explore` | [references/explore.md](references/explore.md) | | Running isolated per-test sessions from your own runner: `E2EHost.OpenAsync`, `OpenHostOptions` (config value or file, target, `EngineFactory`), `host.OpenSessionAsync`, `E2ESession.CloseAsync` and the failures it returns, `Steps()` | `host` | [references/host.md](references/host.md) | -| Testing an iOS or Android app: `MobileEngine`, the `Device` fixture, picking a simulator (without `MobileOptions.Device`, `MobileEngine` skips booted devices another agent-device session holds, and `prepare` fails with `ENGINE_FAILURE` when every booted one is held), video of the device screen, `Device.UnstableRaw` (the test's agent-device session: name, device id and name, selection, and a raw `IDeviceClient` whose calls are not steps, replayed, or redacted; unstable; `INVALID_STATE` after the session is disposed), Limrun hosted devices, a MAUI app through the DevFlow agent, known limits, `Locator.PressAsync("Enter")` tapping an unfocused field first, and the agent's `open_app` returning to the app under test when it names no app, a Compose Multiplatform field on iOS (for an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text, that text is the field's `Placeholder`, the child is left out, and the name is the label before `, `; a non-empty `ContentDescription` stays the name, and that label is then the `Text`; any other field keeps its label and children), and `FillAsync` and `ClearAsync` refusing a node that is not an editable field with `UNSUPPORTED_CAPABILITY` (an iOS `other` that holds a value or only text is typed into) | `mobile` | [references/mobile.md](references/mobile.md) | +| Testing an iOS or Android app: `MobileEngine`, the `Device` fixture, picking a simulator (without `MobileOptions.Device`, `MobileEngine` skips booted devices another agent-device session holds, and `prepare` fails with `ENGINE_FAILURE` when every booted one is held), video of the device screen, `Device.UnstableRaw` (the test's agent-device session: name, device id and name, selection, and a raw `IDeviceClient` whose calls are not steps, replayed, or redacted; unstable; `INVALID_STATE` after the session is disposed), Limrun hosted devices, a MAUI app through the DevFlow agent, known limits, `Locator.PressAsync("Enter")` tapping an unfocused field first, and the agent's `open_app` returning to the app under test when it names no app, a Compose Multiplatform field on iOS (for an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text, that text is the field's `Placeholder`, the child is left out, and the name is the label before `, `; a non-empty `ContentDescription` stays the name, and that label is then the `Text`; any other field keeps its label and children), and `FillAsync` and `ClearAsync` refusing a node that is not an editable field with `UNSUPPORTED_CAPABILITY` (an iOS `other` that holds a value or only text is typed into), reading the field again before typing (a field the new capture no longer lists, an incomplete capture, or a changed input purpose or secure, disabled, or hidden state is a retryable `NODE_STALE` before anything is typed), looking at the screen again after the keyboard opens or closes (an unmoved control is acted on at once only when keyboard visibility is unchanged, else it waits out `TransitionMilliseconds`), and a `target_still_moving` refusal with `dispatched: no` being a retryable `NODE_STALE` | `mobile` | [references/mobile.md](references/mobile.md) | | Testing a Windows desktop app: `WinAppEngine`, `WinAppTest`, the `Window` fixture, the `windows` target in `e2e explore` and `e2e mcp`, the interactive-desktop requirement, known limits | `windows` | [references/windows.md](references/windows.md) | | Asked to bug bash or hunt for bugs: proving each bug with a repro test | `bug-bash` | [references/bug-bash.md](references/bug-bash.md) |