Skip to content

iOS: a node swipe sends points from window-local bounds when agent-device cannot place the window on screen #1864

Description

@kdy1

What happened

On iOS, a node swipe (locator.swipe(), and a locator-scoped scrollUntilVisible) can start and end at the wrong screen points when agent-device cannot place the app's window on the screen. e2e sends the swipe anyway, so the finger lands offset by the window's position and the test fails somewhere later, or passes against the wrong control.

agent-device 0.21.37 (callstack/agent-device#4439) refuses its own ref actions in this state with window-origin-unknown, and #1820 maps that refusal to NOT_ACTIONABLE. A node swipe never reaches that check, because e2e computes the points from the node's rect and sends them as plain coordinates.

Expected: a node swipe aimed from such a capture fails as NOT_ACTIONABLE with nothing sent, the same as a ref tap on it does now.

This was found in review of #1820 (item 2).

Reproduction

No public app repository: this needs an iPad with a placed (Stage Manager) window, iPhone landscape, or an offset root, which no CI device here has. The unit-level reproduction under "Test scenarios" shows the defect with the engine's fake client.

Steps to reproduce

  1. Run an iOS app in a window that does not start at the screen's (0, 0), for example a placed iPad window. agent-device's capture of such a window was measured at (230, 260) in fix(ios): establish the screen origin of a full-screen private AX root callstack/agent-device#4573.
  2. Call swipe() on a locator that resolves to a scrollable node.
  3. Expected NOT_ACTIONABLE with nothing sent. The swipe is sent at points offset by the window's origin.

Versions

Code at e2e 613642198b8c6a43abd2bb3152ecedfebb7c8ea5 (main, 2026-11-17). main at a7d79bd0 has no change to the files named here. @e2e-dev/mobile pins agent-device 0.21.37. agent-device 0.21.38 (bump: #1859) does not change any of this (see below).

Engine

@e2e-dev/mobile

Root cause

  1. When agent-device's capture has no reported screen origin (its private viewport evidence is not reported), the SDK capture.snapshot result leaves out viewport. It adds this string to warnings (PublicSnapshotCaptureAnnotations, client-port.d.ts; produced by runnerSnapshotWarnings in agent-device's Apple interactor):

    This snapshot cannot establish the window origin on screen. Coordinate-derived element interactions are unavailable until a capture measures the viewport.
    

    The node rects in that capture can be window-local, offset from screen coordinates by the window's position (fix(ios): refuse snapshot-derived input without a screen origin callstack/agent-device#4439 body; fix(ios): establish the screen origin of a full-screen private AX root callstack/agent-device#4573). agent-device's CLI help says: "When a fallback cannot establish the screen origin, viewport is omitted and coordinate-derived element actions refuse with window-origin-unknown before input; the snapshot may warn about the unavailable origin."

  2. agent-device applies that refusal only to geometry it takes from its own snapshot: refs, selectors, find, and the gesture viewport fallback. Points a caller writes go through with no origin check and no warning. swipe({ from, to }) builds its plan from the runner's native gesture viewport. ADR 0004 in fix(ios): refuse snapshot-derived input without a screen origin callstack/agent-device#4439: "Independent native queries and authored screen coordinates retain their own authorities." No SDK swipe or scroll takes a ref: SwipeOptions is from/to only.

  3. @e2e-dev/mobile never reads the warnings field of a snapshot. RawSnapshot (packages/mobile/src/surface.ts:142) picks nodes, viewport, truncated, appName, appBundleId, snapshotQuality, keyboard, and refsGeneration only. The only warnings read in packages/mobile/src is lease/daemon.ts:91, which is unrelated.

  4. The node swipe case in perform (packages/mobile/src/surface.ts:1530-1534) re-reads the control, then sends client.interactions.swipe({ ...swipeWithin(rect, action.direction, action.momentum), ... }). swipeWithin (packages/mobile/src/support.ts:165-184) returns points from the node rect alone. Nothing checks whether the snapshot that supplied the rect established a screen origin.

The missing viewport is not the cause. projectSnapshot falls back to viewportOf(raw) (packages/mobile/src/nodes.ts:524, 678-698), but the node swipe does not read the viewport.

Proposed scope

  • Keep agent-device's origin warning on the projected snapshot. Add warnings to the RawSnapshot pick in surface.ts. Mark a projected screen as having no screen origin when its warnings contain the exact agent-device string above. Keep that string in one exported constant, with a comment that names agent-device 0.21.37 as its source. Match the whole string, not a fragment.
  • Use the warning, not a missing viewport, as the signal. Snapshots without viewport also come from older agent-device builds and from tests: 50 of the 54 golden snapshots under packages/mobile/tests/fixtures/snapshots/ and the default fake snapshot (tests/helpers/fake-client.ts:85, SETTINGS_SNAPSHOT) have none. The warning is specific to this state.
  • In the swipe case of perform, after the control is re-read, check the snapshot that supplied the rect passed to swipeWithin. If it has no screen origin, throw EngineError('NOT_ACTIONABLE', …, { retryable: false }) and send nothing. Use notActionable if it fits. A retry does not help: the state lasts as long as the window stays where it is.
  • Message, in the style of the window-origin-unknown hint fix(mobile): fail an action agent-device refuses for an unknown screen origin as an action failure #1820 added to NOT_ACTIONABLE_REASONS in errors.ts: perform swipe failed: the app's window has no known position on the screen, so node bounds, boundingBox() included, may be offset from screen coordinates. Nothing was sent. Swipe the screen with screen.swipe({ direction }), or between screen points with screen.swipe({ from, to }) not taken from those bounds. Core's screen.swipe takes both forms (packages/core/src/locator/screen.ts:232).
  • The check applies per snapshot, not per session. A later capture without the warning swipes normally.
  • Unchanged: ref actions (tap, fill, long press, dragTo, and the rest), which agent-device already refuses itself. Root and screen swipes by direction (interactions.scroll), which the runner aims itself. Point actions (performAt, screen.tapAt, screen.swipe({ from, to })), where the caller owns the coordinates. Android, where agent-device sends no such warning.
  • Add a .changeset entry for @e2e-dev/mobile (patch) that describes the new failure. No docs page describes the swipe's error classes today, so no docs change is needed.

Acceptance criteria

  • A node swipe whose rect comes from a snapshot with agent-device's origin warning fails with NOT_ACTIONABLE, retryable: false, and the message above. agent-device receives no interactions.swipe call.
  • The same node swipe on a snapshot without the warning sends the same from/to points as today.
  • A locator-scoped scrollUntilVisible on such a snapshot fails the same way, because it goes through the node swipe.
  • Ref actions, root swipes, and point actions on the warned snapshot behave as before.
  • The warning text lives in one constant, and a unit test uses that constant.

Test scenarios

All in packages/mobile/tests/unit/engine.test.ts, beside re-reads a control before a node swipe too, so the swipe starts from where the control rests (line 3621). They reuse harness, openAttempt, observed(h, 'Scroller'), test(), SETTINGS_SNAPSHOT, and SETTINGS_NODES.

  1. Refused. h.fake.respond('capture.snapshot', () => ({ ...SETTINGS_SNAPSHOT, warnings: [ORIGIN_WARNING] })), with no viewport. Then perform(scroller.ref, { kind: 'swipe', direction: 'down' }, test()) rejects with { code: 'NOT_ACTIONABLE', retryable: false } and the message above. h.fake.methods() does not contain interactions.swipe.
  2. Warning only on the re-read. The first snapshot has no warning and the re-read before the swipe has it. The swipe is refused, which shows that the snapshot that supplied the rect decides.
  3. Not sticky. The first re-read carries the warning (refused), and the next capture has none. A second swipe sends interactions.swipe with from/to equal to swipeWithin(rect, 'down', undefined) for the Scroller's rect.
  4. Unaffected paths. On the warned snapshot, perform(root.ref, { kind: 'swipe', direction: 'down' }) still calls interactions.scroll. A ref tap still calls interactions.press with the ref; agent-device's own refusal of it is covered in errors.test.ts.
  5. Counterfactual. Scenario 1 without the warnings entry sends the swipe. On the current code, scenario 1 sends it too, which is the defect.

Out of scope

These paths also turn node geometry into screen points. Each needs its own decision, so they are not part of this issue:

  • locator.tap({ position }): core computes the point from the node's origin ?? rect (locator/engine.ts:540-542) and calls performAt, which cannot tell it from screen.tapAt.
  • The keyboard dismiss drag (surface.ts:1695-1708, keyboardDragStart in keyboard.ts:73-91) and the simulator typing tip's Continue button (surface.ts:1732).
  • The agent's point verbs, which clamp to observation.viewport. With no viewport, that is the size derived by viewportOf (core agent/action-dispatcher.ts).
  • project() learning knownViewport from a derived size (surface.ts:1136-1137).
  • Asking agent-device for a structured code instead of the warning string.

Activity

  1. added
    bugSomething isn't working
    P2Improvement / follow-up
    mobileMobile engine (@e2e-dev/mobile)
    on Nov 17, 2026
  2. added theissue type on Nov 17, 2026
  3. orcnd commented on Nov 17, 2026

    @orcnd

    I'd like to fix this as the issue proposes: read the snapshot's warnings, match agent-device's origin warning from one exported constant, and refuse the node swipe in perform with NOT_ACTIONABLE (retryable: false) and nothing sent. The five scenarios would be unit tests in engine.test.ts.

  4. okwasniewski commented on Nov 17, 2026

    @okwasniewski
    Member

    Heads-up on overlapping work: @orcnd offered to fix this, and @JaynouOliver has since opened #1866 with the same shape (refuse the node swipe as NOT_ACTIONABLE when the capture carries agent-device's origin warning). Review continues on #1866. @orcnd, thanks for offering; a review there is welcome instead of a second PR.

    Automated triage by Claude Code on behalf of the maintainers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Improvement / follow-upbugSomething isn't workingmobileMobile engine (@e2e-dev/mobile)

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions