Repository navigation
Description
What happened
On iOS, a node swipe (locator.swipe(), and a locator-scoped scrollUntilVisible) can start and end at the wrong screen points when agent-device cannot place the app's window on the screen. e2e sends the swipe anyway, so the finger lands offset by the window's position and the test fails somewhere later, or passes against the wrong control.
agent-device 0.21.37 (callstack/agent-device#4439) refuses its own ref actions in this state with window-origin-unknown, and #1820 maps that refusal to NOT_ACTIONABLE. A node swipe never reaches that check, because e2e computes the points from the node's rect and sends them as plain coordinates.
Expected: a node swipe aimed from such a capture fails as NOT_ACTIONABLE with nothing sent, the same as a ref tap on it does now.
This was found in review of #1820 (item 2).
Reproduction
No public app repository: this needs an iPad with a placed (Stage Manager) window, iPhone landscape, or an offset root, which no CI device here has. The unit-level reproduction under "Test scenarios" shows the defect with the engine's fake client.
Steps to reproduce
- Run an iOS app in a window that does not start at the screen's (0, 0), for example a placed iPad window. agent-device's capture of such a window was measured at (230, 260) in fix(ios): establish the screen origin of a full-screen private AX root callstack/agent-device#4573.
- Call
swipe()on a locator that resolves to a scrollable node. - Expected
NOT_ACTIONABLEwith nothing sent. The swipe is sent at points offset by the window's origin.
Versions
Code at e2e 613642198b8c6a43abd2bb3152ecedfebb7c8ea5 (main, 2026-11-17). main at a7d79bd0 has no change to the files named here. @e2e-dev/mobile pins agent-device 0.21.37. agent-device 0.21.38 (bump: #1859) does not change any of this (see below).
Engine
@e2e-dev/mobile
Root cause
-
When agent-device's capture has no reported screen origin (its private viewport evidence is not
reported), the SDKcapture.snapshotresult leaves outviewport. It adds this string towarnings(PublicSnapshotCaptureAnnotations,client-port.d.ts; produced byrunnerSnapshotWarningsin agent-device's Apple interactor):This snapshot cannot establish the window origin on screen. Coordinate-derived element interactions are unavailable until a capture measures the viewport.The node rects in that capture can be window-local, offset from screen coordinates by the window's position (fix(ios): refuse snapshot-derived input without a screen origin callstack/agent-device#4439 body; fix(ios): establish the screen origin of a full-screen private AX root callstack/agent-device#4573). agent-device's CLI help says: "When a fallback cannot establish the screen origin, viewport is omitted and coordinate-derived element actions refuse with window-origin-unknown before input; the snapshot may warn about the unavailable origin."
-
agent-device applies that refusal only to geometry it takes from its own snapshot: refs, selectors,
find, and the gesture viewport fallback. Points a caller writes go through with no origin check and no warning.swipe({ from, to })builds its plan from the runner's native gesture viewport. ADR 0004 in fix(ios): refuse snapshot-derived input without a screen origin callstack/agent-device#4439: "Independent native queries and authored screen coordinates retain their own authorities." No SDK swipe or scroll takes a ref:SwipeOptionsisfrom/toonly. -
@e2e-dev/mobilenever reads thewarningsfield of a snapshot.RawSnapshot(packages/mobile/src/surface.ts:142) picksnodes,viewport,truncated,appName,appBundleId,snapshotQuality,keyboard, andrefsGenerationonly. The onlywarningsread inpackages/mobile/srcislease/daemon.ts:91, which is unrelated. -
The node swipe case in
perform(packages/mobile/src/surface.ts:1530-1534) re-reads the control, then sendsclient.interactions.swipe({ ...swipeWithin(rect, action.direction, action.momentum), ... }).swipeWithin(packages/mobile/src/support.ts:165-184) returns points from the node rect alone. Nothing checks whether the snapshot that supplied the rect established a screen origin.
The missing viewport is not the cause. projectSnapshot falls back to viewportOf(raw) (packages/mobile/src/nodes.ts:524, 678-698), but the node swipe does not read the viewport.
Proposed scope
- Keep agent-device's origin warning on the projected snapshot. Add
warningsto theRawSnapshotpick insurface.ts. Mark a projected screen as having no screen origin when itswarningscontain the exact agent-device string above. Keep that string in one exported constant, with a comment that names agent-device 0.21.37 as its source. Match the whole string, not a fragment. - Use the warning, not a missing
viewport, as the signal. Snapshots withoutviewportalso come from older agent-device builds and from tests: 50 of the 54 golden snapshots underpackages/mobile/tests/fixtures/snapshots/and the default fake snapshot (tests/helpers/fake-client.ts:85,SETTINGS_SNAPSHOT) have none. The warning is specific to this state. - In the
swipecase ofperform, after the control is re-read, check the snapshot that supplied the rect passed toswipeWithin. If it has no screen origin, throwEngineError('NOT_ACTIONABLE', …, { retryable: false })and send nothing. UsenotActionableif it fits. A retry does not help: the state lasts as long as the window stays where it is. - Message, in the style of the
window-origin-unknownhint fix(mobile): fail an action agent-device refuses for an unknown screen origin as an action failure #1820 added toNOT_ACTIONABLE_REASONSinerrors.ts:perform swipe failed: the app's window has no known position on the screen, so node bounds, boundingBox() included, may be offset from screen coordinates. Nothing was sent. Swipe the screen with screen.swipe({ direction }), or between screen points with screen.swipe({ from, to }) not taken from those bounds.Core'sscreen.swipetakes both forms (packages/core/src/locator/screen.ts:232). - The check applies per snapshot, not per session. A later capture without the warning swipes normally.
- Unchanged: ref actions (tap, fill, long press,
dragTo, and the rest), which agent-device already refuses itself. Root and screen swipes by direction (interactions.scroll), which the runner aims itself. Point actions (performAt,screen.tapAt,screen.swipe({ from, to })), where the caller owns the coordinates. Android, where agent-device sends no such warning. - Add a
.changesetentry for@e2e-dev/mobile(patch) that describes the new failure. No docs page describes the swipe's error classes today, so no docs change is needed.
Acceptance criteria
- A node swipe whose rect comes from a snapshot with agent-device's origin warning fails with
NOT_ACTIONABLE,retryable: false, and the message above. agent-device receives nointeractions.swipecall. - The same node swipe on a snapshot without the warning sends the same
from/topoints as today. - A locator-scoped
scrollUntilVisibleon such a snapshot fails the same way, because it goes through the node swipe. - Ref actions, root swipes, and point actions on the warned snapshot behave as before.
- The warning text lives in one constant, and a unit test uses that constant.
Test scenarios
All in packages/mobile/tests/unit/engine.test.ts, beside re-reads a control before a node swipe too, so the swipe starts from where the control rests (line 3621). They reuse harness, openAttempt, observed(h, 'Scroller'), test(), SETTINGS_SNAPSHOT, and SETTINGS_NODES.
- Refused.
h.fake.respond('capture.snapshot', () => ({ ...SETTINGS_SNAPSHOT, warnings: [ORIGIN_WARNING] })), with noviewport. Thenperform(scroller.ref, { kind: 'swipe', direction: 'down' }, test())rejects with{ code: 'NOT_ACTIONABLE', retryable: false }and the message above.h.fake.methods()does not containinteractions.swipe. - Warning only on the re-read. The first snapshot has no warning and the re-read before the swipe has it. The swipe is refused, which shows that the snapshot that supplied the rect decides.
- Not sticky. The first re-read carries the warning (refused), and the next capture has none. A second swipe sends
interactions.swipewithfrom/toequal toswipeWithin(rect, 'down', undefined)for the Scroller's rect. - Unaffected paths. On the warned snapshot,
perform(root.ref, { kind: 'swipe', direction: 'down' })still callsinteractions.scroll. A ref tap still callsinteractions.presswith the ref; agent-device's own refusal of it is covered inerrors.test.ts. - Counterfactual. Scenario 1 without the
warningsentry sends the swipe. On the current code, scenario 1 sends it too, which is the defect.
Out of scope
These paths also turn node geometry into screen points. Each needs its own decision, so they are not part of this issue:
locator.tap({ position }): core computes the point from the node'sorigin ?? rect(locator/engine.ts:540-542) and callsperformAt, which cannot tell it fromscreen.tapAt.- The keyboard dismiss drag (
surface.ts:1695-1708,keyboardDragStartinkeyboard.ts:73-91) and the simulator typing tip's Continue button (surface.ts:1732). - The agent's point verbs, which clamp to
observation.viewport. With noviewport, that is the size derived byviewportOf(coreagent/action-dispatcher.ts). project()learningknownViewportfrom a derived size (surface.ts:1136-1137).- Asking agent-device for a structured code instead of the warning string.
Activity
orcnd commentedon Nov 17, 2026
I'd like to fix this as the issue proposes: read the snapshot's warnings, match agent-device's origin warning from one exported constant, and refuse the node swipe in perform with NOT_ACTIONABLE (retryable: false) and nothing sent. The five scenarios would be unit tests in engine.test.ts.
okwasniewski commentedon Nov 17, 2026
Heads-up on overlapping work: @orcnd offered to fix this, and @JaynouOliver has since opened #1866 with the same shape (refuse the node swipe as
NOT_ACTIONABLEwhen the capture carries agent-device's origin warning). Review continues on #1866. @orcnd, thanks for offering; a review there is welcome instead of a second PR.Automated triage by Claude Code on behalf of the maintainers.