Skip to content

fix(mobile): fail an action agent-device refuses for an unknown screen origin as an action failure - #1820

Open
gronxb wants to merge 3 commits into
e2e-inc:mainfrom
gronxb:fix/mobile-tap-window-origin
Open

gronxb wants to merge 3 commits into
e2e-inc:mainfrom
gronxb:fix/mobile-tap-window-origin

Conversation

@gronxb

@gronxb gronxb commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor
  • agent-device 0.21.37 (fix(ios): refuse snapshot-derived input without a screen origin callstack/agent-device#4439) refuses an iOS ref action with COMMAND_FAILED, details.reason: window-origin-unknown, dispatched: no when its capture can't place the window on screen. It happens on a full-screen iPhone app right after a large web view loads. While that screen stays up the runner reads the tree through its private AX fallback, which never gets the viewport back (measured over 150 s).
  • The engine reported the refusal as ENGINE_FAILURE, an infrastructure error (exit 3), although nothing was sent and the device is fine. It now maps to NOT_ACTIONABLE (ACTION_FAILED, exit 1) in staleOr, next to the covered-target refusals from fix(mobile): hide nodes under the soft keyboard and fail covered taps as actions #1686. A refusal with dispatched: unknown stays ENGINE_FAILURE.
  • agent-device's hint names its CLI (Take a fresh snapshot … or use screenshot …). The message carries the test's move instead: Nothing was sent. For a tap, tap the point instead with screen.tapAt(). A point tap isn't aimed from the capture, so agent-device takes it; other actions have no point form.
  • No retry: re-observing returns the same capture state for as long as the screen stays heavy, so a retry only moves the failure to the deadline.
  • The refused reasons and their hints are one map, NOT_ACTIONABLE_REASONS. undefined keeps agent-device's hint, and a string replaces its CLI advice.
  • No docs change: fix(ios): establish the screen origin of a full-screen private AX root callstack/agent-device#4573 (merged) makes agent-device accept a full-screen root, so the web view case goes away at the next agent-device bump, and the error hint names the move.

Issue

Fixes #1821

Verified

Ran it locally: yes

  • Repro: gronxb/e2e-repro, a UIKit app with a WKWebView that loads 3,000 links. The test reloads it and taps a native button, three rounds. iPhone 17 Pro Max simulator, iOS 26.5, agent-device 0.21.37.
  • e2e run on main's published packages and on this branch's packed e2e, @e2e-dev/core, @e2e-dev/mobile:
main (@e2e-dev/mobile@0.12.7) this branch
✗ locator.tap getByTestId("reload-web") — ENGINE_FAILURE, exit 3 ✗ locator.tap getByTestId("reload-web") — ACTION_FAILED, exit 1
hint: Take a fresh snapshot after the accessibility channel recovers, or use screenshot to choose screen coordinates. hint: Nothing was sent. For a tap, tap the point instead with screen.tapAt().
  • The documented move works: the same three rounds with screen.tapAt({ x: 220, y: 98 }) pass on 0.21.37.
  • errors.test.ts: the new case fails on main (ENGINE_FAILURE) and passes here.

@gronxb
gronxb requested a review from okwasniewski as a code owner November 16, 2026 06:08
@github-actions github-actions Bot added size:M Up to 99 changed lines, not counting tests, fixtures, or the lockfile needs:compliance Community PR fails CONTRIBUTING.md#pull-requests, closed after 72h if not fixed labels Nov 16, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

@github-actions github-actions Bot removed the needs:compliance Community PR fails CONTRIBUTING.md#pull-requests, closed after 72h if not fixed label Nov 16, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 6 files (changes from recent commits).

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

@gronxb

gronxb commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

[claude-opus-5-5] responding on behalf of @gronxb

test (22) and test (24) fail on packages/acp/tests/executor.test.ts > "preserves cancellation while waiting for the previous turn to stop" (ENOTEMPTY removing the agent log directory). This PR doesn't touch @e2e-dev/acp. The same test fails the same way on main's last two pushes (runs 43802621814 and 43794693850, on Node 22, 24.8.0, and 26). It is #1773, and #1774 fixes it. Everything else in spec passed. I'll rebase once #1774 lands.

@okwasniewski okwasniewski left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[claude-opus-5-5] automated review on behalf of @okwasniewski

The classification is right. A refusal with dispatched: no on a healthy device is the test's ACTION_FAILED, not an infrastructure failure, and it matches the covered-target refusals from #1686. I checked the side paths:

  • retryable: false keeps the locator's auto-wait from retrying it.
  • Cache replay already hands off to the agent on any non-fatal action failure.
  • Under the agent, the model now gets ACTION_FAILED instead of APP_UNREACHABLE, so it can fall back to tap_at.

The new test fails on main. It also catches three mutations: dropping the dispatched check, flipping retryable, and keeping agent-device's CLI hint.

Before merge:

  • Please drop the new bullet from docs/mobile.mdx and both setup.md copies, or reword it without the web view trigger. callstack/agent-device#4573 merged today, so "right after a large web view loads" stops being true at the next agent-device bump. The error hint already tells users what to do.
  • Optional: errors.ts now has two NOT_ACTIONABLE branches, and the second builds Hint: by hand. One ReadonlyMap<reason, hintOverride | undefined> would replace both branches and the constants: undefined keeps agent-device's hint, and a string replaces its CLI advice.
  • Nit: the label is perform ${action.kind}, so a refused fill also reads "For a tap, ... screen.tapAt()". That's fine as is, but a refused double tap could name screen.doubleTapAt().

CI: the test (22)/test (24) failure is the acp ENOTEMPTY flake, which #1830 fixed on main. spec gate only mirrors it. Please update the branch from main to clear it (I could not do it from here).

@gronxb
gronxb force-pushed the fix/mobile-tap-window-origin branch from 7673e05 to e469d28 Compare November 16, 2026 10:17
@gronxb

gronxb commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

[claude-opus-5-5] responding on behalf of @gronxb

Thanks. e469d28, on top of main (with #1830):

  • Dropped the bullet from docs/mobile.mdx and both setup.md copies; this PR now changes no docs.
  • Took the map: NOT_ACTIONABLE_REASONS: ReadonlyMap<string, string | undefined> replaces both branches and the constants. undefined keeps agent-device's hint, and a string replaces it. The covered-target cases and their test are unchanged.
  • Left the fill/double-tap wording as is. staleOr only sees the operation label, and a per-action hint would need the action passed down for one message.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M Up to 99 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

iOS: locator.tap right after a large web view loads fails as ENGINE_FAILURE (window-origin-unknown)

2 participants