Repository navigation
fix(mobile): fail an action agent-device refuses for an unknown screen origin as an action failure - #1820
fix(mobile): fail an action agent-device refuses for an unknown screen origin as an action failure#1820gronxb wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
All reported issues were addressed across 6 files
Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.
View guided diff | Re-trigger cubic
There was a problem hiding this comment.
0 issues found across 6 files (changes from recent commits).
Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.
View guided diff | Re-trigger cubic
|
[claude-opus-5-5] responding on behalf of @gronxb
|
There was a problem hiding this comment.
[claude-opus-5-5] automated review on behalf of @okwasniewski
The classification is right. A refusal with dispatched: no on a healthy device is the test's ACTION_FAILED, not an infrastructure failure, and it matches the covered-target refusals from #1686. I checked the side paths:
retryable: falsekeeps the locator's auto-wait from retrying it.- Cache replay already hands off to the agent on any non-fatal action failure.
- Under the agent, the model now gets ACTION_FAILED instead of APP_UNREACHABLE, so it can fall back to
tap_at.
The new test fails on main. It also catches three mutations: dropping the dispatched check, flipping retryable, and keeping agent-device's CLI hint.
Before merge:
- Please drop the new bullet from
docs/mobile.mdxand bothsetup.mdcopies, or reword it without the web view trigger. callstack/agent-device#4573 merged today, so "right after a large web view loads" stops being true at the next agent-device bump. The error hint already tells users what to do. - Optional:
errors.tsnow has two NOT_ACTIONABLE branches, and the second buildsHint:by hand. OneReadonlyMap<reason, hintOverride | undefined>would replace both branches and the constants:undefinedkeeps agent-device's hint, and a string replaces its CLI advice. - Nit: the label is
perform ${action.kind}, so a refusedfillalso reads "For a tap, ... screen.tapAt()". That's fine as is, but a refused double tap could namescreen.doubleTapAt().
CI: the test (22)/test (24) failure is the acp ENOTEMPTY flake, which #1830 fixed on main. spec gate only mirrors it. Please update the branch from main to clear it (I could not do it from here).
…n origin as an action failure
… hints; drop the web view note
7673e05 to
e469d28
Compare
|
[claude-opus-5-5] responding on behalf of @gronxb Thanks. e469d28, on top of main (with #1830):
|
COMMAND_FAILED,details.reason: window-origin-unknown,dispatched: nowhen its capture can't place the window on screen. It happens on a full-screen iPhone app right after a large web view loads. While that screen stays up the runner reads the tree through its private AX fallback, which never gets the viewport back (measured over 150 s).ENGINE_FAILURE, an infrastructure error (exit 3), although nothing was sent and the device is fine. It now maps toNOT_ACTIONABLE(ACTION_FAILED, exit 1) instaleOr, next to the covered-target refusals from fix(mobile): hide nodes under the soft keyboard and fail covered taps as actions #1686. A refusal withdispatched: unknownstaysENGINE_FAILURE.Take a fresh snapshot … or use screenshot …). The message carries the test's move instead:Nothing was sent. For a tap, tap the point instead with screen.tapAt().A point tap isn't aimed from the capture, so agent-device takes it; other actions have no point form.NOT_ACTIONABLE_REASONS.undefinedkeeps agent-device's hint, and a string replaces its CLI advice.Issue
Fixes #1821
Verified
Ran it locally: yes
WKWebViewthat loads 3,000 links. The test reloads it and taps a native button, three rounds. iPhone 17 Pro Max simulator, iOS 26.5, agent-device 0.21.37.e2e runon main's published packages and on this branch's packede2e,@e2e-dev/core,@e2e-dev/mobile:@e2e-dev/mobile@0.12.7)✗ locator.tap getByTestId("reload-web") — ENGINE_FAILURE, exit 3✗ locator.tap getByTestId("reload-web") — ACTION_FAILED, exit 1Take a fresh snapshot after the accessibility channel recovers, or use screenshot to choose screen coordinates.Nothing was sent. For a tap, tap the point instead with screen.tapAt().screen.tapAt({ x: 220, y: 98 })pass on 0.21.37.errors.test.ts: the new case fails on main (ENGINE_FAILURE) and passes here.