Skip to content

fix(ios): establish the screen origin of a full-screen private AX root - #4573

Merged
thymikee merged 5 commits into
callstack:mainfrom
gronxb:fix/ios-private-ax-full-screen-origin
Nov 16, 2026
Merged

thymikee merged 5 commits into
callstack:mainfrom
gronxb:fix/ios-private-ax-full-screen-origin

Conversation

@gronxb

@gronxb gronxb commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Since 0.21.37 (#4439), iOS simulator press @ref on a full-screen app fails with window-origin-unknown while the XCTest snapshot channel is penalized, e.g. right after a WKWebView loads a large page. The private AX fallback then declares its root derived, although on an iPhone that root is the screen at the origin (0,0,440,956). The refusal held for over 150 s.

privateAXSnapshotViewport now asks AXRuntime whether the target is the primary app filling the screen (the probe RunnerResolveSoleScreen uses; it walks no tree). A root exactly equal to that screen frame at the origin is reported. A placed iPad window (#4399), an offset root, or a swapped landscape root stays derived. Runner-side counterpart of #4514.

7 files: runner Swift/ObjC, a runner unit test, runner README, ADR 0004, changeset.

Validation

Head 586d58a76 on main 189570486. Live runs: base 26a8a1b30 against head 8cf07fa/2bd2643, which have the same runner code apart from a log line, iPhone 17 Pro Max simulator, iOS 26.5, gronxb/e2e-repro: reload the web view, snapshot -i, press @e2, four rounds.

base head
rounds 2–4 window-origin-unknown, no viewport tapped, viewport 440x956
counter Taps: 1 Taps: 4 (3/3 runs)

The head runner logged PRIVATE_AX_SNAPSHOT_USED and CHANNEL_PENALIZED. e2e's locator.tap suite passes 2/2 on head.

  • testPrivateAXRootViewportReportsOnlyAFullScreenRootAtTheOrigin passes on the simulator. The iOS, macOS, and tvOS runner builds succeed. check:xctest-selection passed.
  • check:affected --run --head 586d58a76: passed, exactCommit. All 16 CI checks pass at 586d58a.
  • iPhone landscape and iPad full-screen / placed-window runs: see the PR comments.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files

You’re at about 95% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread apple/runner/README.md Resolved
@thymikee

Copy link
Copy Markdown
Member

I found no code defects in 8cf07fa, but the change is not ready to merge until two live runs exist. The CI failure is in the Coverage job, which runs the TypeScript suites; this diff touches only Swift/ObjC runner sources, the runner README, ADR 0004 and a changeset, so it looks unrelated. There are no conflicts.

The change turns a refusal into dispatch on the private-AX fallback route for every app the probe calls full-screen, and the two states that decide whether that is safe have not been run live. Please run head 8cf07fa with the XCTest channel penalty (same recipe as the iPhone repro) for two cases. First, the #4399 iPad fixture with the placed window should give window-origin-unknown with dispatched: no, no viewport reported, and a Windowed probe result in the runner log. An iPad app in full screen should also get viewport = screen and its taps should land. Second, the iPhone repro rotated to landscape should give either a refusal or a tap whose native counter advances and whose dispatched point equals the control center. ADR 0004 says AXRuntime reports in one native space, so in landscape the private-AX root and the system-app frame may both be the native portrait box. The equality would then pass and the root would become .reported with orientation unknown, and SnapshotGeometrySpace.space (SnapshotCoordinateSpace.swift:198-206) only rotates for a named landscape orientation. That would publish unrotated frames while taps use the interface space, which is a silent wrong tap where the old code gave a typed refusal. If the landscape run taps wrongly, please make the landscape case derived by reading the interface orientation on the probe route, not by comparing two native-space frames. This reading comes from the ADR, not from observed frames. The iPhone portrait run you reported is acceptable.

Not blocking, and you can take or leave these: the last sentence of the paragraph at apple/runner/README.md:138 still says "without a new probe" right after the paragraph that adds one, so it could say the probe adds no XCTest tree read and no change to the channel budget. The unit test at RunnerTests+AXSnapshotFallbackTests.swift:8 only exercises privateAXRootViewport with hand-built frames, so a fallback exit that forgets the probe is caught only by a live run. If the landscape run shows native-space equality, the landscape case could use a fixture of the real reported frames instead of swapped dimensions.

The open README thread (#4573 (comment)) still applies. The exact-equality thread (#4573 (comment)) does not apply: exact equality fails closed back to the old refusal and never widens acceptance, and it matches the #4514 rule, while the 0.5pt tolerance already covers the fill-the-screen proof in RunnerXCTestEventBridge.m:373, so you can resolve it.

Before merge, please post the iPad placed-window and iPhone landscape runs for 8cf07fa, then fix the README sentence.

@gronxb

gronxb commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

iPhone landscape at 2bd26431d (the same code as 8cf07fa plus a runner log line that prints the probe's verdict with both frames). iPhone 17 Pro Max simulator, iOS 26.5, repro app with landscape enabled, under the XCTest channel penalty: reload the web view, snapshot -i, press @e2, 3 rounds each.

orientation press counter runner log
landscape-left window-origin-unknown, dispatched: no ×3, no viewport Taps: 0 kind=derived root=(0.0, 0.0, 956.0, 440.0) fullScreen=(0.0, 0.0, 440.0, 956.0)
landscape-right same ×3 Taps: 0 same
portrait tapped (220, 98) ×4, viewport 440x956 Taps: 4 kind=reported root=(0.0, 0.0, 440.0, 956.0) fullScreen=(0.0, 0.0, 440.0, 956.0)

The private-AX root is in interface space and AXRuntime's screen frame is native portrait, so in landscape the equality fails and the capture stays derived. Landscape keeps the typed refusal, never an unrotated tap. The unit test's swapped-dimension case uses exactly these measured frames; I added a comment saying so.

Also pushed:

  • d24168a20: AGENT_DEVICE_RUNNER_PRIVATE_AX_ROOT_VIEWPORT logs the verdict and both frames (a null probe prints fullScreen=none), so the iPad run can show the Windowed case. The iOS, macOS, and tvOS runner builds pass.
  • 2bd26431d: the README sentence.

The iPad placed-window and iPad full-screen runs are still to come. This machine has no iPad simulator yet; I'll post them here.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files (changes from recent commits).

You’re at about 95% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

@gronxb

gronxb commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

iPad runs at 2bd26431d (the same code as 8cf07fa plus the verdict log line). iPad Pro 11-inch (M5) simulator, iOS 26.5, Windowed Apps mode, repro app. Under the XCTest channel penalty: reload the web view, snapshot -i, press @e2, 3–4 rounds.

window press counter runner log
full screen tapped (417, 68) ×4, the control center, viewport 834x1210 Taps: 4 kind=reported root=(0.0, 0.0, 834.0, 1210.0) fullScreen=(0.0, 0.0, 834.0, 1210.0)
placed window at (230, 260), 375x575 window-origin-unknown, dispatched: no ×3, no viewport Taps: 0 kind=derived root=(0.0, 0.0, 375.0, 575.0) fullScreen=none

The placed window was made by dragging the window's resize handle. Its private-AX root is window-local, and the probe returned no full-screen frame, so the capture stays derived. No synthesized dispatch was logged during the placed run. Had the local root been accepted, the tap would have landed at (187, 45) on the wallpaper.

fullScreen=none covers both Windowed and Unknown: the log line prints the frame RunnerFullScreenApplicationFrame returned, not the evidence enum. I can split the two if you want it in the log.

Together with the iPhone runs above: iPhone portrait and iPad full screen tap at the control center, while iPhone landscape (both directions) and the iPad placed window keep the typed refusal.

@gronxb
gronxb force-pushed the fix/ios-private-ax-full-screen-origin branch from 2bd2643 to d228f57 Compare November 16, 2026 07:26
@thymikee

Copy link
Copy Markdown
Member

I found no code problems at 586d58a, and the required live evidence is now in place. Since the earlier review at 8cf07fa, the evidence gap is closed by your posted iPhone and iPad runs at 2bd2643, which match this head except for the log-only .missing branch. All 16 checks pass. The earlier Coverage failure ran TypeScript suites that this Swift, ObjC and docs diff does not touch, and the command-surface hunks in the new delta come from the rebase onto main. There are no conflicts, and nothing else needs to happen before merge.

The three review threads need no further code change. Please resolve #4573 (comment) (kind=missing is now logged for an invalid root), #4573 (comment) (the README now describes the two-frame probe accurately) and #4573 (comment) (exact equality only fails closed to .derived, the probe already applies the 0.5pt tolerance, and measured frames are exact).

Not blocking: the posted runs do not cover iPad landscape or portraitUpsideDown full-screen. Upside-down becomes reported with unknown orientation, which is correct only if the private-AX frames there are in interface space, as measured for landscape. The probe now runs from the private-AX capture thread, not the MainActor path that RunnerResolveSoleScreen uses. The live runs show it answers there; a note on why that call is safe off the MainActor would help future readers.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 16, 2026
@gronxb

gronxb commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. All three threads were already resolved.

On the two non-blocking notes:

  • Upside-down: I haven't run it. The landscape frames suggest the private-AX root is in interface space, so an upside-down full-screen root would be 0,0,W,H with the same size as the native screen. It would become reported with unknown orientation and go unrotated. I don't have a measurement to say whether taps there land, so I'm leaving it as a known gap rather than guessing in code.
  • Off-MainActor: the probe makes AXRuntime client calls and reads XCUIApplication.processID. RunnerAXSnapshotBridge already makes the same calls on that capture thread for the tree it serializes, which is why it answers there. I can add that as a comment in a follow-up, so this head and its checks stay as they are.

@thymikee
thymikee merged commit b96eb82 into callstack:main Nov 16, 2026
16 checks passed
thymikee added a commit that referenced this pull request Nov 16, 2026
* origin/main:
  fix(alert): derive the request envelope from an alert wait window (#4581)
  fix(android): require a nonzero chunk before a recorder counts as ready (#4559)
  chore(deps): clear astro and node-forge security alerts (#4577)
  fix(ios): establish the screen origin of a full-screen private AX root (#4573)
  docs(agents): trace behavior claims during review (#4576)
  fix(mcp): trim the press plan text back under the always-loaded byte budget (#4572)
  refactor(alert): one helper owns the alert runner budget (#4563)
  feat(interaction): multi-ref press plan lowered through the ordinary press lane (#4296)
  fix(ios): use the caller's budget for alert get and type the watchdog outcome (#4540)
  feat(recovery): bound remote health and runner-busy recovery by the command's deadline (#4541)
  fix(ios): find the navigation bar back button outside the bar's subtree (#4542)
  feat(open): report device kind, model and osVersion on open and session identity (#4543)
  fix(apple): use typed screenshot timeouts and observed shutdown recovery (#4544)
  feat(device-status): report per device whether a new session can take it (#4547)
  fix(runner): type the refusals for a control the target cannot supply (#4548)
  refactor(transport): share outgoing transfer phase ownership (#4551)
  fix(remote): bound artifact preparation and upload phases (#4552)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants