diff --git a/.changeset/foldable-scoped-rotation.md b/.changeset/foldable-scoped-rotation.md new file mode 100644 index 0000000000..f327dd3492 --- /dev/null +++ b/.changeset/foldable-scoped-rotation.md @@ -0,0 +1,9 @@ +--- +'agent-device': patch +--- + +Fix foldable iPhone simulator rotation in scoped sets and confirm the lit panel in both default and scoped sets. Rotation in both sets uses the fold helper, with no CoreDevice orientation polling. An already visible target dispatches nothing. Incomplete, unknown-power and multi-lit panel tables neither confirm nor dispatch. A target the panel never shows fails with `orientation-unverified`. + +Use the fixed iPhone Duo picker offsets measured in both sets (outer 0 degrees, inner +90 degrees), replacing runtime calibration. Inner landscape-left sends `portrait-upside-down`, mapped to the HID wire value `pud`; it remains unsupported as a user target. Delayed reads wait without resending the same picker value to the same panel. + +Unresolved guest reads inside the settle loop neither confirm nor dispatch and are retried. An unresolved final read fails with `orientation-panel-table-unavailable`. Multi-panel classification admits rotation without a model-name guard; a new multi-panel model needs its own picker mapping. diff --git a/apple/fold-helper/Fold.m b/apple/fold-helper/Fold.m index cc2c21ca47..60ba17772c 100644 --- a/apple/fold-helper/Fold.m +++ b/apple/fold-helper/Fold.m @@ -106,14 +106,18 @@ static int runSet(NSString *argument) { return dispatchEvents(^BOOL(CFTypeRef client) { return runFrames(client, frames); }); } -// The orientation control has no upside-down value. +// The upside-down picker value is spelled "pud" on the wire; "portrait-upside-down" is +// ignored by the guest. It matters because each picker event rotates the display by +// (new - previous) picker value: without a 180 value, one of the four display orientations +// can never be landed (measured on iPhone Duo). static int runRotate(NSString *orientation) { - if (![@[@"portrait", @"landscape-left", @"landscape-right"] containsObject:orientation]) { - fprintf(stderr, "Expected orientation portrait, landscape-left or landscape-right\n"); + NSString *value = [orientation isEqualToString:@"portrait-upside-down"] ? @"pud" : orientation; + if (![@[@"portrait", @"landscape-left", @"landscape-right", @"pud"] containsObject:value]) { + fprintf(stderr, "Expected orientation portrait, landscape-left, landscape-right or portrait-upside-down\n"); return 2; } return dispatchEvents(^BOOL(CFTypeRef client) { - return sendControl(client, @"orientation-picker-control", @"enum", orientation); + return sendControl(client, @"orientation-picker-control", @"enum", value); }); } diff --git a/docs/adr/0025-foldable-apple-panels.md b/docs/adr/0025-foldable-apple-panels.md index 19f55cf345..e240779db2 100644 --- a/docs/adr/0025-foldable-apple-panels.md +++ b/docs/adr/0025-foldable-apple-panels.md @@ -58,6 +58,9 @@ It is the only reader of panel tables, and it selects the source. For a simulator in a scoped set, the fold helper's `panels` mode is the screen authority. It emits the same entry shape from the guest sources that `dtdeviceinfod` relays to CoreDevice: `CADisplay` and `BLSBacklight`. The guest lists every integrated panel and omits only the facts it cannot read. + Fold pose and rotation lit-panel reads explicitly request the guest table in either simulator set. + Panel-count classification keeps the set-based source: CoreDevice in the default set, guest in a scoped set. + Rotation confirms the lit panel because CoreDevice orientation can match the picker while the visible panel shows another orientation. - **Completeness.** Both sources list one entry per integrated panel. A panel that the parser drops is not single-panel evidence. If two or more integrated entries exist and not all parse, the device is multi-panel without a capture panel. Capture refuses with `capture-panel-unidentified`. @@ -104,7 +107,7 @@ It is the only reader of panel tables, and it selects the source. `IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS` limits the probe to 5s, below the subsequent 20s screenshot deadline. Thus, an unresponsive CoreDevice cannot consume the capture's own budget. Measured cost is 0.16–0.26s, compared with 4.93s for the screenshot. - The guest probe costs about 0.3s. `IOS_SCOPED_DISPLAY_PROBE_BUDGET_MS` limits it to 15s, including the first helper build. + The guest probe costs about 0.3s. `IOS_GUEST_DISPLAY_PROBE_BUDGET_MS` limits it to 15s, including the first helper build. ## Why the default capture had to stop being implicit @@ -370,19 +373,27 @@ Rejected alternatives: `XCUIDevice.orientation` has no effect on an iPhone Duo simulator, and the runner readback repeats the requested value. `devicectl device orientation set` reports success and has no effect on a simulator that `simctl boot` started. `orientation` therefore rotates a multi-panel simulator through the guest device-state control: the HID event that Device Hub's orientation menu sends (`orientation-picker-control`, type `enum`). -The control has no upside-down value, so `portrait-upside-down` is refused with `unsupported-foldable-orientation`. -Routing uses the `fold` admission and the panel table. A resolved panel count is a hardware fact, so the daemon probes each simulator once. - -The verdict comes from `devicectl device orientation get`, a process that never sent the HID event. -CoreDevice answers each query with the value that it sampled at the previous query, so the first read after a dispatch is discarded. -A rotation that no fresh read reports is refused with `orientation-unverified`. -A simulator in a scoped set is refused with `unsupported-device-scope` before dispatch, because no reader can verify its rotation. +The picker accepts `pud` for upside-down dispatch, which the inner panel needs to show landscape-left. +The user target `portrait-upside-down` remains refused with `unsupported-foldable-orientation`. +Routing uses the `fold` admission and the set-based panel-count source. A resolved panel count is a hardware fact, so the daemon probes each simulator once. +A supplied helper is a reuse handle; only an explicit guest-source request changes the default-set source. +Thus, classifying a single-panel default-set simulator does not prepare the fold helper. + +The picker offsets were measured on iPhone Duo; a new multi-panel model needs its own mapping. +In both simulator sets, the helper reads the guest panel table and dispatches the fixed picker value for the established lit panel. +The table maps the outer panel at zero offset and the inner panel at +90 degrees. +Only a fresh `currentOrientation` from an established lit panel confirms the requested visible orientation. +An incomplete table, unknown-power sibling, or multiple lit panels cannot confirm or dispatch. +A repeated visible target dispatches nothing; delayed reads wait without resending to the same panel. +A target that never appears is refused with `orientation-unverified` in either set. +An unresolved guest read inside the settle loop neither confirms nor dispatches; the loop retries it. +An unresolved final read reports `orientation-panel-table-unavailable` and points to the inventory diagnostic. Rejected alternatives: -- **The panel table's `orientation.currentDeviceOrientation`: rejected.** It is screen-relative. On the Duo inner panel, it is a quarter-turn from the device orientation. -- **A guest orientation read: rejected.** CoreMotion `CMDeviceOrientationManager` reports `Ambiguous`, and `BKSHIDServicesIsOrientationLockedWithOrientation` reports 0, after a rotation that CoreDevice and the screenshots confirm. -- **Dispatch without a verdict in a scoped set: rejected.** The runner path already reported a false success there. +- **CoreDevice orientation readback: rejected.** Its picker sample can lag and can match the request while the visible inner panel is a quarter-turn away. +- **Other guest device-orientation APIs: rejected.** CoreMotion `CMDeviceOrientationManager` reports `Ambiguous`, and `BKSHIDServicesIsOrientationLockedWithOrientation` reports 0 after a rotation. +- **Dispatch without a lit-panel verdict: rejected.** An acknowledgement alone can report false success. ## Pose is derived from panel power, and official control does not exist @@ -678,11 +689,6 @@ The silent misses investigated here came from default outer-display routes and i `AgentDeviceMacOSHelper` also reaches it through Screen Capture Kit. Only iOS branches changed target. Desktop behavior remains unmeasured here. -- **Quarter-turn detection.** - Both Duo panels report `currentOrientation: rot90`. - Inventory carries orientation without a test against changed orientation. - `orientation` now rotates a Duo simulator in the default set through guest HID (2026-10-26). - The changed `currentOrientation` values have no test yet. - **A slowly settled hinge.** Four reads consume twenty seconds of streams. A Duo that takes longer to settle inside `half-open` is refused. diff --git a/docs/agents/device-foldable.md b/docs/agents/device-foldable.md index 0719bb2e99..7e06637cd7 100644 --- a/docs/agents/device-foldable.md +++ b/docs/agents/device-foldable.md @@ -1,11 +1,12 @@ # Foldable Apple verification -Before you change capture behavior on a multi-panel device, read ADR 0025. -To list the panels, run `xcrun devicectl device info displays --device `. -In a scoped set, run `xcrun simctl --set spawn ~/.agent-device/fold-helper//fold-helper panels`. +Before capture changes, read ADR 0025. +For default-set capture, run `xcrun devicectl device info displays --device `. +For scoped capture or rotation, run `xcrun simctl --set spawn ~/.agent-device/fold-helper//fold-helper panels`. +For default-set rotation, omit `--set `. - Panels: the list shows each integrated panel with `backlightState`. Only the lit panel permits capture. - A capture of the dark panel exits 0 and writes an all-black PNG. + A dark-panel capture exits 0 with a black PNG. - Input routing: after each pose change, verify a fresh control. A successful synthesis acknowledgement does not prove a hit. Examine the runner and simulator `testmanagerd`/BackBoard logs for display identity and delivery. @@ -14,13 +15,13 @@ In a scoped set, run `xcrun simctl --set spawn ~/.agent-device/fold The same helper reads the hinge back in `hinge` mode. After a fold, take another snapshot. Folds invalidate refs and coordinates. Verify locally with Device Hub stopped and the simulator booted through `simctl boot`. - Test these items in the default set and with `--ios-simulator-device-set`: + Test both sets (`--ios-simulator-device-set`): - All three poses. - Capture of the active panel. - An app interaction. Until GHA supports the runtime, Duo coverage remains local. After a fresh boot, a closed hinge can keep the inner panel lit for more than 30 seconds. - The lit panel is evidence only. It cannot separate `half-open` from `open`. + Panel power cannot separate `half-open` from `open`. - Touch overlays export at the captured track size. The compositor checks its output size and non-black content against the raw capture. If that check fails, `record stop` keeps the raw file. It reports `overlayWarning`. diff --git a/packages/command-surface/src/system/index.ts b/packages/command-surface/src/system/index.ts index 8c9ff9ecb4..947cf830b6 100644 --- a/packages/command-surface/src/system/index.ts +++ b/packages/command-surface/src/system/index.ts @@ -290,7 +290,7 @@ const backCommandDescription = const homeCommandDescription = 'Send the selected device to its home screen. This leaves the app session open but moves the foreground away from the app.'; const orientationCommandDescription = - 'Set device orientation on iOS and Android. auto hands rotation back to the sensor on ADB-backed Android (local and Limrun); other backends refuse it. A foldable iPhone simulator (iPhone Duo) refuses portrait-upside-down, and refuses every rotation in a scoped simulator set.'; + 'Set device orientation on iOS and Android. auto hands rotation back to the sensor on ADB-backed Android (local and Limrun); other backends refuse it. In both default and scoped simulator sets, an iPhone Duo classified with more than one integrated panel refuses portrait-upside-down and rotates through the fold helper; it succeeds only when the lit panel shows the requested orientation, otherwise it fails with orientation-unverified, or orientation-panel-table-unavailable when the final guest panel table cannot be read. A simulator classified with one panel, or whose first classification read is unresolved, uses the standard runner rotation.'; const foldGuidance = { hingeEvent: 'simulator HID hinge event', permissions: 'Device Hub and host Accessibility permission are not required.', diff --git a/packages/platform-apple/src/core/__tests__/display-inventory.test.ts b/packages/platform-apple/src/core/__tests__/display-inventory.test.ts index 69a1576ece..39ce493a6f 100644 --- a/packages/platform-apple/src/core/__tests__/display-inventory.test.ts +++ b/packages/platform-apple/src/core/__tests__/display-inventory.test.ts @@ -33,12 +33,15 @@ vi.mock('@agent-device/host-kit/host-file', async (importOriginal) => { }); import { + appleLitPanel, appleSimulatorDisplayArgvFragment, + establishedLitPanel, buildInventory, parseCoreDeviceDisplays, readPanelPower, resolveAppleCaptureDisplay, queryAppleDisplayInventory, + panelContentRotation, resolveAppleCaptureTarget, type AppleDeviceDisplay, } from '../display-inventory.ts'; @@ -362,6 +365,75 @@ describe('resolveAppleCaptureDisplay', () => { }); }); +describe('panelContentRotation', () => { + test('parses each panel content rotation and nothing else', () => { + assert.equal(panelContentRotation('rot0'), 'portrait'); + assert.equal(panelContentRotation('rot90'), 'landscape-right'); + assert.equal(panelContentRotation('rot180'), 'portrait-upside-down'); + assert.equal(panelContentRotation('rot270'), 'landscape-left'); + for (const value of ['unknown', 'constructor', 'rot45', '']) { + assert.equal(panelContentRotation(value), undefined); + } + }); +}); + +describe('establishedLitPanel', () => { + beforeEach(() => { + mockReadHostTextFile.mockReset(); + mockReadHostTextFile.mockResolvedValue(''); + }); + + function duoTable(mapEntry: (entry: Record) => Record) { + return JSON.stringify({ + result: { displays: IPHONE_DUO_CLOSED.result.displays.map(mapEntry) }, + }); + } + + test('a complete table with one lit panel establishes that panel for both selectors', () => { + const inventory = buildInventory([outerPanel, innerPanel]); + assert.equal(establishedLitPanel(inventory), outerPanel); + assert.deepEqual(appleLitPanel(inventory), { displayId: 1, widthPt: 466, heightPt: 678 }); + }); + + test('two lit panels establish none', () => { + const inventory = buildInventory([outerPanel, { ...innerPanel, power: 'lit' }]); + assert.equal(establishedLitPanel(inventory), undefined); + assert.equal(appleLitPanel(inventory), undefined); + }); + + test('a lit panel beside a panel of unknown power establishes none', () => { + // Unknown power may be showing content, so the lit panel is not the only one that may be lit. + const inventory = buildInventory([outerPanel, { ...innerPanel, power: 'unknown' }]); + assert.equal(establishedLitPanel(inventory), undefined); + assert.equal(appleLitPanel(inventory), undefined); + }); + + test('an incomplete table establishes no panel even when one parsed panel is the only lit one', async () => { + // Two activeOn integrated entries; the inner one has no geometry and does not parse. + mockReadHostTextFile.mockResolvedValue( + duoTable((entry) => { + const lit: Record = { ...entry, backlightState: 'activeOn' }; + if (entry.name !== 'LCD-1') return lit; + const { nativeSize: _nativeSize, ...withoutGeometry } = lit; + return withoutGeometry; + }), + ); + const inventory = await queryAppleDisplayInventory(IOS_TEST_SIMULATOR, {}); + assert.equal(inventory.multiScreen, true); + assert.equal(inventory.displays.filter((display) => display.power === 'lit').length, 1); + assert.equal(establishedLitPanel(inventory), undefined); + assert.equal(appleLitPanel(inventory), undefined); + }); + + test('a single-panel or unresolved table establishes no foldable panel', () => { + assert.equal(establishedLitPanel(buildInventory([outerPanel])), undefined); + assert.equal( + establishedLitPanel({ displays: [], multiScreen: false, ambiguous: false, unresolved: true }), + undefined, + ); + }); +}); + describe('resolveAppleCaptureTarget', () => { beforeEach(() => { mockRunXcrun.mockClear(); @@ -678,7 +750,10 @@ describe('scoped simulator panel table', () => { assert.equal(mockRunXcrun.mock.calls.length, 0); }); - test('reads with the fold helper the caller already opened', async () => { + test.each([ + ['scoped', scopedDuo], + ['default', { ...scopedDuo, simulatorSetPath: undefined }], + ] as const)('reads with the supplied fold helper in the %s set', async (_set, device) => { vi.mocked(ensureFoldHelperBinary).mockClear(); const readPanelPayload = vi.fn(async () => ({ result: { displays: (JSON.parse(GUEST_PANELS_OPEN) as { displays: unknown[] }).displays }, @@ -690,12 +765,33 @@ describe('scoped simulator panel table', () => { readPanelPayload, }; - const inventory = await queryAppleDisplayInventory(scopedDuo, { foldHelper }); + const inventory = await queryAppleDisplayInventory(device, { source: 'guest', foldHelper }); assert.equal(inventory.activeDisplay?.name, 'LCD-1'); assert.equal(readPanelPayload.mock.calls.length, 1); assert.equal(vi.mocked(ensureFoldHelperBinary).mock.calls.length, 0); assert.equal(mockRunSimctlForDevice.mock.calls.length, 0); + assert.equal(mockRunXcrun.mock.calls.length, 0); + }); + + test('a helper handle alone keeps the default-set CoreDevice source', async () => { + mockReadHostTextFile.mockResolvedValue(JSON.stringify(IPHONE_DUO_CLOSED)); + const readPanelPayload = vi.fn(async () => { + throw new Error('guest must not classify'); + }); + const foldHelper = { + setPose: vi.fn(), + rotate: vi.fn(), + readHingeAngle: vi.fn(), + readPanelPayload, + }; + const inventory = await queryAppleDisplayInventory( + { ...scopedDuo, simulatorSetPath: undefined }, + { foldHelper }, + ); + assert.equal(inventory.multiScreen, true); + assert.equal(readPanelPayload.mock.calls.length, 0); + assert.equal(mockRunXcrun.mock.calls.length, 1); }); test('applies the same rule to a CoreDevice table whose second panel does not parse', async () => { diff --git a/packages/platform-apple/src/core/config.ts b/packages/platform-apple/src/core/config.ts index 898dfe8fd2..b81ecc0233 100644 --- a/packages/platform-apple/src/core/config.ts +++ b/packages/platform-apple/src/core/config.ts @@ -31,10 +31,10 @@ export const IOS_SIMULATOR_SCREENSHOT_TIMEOUT_MS = 20_000; // request-level daemon reset. Measured probe cost is ~0.2s. export const IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS = 5_000; /** - * A scoped simulator's panel table comes from a guest helper that the first read on a host also + * A guest panel table comes from a helper that the first read on a host also * builds. This ceiling on the whole read, build included, stays below the capture's own budget. */ -export const IOS_SCOPED_DISPLAY_PROBE_BUDGET_MS = 15_000; +export const IOS_GUEST_DISPLAY_PROBE_BUDGET_MS = 15_000; /** * One guest hinge read, before its sampling window: the fold helper waits at most 3s for its first * CoreMotion sample, so this exec deadline only guards a wedged `simctl spawn`. @@ -55,14 +55,11 @@ export const IOS_FOLD_POSE_READ_INTERVAL_MS = 1_000; /** Two consecutive reads this close together mean the hinge has stopped moving. */ export const IOS_FOLD_POSE_STABLE_DEGREES = 0.5; /** - * How many fresh CoreDevice orientation reads a dispatched foldable rotation gets before it is - * refused. One read costs ~0.2s, and the iPhone Duo reported the new orientation on the first - * fresh read; the margin covers a loaded host. A read that times out counts as one that saw no - * orientation. With the routing probe, a cold fold-helper build, and the HID dispatch, the worst - * case stays inside the default 90s request envelope. + * Bounded lit-panel reads cover delayed HID effects and the two-lit interval after a fold. + * A fixed picker value is sent once per lit panel; the remaining reads only observe. */ -export const IOS_FOLDABLE_ORIENTATION_SETTLE_ATTEMPTS = 8; -export const IOS_FOLDABLE_ORIENTATION_READ_TIMEOUT_MS = 2_000; +export const IOS_FOLDABLE_ORIENTATION_SETTLE_ATTEMPTS = 6; +export const IOS_FOLDABLE_ORIENTATION_SETTLE_MS = 600; // CoreSimulator can briefly stall while it services the scale lookup immediately // after a keyboard transition. Keep this bounded below the full capture budget. diff --git a/packages/platform-apple/src/core/device-orientation.ts b/packages/platform-apple/src/core/device-orientation.ts deleted file mode 100644 index d4ad18301c..0000000000 --- a/packages/platform-apple/src/core/device-orientation.ts +++ /dev/null @@ -1,46 +0,0 @@ -import type { DeviceRotation } from '@agent-device/contracts/device'; -import type { DeviceInfo } from '@agent-device/kernel/device'; -import { isCommandTimeoutError } from '@agent-device/host-kit/command'; - -import { runIosDevicectlJsonRequest, type IosDevicectlJsonOutcome } from './devicectl.ts'; - -/** CoreDevice `deviceOrientation` names, which are UIDeviceOrientation case names. */ -const CORE_DEVICE_ORIENTATIONS: ReadonlyMap = new Map< - unknown, - DeviceRotation ->([ - ['portrait', 'portrait'], - ['portraitUpsideDown', 'portrait-upside-down'], - ['landscapeLeft', 'landscape-left'], - ['landscapeRight', 'landscape-right'], -]); - -/** - * Reads the orientation of the device itself through `devicectl device orientation get`. - * The display table's `currentDeviceOrientation` is not a replacement: on the iPhone Duo inner - * panel, it is a quarter-turn from the device orientation. - * - * CoreDevice answers each query with the orientation it sampled at the previous query, so the - * first read after a change can be stale. A failed or timed-out read, face-up, and face-down - * return nothing. - */ -export async function readAppleDeviceOrientation( - device: DeviceInfo, - options: { timeoutMs: number; signal?: AbortSignal }, -): Promise { - let outcome: IosDevicectlJsonOutcome; - try { - outcome = await runIosDevicectlJsonRequest({ - jsonPrefix: 'agent-device-apple-orientation', - args: ['devicectl', 'device', 'orientation', 'get', '--device', device.id], - timeoutMs: options.timeoutMs, - signal: options.signal, - }); - } catch (error) { - if (options.signal?.aborted || !isCommandTimeoutError(error)) throw error; - return undefined; - } - if (!outcome.ok) return undefined; - const payload = outcome.payload as { result?: { deviceOrientation?: unknown } } | null; - return CORE_DEVICE_ORIENTATIONS.get(payload?.result?.deviceOrientation); -} diff --git a/packages/platform-apple/src/core/display-inventory.ts b/packages/platform-apple/src/core/display-inventory.ts index 476cd6b774..aeb192e121 100644 --- a/packages/platform-apple/src/core/display-inventory.ts +++ b/packages/platform-apple/src/core/display-inventory.ts @@ -5,10 +5,7 @@ import { AppError } from '@agent-device/kernel/errors'; import { isCommandTimeoutError } from '@agent-device/host-kit/command'; import { emitDiagnostic } from '@agent-device/host-kit/diagnostics'; -import { - IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS, - IOS_SCOPED_DISPLAY_PROBE_BUDGET_MS, -} from './config.ts'; +import { IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS, IOS_GUEST_DISPLAY_PROBE_BUDGET_MS } from './config.ts'; import type { FoldHelper } from '../foldable/fold-helper.ts'; import { runIosDevicectlJsonRequest } from './devicectl.ts'; import { simulatorAddressFor } from './simctl.ts'; @@ -93,7 +90,9 @@ type CoreDeviceDisplayPayload = { type AppleDisplayInventoryOptions = { timeoutMs?: number; signal?: AbortSignal; - /** A fold helper the caller already holds, so a scoped read does not probe its build again. */ + /** Requests the guest table for a lit-panel read in either simulator set. */ + source?: 'guest'; + /** Reuses a helper without changing the simulator set's table source. */ foldHelper?: FoldHelper; }; @@ -102,7 +101,7 @@ const DISPLAYS_UNSUPPORTED_HINT = /** * Reads the CoreDevice display table for one Apple device, or the guest's equivalent for a - * simulator in a scoped device set. + * simulator in a scoped device set or a caller that explicitly requests the guest table. * * Deliberately uncached: which panel is lit is exactly what changes when an * operator folds or opens the device, and a cached answer would resume capturing @@ -116,9 +115,10 @@ export async function queryAppleDisplayInventory( device: DeviceInfo, options: AppleDisplayInventoryOptions, ): Promise { - const table = simulatorAddressFor(device).simulatorSetPath - ? await readGuestPanelTable(device, options) - : await readCoreDevicePanelTable(device, options); + const table = + options.source === 'guest' || simulatorAddressFor(device).simulatorSetPath + ? await readGuestPanelTable(device, options) + : await readCoreDevicePanelTable(device, options); if (!table.ok) { emitUnresolvedDiagnostic(device, table.reason, table.detail); return unresolvedInventory(); @@ -153,14 +153,15 @@ async function readCoreDevicePanelTable( /** * CoreDevice resolves a simulator only in the default device set, so a scoped simulator's panel * table comes from the fold helper inside the guest. It reads the same CADisplay and BLSBacklight - * sources `dtdeviceinfod` reports to CoreDevice, in CoreDevice's payload shape. + * sources `dtdeviceinfod` reports to CoreDevice, in CoreDevice's payload shape. Foldable + * operations use this guest source in either set. */ async function readGuestPanelTable( device: DeviceInfo, options: AppleDisplayInventoryOptions, ): Promise { // The first read on a host also builds the helper; the budget covers that build too. - const budget = AbortSignal.timeout(IOS_SCOPED_DISPLAY_PROBE_BUDGET_MS); + const budget = AbortSignal.timeout(IOS_GUEST_DISPLAY_PROBE_BUDGET_MS); const signal = options.signal ? AbortSignal.any([options.signal, budget]) : budget; try { const { createFoldHelper } = await import('../foldable/fold-helper.ts'); @@ -411,14 +412,33 @@ function warnWhenPrimaryPanelIsNotTheSmallest(panels: AppleDeviceDisplay[]): voi } /** - * The panel a runner request names, only when panel power decided exactly one panel. Unlike a - * capture, a surface request does not fall back to the `primary` panel: it can keep the runner's - * historic route instead of guessing. Display 0 is the runtime's "no display" identity. + * The one panel the panel table establishes as lit, or nothing. It takes a resolved multi-panel + * table that parsed every integrated entry, and so named an active display, with exactly one lit + * panel, which is that active display, and no panel of unknown power. A table with an integrated + * entry missing its identity or geometry, or with a panel whose power is unknown, establishes no + * panel, even when the others show a single lit one: that entry may be lit too. Runner routing (`appleLitPanel`) and the foldable + * rotation controller select through this one rule. */ -export function appleLitPanel(inventory: AppleDisplayInventory): RunnerCommand['litPanel'] { +export function establishedLitPanel( + inventory: AppleDisplayInventory, +): AppleDeviceDisplay | undefined { if (!inventory.multiScreen || inventory.unresolved || !inventory.activeDisplay) return undefined; + // Unknown power may be showing content (see readPanelPower), so beside it no lit panel is the + // only one that may be lit. + if (inventory.displays.some((display) => display.power === 'unknown')) return undefined; const litPanels = inventory.displays.filter((display) => display.power === 'lit'); - const panel = litPanels.length === 1 ? litPanels[0]! : undefined; + if (litPanels.length !== 1 || litPanels[0] !== inventory.activeDisplay) return undefined; + return litPanels[0]; +} + +/** + * The panel a runner request names, only when the panel table establishes one lit panel (see + * {@link establishedLitPanel}). Unlike a capture, a surface request does not fall back to the + * `primary` panel: it can keep the runner's historic route instead of guessing. Display 0 is the + * runtime's "no display" identity. + */ +export function appleLitPanel(inventory: AppleDisplayInventory): RunnerCommand['litPanel'] { + const panel = establishedLitPanel(inventory); if (!panel || panel.displayId <= 0) return undefined; return { displayId: panel.displayId, @@ -487,15 +507,28 @@ export type AppleCaptureTarget = * `rot270` is what the display shows after a rotation to landscape-left. An unrecognized or * missing value maps to no rotation. */ -const CORE_DEVICE_ROTATIONS: ReadonlyMap = new Map([ +const PANEL_CONTENT_ROTATIONS: ReadonlyMap = new Map< + string, + DeviceRotation +>([ ['rot0', 'portrait'], ['rot90', 'landscape-right'], ['rot180', 'portrait-upside-down'], ['rot270', 'landscape-left'], ]); +/** + * The content rotation a panel table `currentOrientation` names, in that panel's own frame, or + * nothing for an unrecognized value. This is the one parser of the field: a single-panel capture + * reads it as the device rotation, and the foldable rotation controller reads it as the lit + * panel's rotation before compensating that panel's picker offset. + */ +export function panelContentRotation(currentOrientation: string): DeviceRotation | undefined { + return PANEL_CONTENT_ROTATIONS.get(currentOrientation); +} + function appleDisplayRotation(display: AppleDeviceDisplay): DeviceRotation | undefined { - return CORE_DEVICE_ROTATIONS.get(display.currentOrientation); + return panelContentRotation(display.currentOrientation); } /** diff --git a/packages/platform-apple/src/foldable/fold-helper.test.ts b/packages/platform-apple/src/foldable/fold-helper.test.ts index d2afc0a755..6049158393 100644 --- a/packages/platform-apple/src/foldable/fold-helper.test.ts +++ b/packages/platform-apple/src/foldable/fold-helper.test.ts @@ -114,6 +114,21 @@ test('every mode addresses the UDID inside its scoped simulator set', async () = ]); }); +test('rotate dispatches every picker value, upside-down included', async () => { + const { calls } = await withHelper(scopedDuo, { stdout: '', exitCode: 0 }, async (helper) => { + await helper.rotate('portrait'); + await helper.rotate('landscape-left'); + await helper.rotate('landscape-right'); + await helper.rotate('portrait-upside-down'); + }); + expect(calls.map((call) => call.args.slice(-2))).toEqual([ + ['rotate', 'portrait'], + ['rotate', 'landscape-left'], + ['rotate', 'landscape-right'], + ['rotate', 'portrait-upside-down'], + ]); +}); + test('reads one guest hinge sample without sending HID', async () => { const { value, calls } = await withHelper( scopedDuo, diff --git a/packages/platform-apple/src/foldable/fold-helper.ts b/packages/platform-apple/src/foldable/fold-helper.ts index 1891ed319f..ef1908af54 100644 --- a/packages/platform-apple/src/foldable/fold-helper.ts +++ b/packages/platform-apple/src/foldable/fold-helper.ts @@ -19,9 +19,17 @@ const HINGE_ANGLE_NO_SAMPLE_HINT = const OUTPUT_VERSION = 1; -/** The orientations the guest's device-state control offers; it has no upside-down value. */ +/** The orientations a foldable accepts as a user target; upside-down stays refused there. */ export type FoldableRotation = Exclude; +/** + * Every value the guest's orientation picker accepts, upside-down included: its wire spelling + * is `pud`, which the fold helper maps from `portrait-upside-down`. The picker needs all four + * values because the lit panel adds a fixed offset to the picker, so reaching one target + * can require dispatching another value. Callers keep user targets to {@link FoldableRotation}. + */ +export type FoldPickerValue = DeviceRotation; + /** The guest panel table in the `devicectl device info displays` payload shape. */ export type SimulatorPanelPayload = { result: { displays: unknown[] } }; @@ -33,8 +41,8 @@ export type SimulatorPanelPayload = { result: { displays: unknown[] } }; export type FoldHelper = Readonly<{ /** Dispatches a preset pose or a keyframe trajectory through simulator HID. */ setPose(pose: FoldPose | readonly FoldKeyframe[], signal?: AbortSignal): Promise; - /** Dispatches a device orientation through the same simulator HID control channel. */ - rotate(orientation: FoldableRotation, signal?: AbortSignal): Promise; + /** Dispatches one picker value through the same simulator HID control channel. */ + rotate(orientation: FoldPickerValue, signal?: AbortSignal): Promise; /** * One fresh CoreMotion `CMAngleManager` subscription, the source `devicectl device motion * hinge-angle` relays. It reports the latest sample once `windowMs` has passed since the first, diff --git a/packages/platform-apple/src/foldable/orientation.test.ts b/packages/platform-apple/src/foldable/orientation.test.ts index 85420a7236..6339e716a2 100644 --- a/packages/platform-apple/src/foldable/orientation.test.ts +++ b/packages/platform-apple/src/foldable/orientation.test.ts @@ -1,5 +1,5 @@ import fs from 'node:fs'; -import { beforeEach, expect, test, vi } from 'vitest'; +import { beforeEach, describe, expect, test, vi } from 'vitest'; import { bindTestCommandHost, createLiveTestHostEnvironment } from '@agent-device/test-kit'; vi.mock('./fold-helper-cache.ts', () => ({ @@ -15,7 +15,9 @@ import { AppError } from '@agent-device/kernel/errors'; import { runAppleRunnerCommand } from '../core/runner-client.ts'; import { createLocalAppleToolProvider, withAppleToolProvider } from '../core/tool-provider.ts'; import { createAppleInteractor } from '../interactor.ts'; -import { readFoldablePanelTable } from './orientation.ts'; +import { isAppleFoldableSimulator, readFoldablePanelTable } from './orientation.ts'; +import { ensureFoldHelperBinary } from './fold-helper-cache.ts'; +import { IOS_FOLDABLE_ORIENTATION_SETTLE_ATTEMPTS } from '../core/config.ts'; import { IOS_DEVICE, IOS_SIMULATOR } from '../__tests__/device-fixtures.ts'; import { IPHONE_17_SINGLE_PANEL, IPHONE_DUO_CLOSED } from '../__tests__/display-fixtures.ts'; @@ -25,77 +27,50 @@ const mockRunner = vi.mocked(runAppleRunnerCommand); beforeEach(() => { mockRunner.mockClear(); + vi.mocked(ensureFoldHelperBinary).mockReset().mockResolvedValue({ path: '/cache/fold-helper' }); }); /** A fresh simulator id per test: the daemon keeps each simulator's resolved panel count. */ let simulatorCount = 0; function simulator(overrides: Partial = {}): DeviceInfo { simulatorCount += 1; - return { ...IOS_SIMULATOR, id: `${IOS_SIMULATOR.id}-${simulatorCount}`, ...overrides }; + return { + ...IOS_SIMULATOR, + id: `${IOS_SIMULATOR.id}-${simulatorCount}`, + ...overrides, + }; } -const DUO_DISPLAYS = IPHONE_DUO_CLOSED.result.displays; const SINGLE_PANEL_DISPLAYS = IPHONE_17_SINGLE_PANEL.result.displays; -const CORE_DEVICE_NAMES: Record = { - 'landscape-left': 'landscapeLeft', - 'landscape-right': 'landscapeRight', - portrait: 'portrait', -}; - -/** - * A simulator as CoreDevice and simctl see it. A guest HID orientation event takes effect at once - * unless `followsHid` is false. As measured on iPhone Duo, `devicectl device orientation get` - * answers with the orientation it sampled at the previous query, starting from `sampled`. The - * first `wedgedReads` orientation reads time out. The fold helper's `panels` mode serves the - * displays of a simulator in a scoped set. - */ -function fakeSimulator( - device: DeviceInfo, - displays: readonly object[], - options: { followsHid?: boolean; sampled?: string; wedgedReads?: number } = {}, -) { +/** Single-panel and physical targets keep their runner rotation. */ +function fakeSimulator(device: DeviceInfo, displays: readonly object[]) { const helpers: string[][] = []; - const devicectl: string[] = []; - let orientation = 'portrait'; - let sampled = options.sampled ?? orientation; - let wedgedReads = options.wedgedReads ?? 0; + const unexpectedTools: string[] = []; + const classificationReads: string[] = []; const provider = createLocalAppleToolProvider({ runCommand: async (_command, args) => { - const [tool, ...rest] = args; - if (tool === 'simctl') { - const spawn = rest.slice(rest.indexOf('spawn')); - expect(spawn.slice(0, 3)).toEqual(['spawn', device.id, '/cache/fold-helper']); - const mode = spawn.slice(3); - helpers.push(mode); - if (mode[0] === 'panels') { - return { stdout: JSON.stringify({ version: 1, displays }), stderr: '', exitCode: 0 }; - } - expect(mode[0]).toBe('rotate'); - if (options.followsHid !== false) orientation = CORE_DEVICE_NAMES[mode[1]!]!; - return { stdout: '', stderr: '', exitCode: 0 }; - } - const jsonPath = args.at(-1)!; - const query = args.slice(0, 4).join(' '); - expect(args.slice(4)).toEqual(['--device', device.id, '--json-output', jsonPath]); - devicectl.push(query); - if (query === 'devicectl device info displays') { - fs.writeFileSync(jsonPath, JSON.stringify({ result: { displays } })); + if (args.slice(0, 4).join(' ') === 'devicectl device info displays') { + classificationReads.push(args.join(' ')); + fs.writeFileSync(args.at(-1)!, JSON.stringify({ result: { displays } })); return { stdout: '', stderr: '', exitCode: 0 }; } - expect(query).toBe('devicectl device orientation get'); - if (wedgedReads > 0) { - wedgedReads -= 1; - throw new AppError('COMMAND_FAILED', 'xcrun timed out', { timeoutMs: 2_000 }); + if (args[0] !== 'simctl') { + unexpectedTools.push(args.join(' ')); + throw new Error(`unexpected tool call: ${args.join(' ')}`); } - fs.writeFileSync(jsonPath, JSON.stringify({ result: { deviceOrientation: sampled } })); - sampled = orientation; - return { stdout: '', stderr: '', exitCode: 0 }; + const spawn = args.slice(args.indexOf('spawn')); + expect(spawn.slice(0, 3)).toEqual(['spawn', device.id, '/cache/fold-helper']); + const mode = spawn.slice(3); + helpers.push(mode); + expect(mode).toEqual(['panels']); + return { stdout: JSON.stringify({ version: 1, displays }), stderr: '', exitCode: 0 }; }, }); return { helpers, - devicectl, + unexpectedTools, + classificationReads, rotate: (rotation: DeviceRotation) => withAppleToolProvider(provider, () => createAppleInteractor(device, {}, createLiveTestHostEnvironment()).setOrientation(rotation), @@ -103,34 +78,14 @@ function fakeSimulator( }; } -test.each(['landscape-left', 'portrait', 'landscape-right'] as const)( - 'a foldable simulator rotates to %s through guest HID once CoreDevice reports it', - async (rotation) => { - const device = simulator(); - const sim = fakeSimulator(device, DUO_DISPLAYS, { - sampled: rotation === 'portrait' ? 'landscapeLeft' : 'portrait', - }); - await expect(sim.rotate(rotation)).resolves.toEqual({ orientation: rotation }); - expect(sim.helpers).toEqual([['rotate', rotation]]); - // The routing probe, the read that can be stale, and the read that reports the rotation. - expect(sim.devicectl).toEqual([ - 'devicectl device info displays', - 'devicectl device orientation get', - 'devicectl device orientation get', - ]); - expect(mockRunner).not.toHaveBeenCalled(); - }, -); - -test('a simulator is probed for its panels once', async () => { - const sim = fakeSimulator(simulator(), DUO_DISPLAYS); - await sim.rotate('landscape-left'); - await sim.rotate('portrait'); - expect(sim.devicectl.filter((query) => query.endsWith('displays'))).toHaveLength(1); - expect(sim.helpers).toEqual([ - ['rotate', 'landscape-left'], - ['rotate', 'portrait'], - ]); +test('a simulator is probed for its panel count once', async () => { + const sim = fakeDuo(simulator(), { lit: 'outer', picker: 'portrait' }); + await rotateDuo(sim, 'landscape-left'); + await rotateDuo(sim, 'portrait'); + expect(sim.panelReads()).toBe(5); + expect(sim.classificationReads).toHaveLength(1); + expect(sim.rotations()).toEqual(['landscape-left', 'portrait']); + expect(sim.unexpectedTools).toEqual([]); }); test('a single-panel simulator keeps the runner rotation', async () => { @@ -138,6 +93,8 @@ test('a single-panel simulator keeps the runner rotation', async () => { const sim = fakeSimulator(device, SINGLE_PANEL_DISPLAYS); await expect(sim.rotate('landscape-left')).resolves.toEqual({ orientation: 'landscape-left' }); expect(sim.helpers).toEqual([]); + expect(ensureFoldHelperBinary).not.toHaveBeenCalled(); + expect(sim.classificationReads).toHaveLength(1); expect(mockRunner).toHaveBeenCalledWith( device, expect.objectContaining({ command: 'rotate', orientation: 'landscape-left' }), @@ -145,67 +102,598 @@ test('a single-panel simulator keeps the runner rotation', async () => { ); }); -test('a physical device keeps the runner rotation without a CoreDevice probe', async () => { - const sim = fakeSimulator(IOS_DEVICE, DUO_DISPLAYS); - await expect(sim.rotate('landscape-left')).resolves.toEqual({ orientation: 'landscape-left' }); - expect(sim.devicectl).toEqual([]); - expect(mockRunner).toHaveBeenCalledTimes(1); +test('a cold default Duo classifies without preparing the guest helper', async () => { + const device = simulator(); + const calls: string[] = []; + vi.mocked(ensureFoldHelperBinary).mockRejectedValue( + new Error('cold helper exceeds guest budget'), + ); + const provider = createLocalAppleToolProvider({ + runCommand: async (_command, args) => { + calls.push(args.join(' ')); + expect(args.slice(0, 4)).toEqual(['devicectl', 'device', 'info', 'displays']); + fs.writeFileSync(args.at(-1)!, JSON.stringify(IPHONE_DUO_CLOSED)); + return { stdout: '', stderr: '', exitCode: 0 }; + }, + }); + await expect( + withAppleToolProvider(provider, () => isAppleFoldableSimulator(device, undefined)), + ).resolves.toBe(true); + expect(calls).toHaveLength(1); + expect(ensureFoldHelperBinary).not.toHaveBeenCalled(); }); -test('a foldable simulator in a scoped set refuses rotation before any HID dispatch', async () => { - const scoped = simulator({ simulatorSetPath: '/tmp/scoped-set' }); - const sim = fakeSimulator(scoped, DUO_DISPLAYS); - await expect(sim.rotate('landscape-left')).rejects.toMatchObject({ - code: 'UNSUPPORTED_OPERATION', - details: { reason: 'unsupported-device-scope' }, +test('a cold default Duo helper failure cannot fall back to runner success', async () => { + const sim = fakeDuo(simulator(), { lit: 'outer', picker: 'portrait' }); + vi.mocked(ensureFoldHelperBinary).mockRejectedValue( + new Error('cold helper exceeds guest budget'), + ); + await expect(rotateDuo(sim, 'landscape-left')).resolves.toMatchObject({ + code: 'COMMAND_FAILED', + details: { + reason: 'orientation-panel-table-unavailable', + hint: expect.stringContaining('apple_display_inventory_unresolved'), + }, }); - expect(sim.helpers).toEqual([['panels']]); - expect(sim.devicectl).toEqual([]); + expect(sim.classificationReads).toHaveLength(1); expect(mockRunner).not.toHaveBeenCalled(); + expect(sim.rotations()).toEqual([]); }); -test('a single-panel simulator in a scoped set keeps the runner rotation', async () => { - const sim = fakeSimulator( - simulator({ simulatorSetPath: '/tmp/scoped-set' }), - SINGLE_PANEL_DISPLAYS, - ); +test('a physical device keeps the runner rotation without a CoreDevice probe', async () => { + const sim = fakeSimulator(IOS_DEVICE, []); await expect(sim.rotate('landscape-left')).resolves.toEqual({ orientation: 'landscape-left' }); + expect(sim.unexpectedTools).toEqual([]); + expect(sim.classificationReads).toEqual([]); expect(mockRunner).toHaveBeenCalledTimes(1); }); -test('a foldable simulator refuses portrait-upside-down before any HID dispatch', async () => { - const sim = fakeSimulator(simulator(), DUO_DISPLAYS); - await expect(sim.rotate('portrait-upside-down')).rejects.toMatchObject({ - code: 'UNSUPPORTED_OPERATION', - details: { reason: 'unsupported-foldable-orientation' }, +/** + * A Duo as measured on the device (iOS 27.3): the guest keeps one + * `orientation-picker-control` value, and the lit panel reads that value plus a fixed + * per-panel offset — outer 0, inner +90 — so successive events look relative on the inner + * panel. The picker survives a fold: the panel that lights afterwards reads it plus its own + * offset. A panel whose foreground UI cannot show `picker + offset` keeps the rotation it last + * showed (measured: after a fold left the picker at the upside-down value, the outer panel kept + * landscape-left). For `bothLitReads` panel reads after a fold to closed, both panels report + * lit, as the Duo's table does for a few seconds. Angles follow CADisplay `rotN`: portrait 0, + * landscape-right 90, portrait-upside-down 180, landscape-left 270. A dark panel reports a + * misleading rotation: only the lit panel's `currentOrientation` may confirm a rotation. + */ +const PANEL_ANGLE: Record = { + portrait: 0, + 'landscape-right': 90, + 'portrait-upside-down': 180, + 'landscape-left': 270, +}; +const MEASURED_OFFSETS = { inner: 90, outer: 0 } as const; + +type DuoPanel = 'inner' | 'outer'; + +function rotationAtAngle(angle: number): DeviceRotation { + const rotation = (Object.keys(PANEL_ANGLE) as DeviceRotation[]).find( + (candidate) => PANEL_ANGLE[candidate] === ((angle % 360) + 360) % 360, + ); + return rotation!; +} + +function fakeDuo( + device: DeviceInfo, + options: { + lit: DuoPanel; + picker: DeviceRotation; + offsets?: { inner: number; outer: number }; + followsHid?: boolean; + unresolvedGuestReads?: readonly number[] | 'all'; + /** Rotations a panel's foreground UI cannot show; the panel keeps its last rotation. */ + cannotShow?: Partial>; + /** The rotation a dark panel last showed, for the fold that lights it. */ + lastShown?: Partial>; + /** From this panel read on (1-based, the routing probe included), both panels report lit. */ + bothLitFromRead?: number; + /** How many panel reads after a picker event still show the panel as it was before it. */ + hidLagReads?: number; + /** A panel the table reports lit (activeOn) but without geometry: an incomplete table. */ + incompleteLitPanel?: DuoPanel; + /** A panel whose backlight state the table reports in an unrecognized form: power unknown. */ + unknownPowerPanel?: DuoPanel; + }, +) { + const helpers: string[][] = []; + const unexpectedTools: string[] = []; + const classificationReads: string[] = []; + const offsets = options.offsets ?? MEASURED_OFFSETS; + let lit = options.lit; + let pickerAngle = PANEL_ANGLE[options.picker]; + let pendingPicker: number | undefined; + let lagReadsLeft = 0; + let bothLitReads = 0; + let panelReads = 0; + let guestReads = 0; + const shown: Record = { + inner: PANEL_ANGLE[options.lastShown?.inner ?? 'portrait'], + outer: PANEL_ANGLE[options.lastShown?.outer ?? 'portrait'], + }; + /** The panel shows `picker + offset` unless its foreground UI cannot. */ + const show = (kind: DuoPanel) => { + const angle = (pickerAngle + offsets[kind]) % 360; + if (!options.cannotShow?.[kind]?.includes(rotationAtAngle(angle))) shown[kind] = angle; + }; + show(lit); + const panel = (kind: DuoPanel, bothLit: boolean) => { + const incomplete = options.incompleteLitPanel === kind; + const unknownPower = options.unknownPowerPanel === kind; + const isLit = bothLit || kind === lit || incomplete; + return { + backlightState: unknownPower ? 'transitioning' : isLit ? 'activeOn' : 'off', + currentOrientation: isLit ? `rot${shown[kind]}` : 'rot180', + displayId: kind === 'outer' ? 1 : 3, + name: kind === 'outer' ? 'LCD' : 'LCD-1', + ...(incomplete ? {} : { nativeSize: kind === 'outer' ? [1398, 2034] : [2007, 2853] }), + pointScale: 3, + primary: kind === 'outer', + type: { integrated: {} }, + }; + }; + const readPanels = () => { + panelReads += 1; + if (lagReadsLeft > 0) lagReadsLeft -= 1; + else if (pendingPicker !== undefined) { + pickerAngle = pendingPicker; + pendingPicker = undefined; + show(lit); + } + const bothLit = + bothLitReads > 0 || + (options.bothLitFromRead !== undefined && panelReads >= options.bothLitFromRead); + if (bothLitReads > 0) bothLitReads -= 1; + return [panel('outer', bothLit), panel('inner', bothLit)]; + }; + const provider = createLocalAppleToolProvider({ + runCommand: async (_command, args) => { + const [tool, ...rest] = args; + if (args.slice(0, 4).join(' ') === 'devicectl device info displays') { + classificationReads.push(args.join(' ')); + fs.writeFileSync(args.at(-1)!, JSON.stringify({ result: { displays: readPanels() } })); + return { stdout: '', stderr: '', exitCode: 0 }; + } + if (tool !== 'simctl') { + unexpectedTools.push(args.join(' ')); + throw new Error(`unexpected tool call: ${args.join(' ')}`); + } + const spawn = rest.slice(rest.indexOf('spawn')); + expect(spawn.slice(0, 3)).toEqual(['spawn', device.id, '/cache/fold-helper']); + const mode = spawn.slice(3); + helpers.push(mode); + if (mode[0] === 'panels') { + guestReads += 1; + if ( + options.unresolvedGuestReads === 'all' || + options.unresolvedGuestReads?.includes(guestReads) + ) { + return { stdout: '', stderr: 'transient guest panel read failure', exitCode: 1 }; + } + const displays = readPanels(); + return { stdout: JSON.stringify({ version: 1, displays }), stderr: '', exitCode: 0 }; + } + expect(mode[0]).toBe('rotate'); + if (options.followsHid !== false && options.hidLagReads) { + pendingPicker = PANEL_ANGLE[mode[1]! as DeviceRotation]; + lagReadsLeft = options.hidLagReads; + } else if (options.followsHid !== false) { + pickerAngle = PANEL_ANGLE[mode[1]! as DeviceRotation]; + show(lit); + } + return { stdout: '', stderr: '', exitCode: 0 }; + }, }); - expect(sim.helpers).toEqual([]); - expect(mockRunner).not.toHaveBeenCalled(); -}); + return { + helpers, + unexpectedTools, + classificationReads, + panelReads: () => panelReads, + rotations: () => helpers.filter((mode) => mode[0] === 'rotate').map((mode) => mode[1]!), + litScreen: () => `rot${shown[lit]}`, + /** A pose change made outside the rotation: the picker stays, the other panel lights. */ + fold: (kind: DuoPanel, foldOptions: { bothLitReads?: number } = {}) => { + lit = kind; + bothLitReads = foldOptions.bothLitReads ?? 0; + show(kind); + helpers.length = 0; + panelReads = 0; + }, + rotate: (rotation: DeviceRotation) => + withAppleToolProvider(provider, () => + createAppleInteractor(device, {}, createLiveTestHostEnvironment()).setOrientation(rotation), + ), + }; +} -test('a timed-out CoreDevice read does not refuse a rotation that a later read reports', async () => { - const sim = fakeSimulator(simulator(), DUO_DISPLAYS, { wedgedReads: 2 }); - await expect(sim.rotate('landscape-left')).resolves.toEqual({ orientation: 'landscape-left' }); -}); +/** Runs one rotation under fake timers: the settle delay must not cost real time. */ +async function rotateDuo( + sim: ReturnType, + rotation: DeviceRotation, +): Promise { + const prepare = vi.mocked(ensureFoldHelperBinary).getMockImplementation()!; + let clockStarted = false; + let ready!: () => void; + const helperStarted = new Promise((resolve) => { + ready = resolve; + }); + vi.mocked(ensureFoldHelperBinary).mockImplementation(async (input) => { + if (!clockStarted) { + vi.useFakeTimers(); + clockStarted = true; + ready(); + } + return prepare(input); + }); + try { + const outcome = sim.rotate(rotation).then( + (value) => value, + (error: unknown) => error, + ); + // CoreDevice classification performs file I/O before any settle timer exists. + await Promise.race([helperStarted, outcome]); + if (clockStarted) await vi.advanceTimersByTimeAsync(30_000); + return await outcome; + } finally { + vi.useRealTimers(); + vi.mocked(ensureFoldHelperBinary).mockImplementation(prepare); + } +} -test('a foldable rotation that CoreDevice never reports is refused as unverified', async () => { - const sim = fakeSimulator(simulator(), DUO_DISPLAYS, { followsHid: false }); - await expect(sim.rotate('landscape-right')).rejects.toMatchObject({ - code: 'COMMAND_FAILED', - details: { - reason: 'orientation-unverified', - requestedOrientation: 'landscape-right', - observedOrientation: 'portrait', +describe.each([undefined, '/tmp/scoped-set'])('foldable rotation, set %s', (simulatorSetPath) => { + test.each([ + ['outer', 'portrait', []], + ['outer', 'landscape-left', ['landscape-left']], + ['outer', 'landscape-right', ['landscape-right']], + ['inner', 'portrait', ['landscape-left']], + ['inner', 'landscape-left', ['portrait-upside-down']], + ['inner', 'landscape-right', []], + ] as const)( + 'confirms %s %s from the lit panel without CoreDevice', + async (lit, target, dispatched) => { + const sim = fakeDuo(simulator({ simulatorSetPath }), { lit, picker: 'portrait' }); + await expect(rotateDuo(sim, target)).resolves.toEqual({ orientation: target }); + expect(sim.litScreen()).toBe(`rot${PANEL_ANGLE[target]}`); + expect(sim.rotations()).toEqual(dispatched); + expect(sim.unexpectedTools).toEqual([]); + expect(mockRunner).not.toHaveBeenCalled(); }, + ); + + test.each([1, 2])('retries unresolved guest read %s before confirming', async (failedRead) => { + const device = simulator({ simulatorSetPath }); + const classification = fakeDuo(device, { lit: 'outer', picker: 'portrait' }); + await expect(rotateDuo(classification, 'portrait')).resolves.toEqual({ + orientation: 'portrait', + }); + const sim = fakeDuo(device, { + lit: 'outer', + picker: 'portrait', + unresolvedGuestReads: [failedRead], + }); + await expect(rotateDuo(sim, 'landscape-left')).resolves.toEqual({ + orientation: 'landscape-left', + }); + expect(sim.helpers.map((mode) => mode[0])).toEqual( + failedRead === 1 + ? ['panels', 'panels', 'rotate', 'panels'] + : ['panels', 'rotate', 'panels', 'panels'], + ); + expect(sim.rotations()).toEqual(['landscape-left']); + expect(mockRunner).not.toHaveBeenCalled(); + }); + + test('always unresolved guest reads exhaust the loop without dispatch', async () => { + const device = simulator({ simulatorSetPath }); + const sim = fakeDuo(device, { + lit: 'outer', + picker: 'portrait', + unresolvedGuestReads: 'all', + }); + // Classify before injecting guest failures, including in a scoped set. + const classification = fakeDuo(device, { lit: 'outer', picker: 'portrait' }); + await expect(rotateDuo(classification, 'portrait')).resolves.toEqual({ + orientation: 'portrait', + }); + await expect(rotateDuo(sim, 'landscape-left')).resolves.toMatchObject({ + code: 'COMMAND_FAILED', + details: { reason: 'orientation-panel-table-unavailable' }, + }); + expect(sim.helpers).toHaveLength(IOS_FOLDABLE_ORIENTATION_SETTLE_ATTEMPTS + 1); + expect(sim.rotations()).toEqual([]); + expect(mockRunner).not.toHaveBeenCalled(); + }); + + test('a foldable rotates on the first command, confirmed by the panel table', async () => { + const scoped = simulator({ simulatorSetPath }); + // The inner panel lights at rot90 while the picker sits at portrait (K = +90, measured). + const sim = fakeDuo(scoped, { lit: 'inner', picker: 'portrait' }); + expect(sim.litScreen()).toBe('rot90'); + await expect(rotateDuo(sim, 'portrait')).resolves.toEqual({ orientation: 'portrait' }); + expect(sim.litScreen()).toBe('rot0'); + // The measured inner offset makes the first dispatch exact: portrait - 90 is landscape-left. + expect(sim.rotations()).toEqual(['landscape-left']); + expect(sim.unexpectedTools).toEqual([]); + expect(mockRunner).not.toHaveBeenCalled(); + }); + + test('inner landscape-left dispatches the upside-down picker value', async () => { + const scoped = simulator({ simulatorSetPath }); + const sim = fakeDuo(scoped, { + lit: 'inner', + picker: 'landscape-left', + }); + expect(sim.litScreen()).toBe('rot0'); + await expect(rotateDuo(sim, 'landscape-left')).resolves.toEqual({ + orientation: 'landscape-left', + }); + expect(sim.litScreen()).toBe('rot270'); + expect(sim.rotations()).toEqual(['portrait-upside-down']); + expect(sim.unexpectedTools).toEqual([]); + }); + + test('the outer panel reads the picker value itself, so its first dispatch is the target', async () => { + const scoped = simulator({ simulatorSetPath }); + const sim = fakeDuo(scoped, { lit: 'outer', picker: 'portrait' }); + expect(sim.litScreen()).toBe('rot0'); + for (const rotation of ['landscape-left', 'landscape-right', 'portrait'] as const) { + sim.helpers.length = 0; + await expect(rotateDuo(sim, rotation)).resolves.toEqual({ orientation: rotation }); + expect(sim.rotations()).toEqual([rotation]); + } + expect(sim.unexpectedTools).toEqual([]); }); - expect(sim.helpers).toHaveLength(1); + + test('an unexpected panel response cannot calibrate another picker value', async () => { + const scoped = simulator({ simulatorSetPath }); + const sim = fakeDuo(scoped, { + lit: 'outer', + picker: 'portrait', + offsets: { inner: 90, outer: 270 }, + }); + const outcome = await rotateDuo(sim, 'landscape-right'); + expect(outcome).toMatchObject({ + code: 'COMMAND_FAILED', + details: { requestedOrientation: 'landscape-right', observedOrientation: 'portrait' }, + }); + expect(sim.litScreen()).toBe('rot0'); + // An unexpected reading is not evidence that an unsupported picker value is safe to send. + expect(sim.rotations()).toEqual(['landscape-right']); + expect(sim.unexpectedTools).toEqual([]); + }); + + test('the picker survives a fold in both directions and each panel adds its own offset', async () => { + const scoped = simulator({ simulatorSetPath }); + // Measured sequence: the inner panel at rot0 with the picker at landscape-left. + const sim = fakeDuo(scoped, { lit: 'inner', picker: 'landscape-left' }); + expect(sim.litScreen()).toBe('rot0'); + + sim.fold('outer'); + expect(sim.litScreen()).toBe('rot270'); + await expect(rotateDuo(sim, 'portrait')).resolves.toEqual({ orientation: 'portrait' }); + expect(sim.rotations()).toEqual(['portrait']); + + sim.fold('inner'); + expect(sim.litScreen()).toBe('rot90'); + await expect(rotateDuo(sim, 'landscape-left')).resolves.toEqual({ + orientation: 'landscape-left', + }); + expect(sim.rotations()).toEqual(['portrait-upside-down']); + }); + + test('a fold that leaves the picker where the outer panel cannot follow still lands in one dispatch', async () => { + const scoped = simulator({ simulatorSetPath }); + // Measured: the inner panel at landscape-left needs the picker at the upside-down value; the + // outer panel's foreground UI has no upside-down, so after the fold it kept landscape-left. + const sim = fakeDuo(scoped, { + lit: 'inner', + picker: 'portrait-upside-down', + cannotShow: { outer: ['portrait-upside-down'] }, + lastShown: { outer: 'landscape-left' }, + }); + expect(sim.litScreen()).toBe('rot270'); + sim.fold('outer'); + expect(sim.litScreen()).toBe('rot270'); + await expect(rotateDuo(sim, 'landscape-right')).resolves.toEqual({ + orientation: 'landscape-right', + }); + expect(sim.rotations()).toEqual(['landscape-right']); + }); + + test('reads with both panels lit after a fold to closed neither confirm nor dispatch', async () => { + const scoped = simulator({ simulatorSetPath }); + const sim = fakeDuo(scoped, { lit: 'inner', picker: 'landscape-left' }); + // A first command routes the device as a foldable (one panel probe) and finds portrait showing. + await expect(rotateDuo(sim, 'portrait')).resolves.toEqual({ orientation: 'portrait' }); + // The outer panel already shows the target, but the table names two lit panels for two reads. + sim.fold('outer', { bothLitReads: 2 }); + expect(sim.litScreen()).toBe('rot270'); + await expect(rotateDuo(sim, 'landscape-left')).resolves.toEqual({ + orientation: 'landscape-left', + }); + // Confirmed by the third read, the first to name one lit panel; nothing was dispatched blind. + expect(sim.panelReads()).toBe(3); + expect(sim.rotations()).toEqual([]); + }); + + test('after the both-lit reads, the first single-lit reading drives one exact dispatch', async () => { + const scoped = simulator({ simulatorSetPath }); + const sim = fakeDuo(scoped, { lit: 'inner', picker: 'landscape-left' }); + await expect(rotateDuo(sim, 'portrait')).resolves.toEqual({ orientation: 'portrait' }); + sim.fold('outer', { bothLitReads: 2 }); + await expect(rotateDuo(sim, 'landscape-right')).resolves.toEqual({ + orientation: 'landscape-right', + }); + expect(sim.panelReads()).toBe(4); + expect(sim.rotations()).toEqual(['landscape-right']); + }); + + test('a table that never names a single lit panel dispatches nothing and is refused', async () => { + const scoped = simulator({ simulatorSetPath }); + const sim = fakeDuo(scoped, { lit: 'outer', picker: 'portrait', bothLitFromRead: 1 }); + const outcome = await rotateDuo(sim, 'landscape-left'); + expect(outcome).toMatchObject({ + code: 'COMMAND_FAILED', + details: { + reason: 'orientation-unverified', + requestedOrientation: 'landscape-left', + }, + }); + expect((outcome as AppError).message).toContain('named no single lit panel'); + expect((outcome as AppError).details).not.toHaveProperty('observedOrientation'); + expect(sim.rotations()).toEqual([]); + }); + + test('an undecided final read does not report an earlier reading as the current one', async () => { + const scoped = simulator({ simulatorSetPath }); + // The routing probe and two decided rotation reads, then both panels report lit to the end. + const sim = fakeDuo(scoped, { + lit: 'inner', + picker: 'portrait', + followsHid: false, + bothLitFromRead: 4, + }); + const outcome = await rotateDuo(sim, 'portrait'); + expect(outcome).toMatchObject({ + code: 'COMMAND_FAILED', + details: { reason: 'orientation-unverified' }, + }); + expect((outcome as AppError).details).not.toHaveProperty('observedOrientation'); + expect((outcome as AppError).details).not.toHaveProperty('rawPanelOrientation'); + expect(sim.rotations()).toEqual(['landscape-left']); + }); + + test('a rotation to the orientation already showing dispatches nothing', async () => { + const scoped = simulator({ simulatorSetPath }); + const sim = fakeDuo(scoped, { + lit: 'inner', + picker: 'landscape-left', + }); + expect(sim.litScreen()).toBe('rot0'); + await expect(rotateDuo(sim, 'portrait')).resolves.toEqual({ orientation: 'portrait' }); + expect(sim.rotations()).toEqual([]); + }); + + test.each([ + ['the parsed panel already shows the target', 'landscape-left'], + ['the parsed panel shows another rotation', 'landscape-right'], + ] as const)( + 'an incomplete panel table never confirms or dispatches (%s)', + async (_case, target) => { + const scoped = simulator({ simulatorSetPath }); + // Two activeOn integrated entries; the inner one has no geometry, so it does not parse. The + // outer panel that did parse is the only lit one left, but no single lit panel is established. + const sim = fakeDuo(scoped, { + lit: 'outer', + picker: 'landscape-left', + incompleteLitPanel: 'inner', + }); + expect(sim.litScreen()).toBe('rot270'); + const outcome = await rotateDuo(sim, target); + expect(outcome).toMatchObject({ + code: 'COMMAND_FAILED', + details: { reason: 'orientation-unverified', requestedOrientation: target }, + }); + expect((outcome as AppError).details).not.toHaveProperty('observedOrientation'); + expect(sim.rotations()).toEqual([]); + expect(sim.unexpectedTools).toEqual([]); + }, + ); + + test.each([ + ['the lit panel already shows the target', 'landscape-left'], + ['the lit panel shows another rotation', 'landscape-right'], + ] as const)( + 'a panel of unknown power beside the lit one never confirms or dispatches (%s)', + async (_case, target) => { + const scoped = simulator({ simulatorSetPath }); + // The outer panel reads lit; the inner one reports a backlight state the parser does not + // recognize, so its power is unknown and it may be lit too. + const sim = fakeDuo(scoped, { + lit: 'outer', + picker: 'landscape-left', + unknownPowerPanel: 'inner', + }); + expect(sim.litScreen()).toBe('rot270'); + const outcome = await rotateDuo(sim, target); + expect(outcome).toMatchObject({ + code: 'COMMAND_FAILED', + details: { reason: 'orientation-unverified', requestedOrientation: target }, + }); + expect((outcome as AppError).details).not.toHaveProperty('observedOrientation'); + expect(sim.rotations()).toEqual([]); + expect(sim.unexpectedTools).toEqual([]); + }, + ); + + test('delayed panel reads wait without resending the picker value', async () => { + const scoped = simulator({ simulatorSetPath }); + // The first two panel reads after each picker event still show the panel as it was before it. + const sim = fakeDuo(scoped, { lit: 'inner', picker: 'portrait', hidLagReads: 2 }); + expect(sim.litScreen()).toBe('rot90'); + await expect(rotateDuo(sim, 'landscape-left')).resolves.toEqual({ + orientation: 'landscape-left', + }); + // One exact dispatch (landscape-left - 90 = portrait-upside-down); the stale reads in between + // did not cause another dispatch. + expect(sim.rotations()).toEqual(['portrait-upside-down']); + }); + + test('a rotation the lit panel never shows is refused as unverified', async () => { + const scoped = simulator({ simulatorSetPath }); + const sim = fakeDuo(scoped, { + lit: 'inner', + picker: 'portrait', + followsHid: false, + }); + await expect(rotateDuo(sim, 'portrait')).resolves.toMatchObject({ + code: 'COMMAND_FAILED', + message: expect.stringContaining('the lit panel shows landscape-right'), + details: { + reason: 'orientation-unverified', + requestedOrientation: 'portrait', + observedOrientation: 'landscape-right', + rawPanelOrientation: 'rot90', + }, + }); + // The panel never moved after the one dispatch, so nothing more was + // dispatched: the loop re-read until its budget ended, then refused. + expect(sim.rotations()).toEqual(['landscape-left']); + expect(sim.unexpectedTools).toEqual([]); + }); + + test('a foldable still refuses portrait-upside-down as a target', async () => { + const scoped = simulator({ simulatorSetPath }); + const sim = fakeDuo(scoped, { + lit: 'inner', + picker: 'portrait', + }); + await expect(sim.rotate('portrait-upside-down')).rejects.toMatchObject({ + code: 'UNSUPPORTED_OPERATION', + details: { reason: 'unsupported-foldable-orientation' }, + }); + expect(sim.rotations()).toEqual([]); + expect(mockRunner).not.toHaveBeenCalled(); + }); +}); + +test('a single-panel simulator in a scoped set keeps the runner rotation', async () => { + const sim = fakeSimulator( + simulator({ simulatorSetPath: '/tmp/scoped-set' }), + SINGLE_PANEL_DISPLAYS, + ); + await expect(sim.rotate('landscape-left')).resolves.toEqual({ orientation: 'landscape-left' }); + expect(mockRunner).toHaveBeenCalledTimes(1); }); /** A rotation reads the panel table itself: the surface probe's retry window never holds it. */ test('a rotation is not routed by a pending panel-table retry', async () => { const device = simulator(); let tableFailures = 0; - const sim = fakeSimulator(device, DUO_DISPLAYS); + const sim = fakeDuo(device, { lit: 'outer', picker: 'portrait' }); const failing = createLocalAppleToolProvider({ runCommand: async (_command, args) => { if (args.slice(0, 4).join(' ') === 'devicectl device info displays') { @@ -220,17 +708,9 @@ test('a rotation is not routed by a pending panel-table retry', async () => { await withAppleToolProvider(failing, () => readFoldablePanelTable(device, { timeoutMs: 5_000 })); expect(tableFailures).toBe(1); - await expect(sim.rotate('landscape-left')).resolves.toEqual({ orientation: 'landscape-left' }); - expect(sim.helpers).toEqual([['rotate', 'landscape-left']]); - expect(mockRunner).not.toHaveBeenCalled(); -}); - -test('a stale CoreDevice sample cannot verify a rotation that did not happen', async () => { - const sim = fakeSimulator(simulator(), DUO_DISPLAYS, { - followsHid: false, - sampled: 'landscapeLeft', - }); - await expect(sim.rotate('landscape-left')).rejects.toMatchObject({ - details: { reason: 'orientation-unverified', observedOrientation: 'portrait' }, + await expect(rotateDuo(sim, 'landscape-left')).resolves.toEqual({ + orientation: 'landscape-left', }); + expect(sim.rotations()).toEqual(['landscape-left']); + expect(mockRunner).not.toHaveBeenCalled(); }); diff --git a/packages/platform-apple/src/foldable/orientation.ts b/packages/platform-apple/src/foldable/orientation.ts index a67802bab1..a6ec1d45b5 100644 --- a/packages/platform-apple/src/foldable/orientation.ts +++ b/packages/platform-apple/src/foldable/orientation.ts @@ -6,26 +6,25 @@ import { createTtlMemo } from '@agent-device/kernel/ttl-memo'; import { IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS, - IOS_FOLDABLE_ORIENTATION_READ_TIMEOUT_MS, IOS_FOLDABLE_ORIENTATION_SETTLE_ATTEMPTS, + IOS_FOLDABLE_ORIENTATION_SETTLE_MS, } from '../core/config.ts'; -import { readAppleDeviceOrientation } from '../core/device-orientation.ts'; import { + establishedLitPanel, + panelContentRotation, queryAppleDisplayInventory, + type AppleDeviceDisplay, type AppleDisplayInventory, } from '../core/display-inventory.ts'; import { simulatorAddressFor } from '../core/simctl.ts'; -import { createFoldHelper } from './fold-helper.ts'; +import { createFoldHelper, type FoldHelper, type FoldableRotation } from './fold-helper.ts'; import { appleFoldableFacts } from './runtime.ts'; const UPSIDE_DOWN_UNSUPPORTED_HINT = - 'A foldable simulator accepts portrait, landscape-left, and landscape-right. Its device-state control has no upside-down orientation.'; - -const SCOPED_SET_UNSUPPORTED_HINT = - 'CoreDevice resolves only simulators in the default device set, so the rotation of a foldable simulator in a scoped set cannot be verified. Rotate it in the default device set.'; + 'A foldable simulator accepts portrait, landscape-left, and landscape-right as rotation targets.'; const ORIENTATION_UNVERIFIED_HINT = - 'The simulator HID command completed, but CoreDevice did not report the new orientation. Read it with xcrun devicectl device orientation get --device , then retry.'; + 'The lit panel did not show the requested orientation, or the simulator never reported a single lit panel; the app in front may not support that orientation. Take a screenshot to see the screen, then retry.'; /** Resolved panel counts are hardware facts, so each simulator is probed once per daemon. */ const multiPanelByDevice = new Map(); @@ -61,7 +60,9 @@ export async function isAppleFoldableSimulator( const known = multiPanelByDevice.get(device.id); if (known !== undefined) return known; const read = ++panelTableReads; - const inventory = await queryAppleDisplayInventory(device, { signal }); + const inventory = await queryAppleDisplayInventory(device, { + signal, + }); if (inventory.unresolved) return false; recordPanelTable(device, inventory, read); return inventory.multiScreen; @@ -101,7 +102,23 @@ export async function readFoldablePanelTable( return inventory.multiScreen ? { kind: 'multi-panel', inventory } : { kind: 'single-panel' }; } -/** Rotates a foldable simulator through guest HID and verifies the rotation through CoreDevice. */ +/** iPhone Duo (iOS 27.3) renders picker + offset: outer 0 degrees, inner +90. */ +const PANEL_PICKERS = { + outer: { + portrait: 'portrait', + 'landscape-left': 'landscape-left', + 'landscape-right': 'landscape-right', + }, + inner: { + portrait: 'landscape-left', + 'landscape-left': 'portrait-upside-down', + 'landscape-right': 'portrait', + }, +} as const; + +type LitPanel = { panel: AppleDeviceDisplay; rotation: DeviceRotation }; + +/** Dispatches through guest HID and confirms from the lit panel in either simulator set. */ export async function setAppleFoldableOrientation( device: DeviceInfo, rotation: DeviceRotation, @@ -116,41 +133,102 @@ export async function setAppleFoldableOrientation( hint: UPSIDE_DOWN_UNSUPPORTED_HINT, }); } - if (simulatorAddressFor(device).simulatorSetPath) { - throw new AppError( - 'UNSUPPORTED_OPERATION', - `${device.name} cannot rotate in a scoped simulator set`, - { - deviceId: device.id, - reason: 'unsupported-device-scope', - hint: SCOPED_SET_UNSUPPORTED_HINT, - }, - ); - } - await createFoldHelper(device).rotate(rotation, signal); - emitDiagnostic({ - level: 'info', - phase: 'apple_foldable_orientation_dispatched', - data: { deviceId: device.id, orientation: rotation }, - }); - const read = { signal, timeoutMs: IOS_FOLDABLE_ORIENTATION_READ_TIMEOUT_MS }; - // This read can return the orientation CoreDevice sampled before the dispatch. - await readAppleDeviceOrientation(device, read); - let observed: DeviceRotation | undefined; + const helper = createFoldHelper(device); + let dispatchedDisplayId: number | undefined; + let reading: LitPanel | undefined; for (let attempt = 1; attempt <= IOS_FOLDABLE_ORIENTATION_SETTLE_ATTEMPTS; attempt += 1) { signal?.throwIfAborted(); - observed = await readAppleDeviceOrientation(device, read); - if (observed === rotation) return { orientation: rotation }; + reading = await readLitPanel(device, helper, { signal }); + if (reading?.rotation === rotation) return { orientation: rotation }; + if (reading && reading.panel.displayId !== dispatchedDisplayId) { + const mapping = reading.panel.primary ? PANEL_PICKERS.outer : PANEL_PICKERS.inner; + const value = mapping[rotation]; + await helper.rotate(value, signal); + dispatchedDisplayId = reading.panel.displayId; + emitDiagnostic({ + level: 'info', + phase: 'apple_foldable_orientation_dispatched', + data: { + deviceId: device.id, + orientation: value, + requestedOrientation: rotation, + scopedSet: Boolean(simulatorAddressFor(device).simulatorSetPath), + attempt, + observedOrientation: reading.rotation, + litDisplay: reading.panel.name, + }, + }); + } + await settleRotation(); } - throw new AppError( + reading = await readLitPanel(device, helper, { signal, requireTable: true }); + if (reading?.rotation === rotation) return { orientation: rotation }; + throw unverifiedError(device, rotation, reading); +} + +/** An undecided final read reports no rotation, even if an earlier read named one. */ +function unverifiedError( + device: DeviceInfo, + rotation: FoldableRotation, + reading: LitPanel | undefined, +): AppError { + return new AppError( 'COMMAND_FAILED', - `${device.name} did not rotate to ${rotation}: CoreDevice reports ${observed ?? 'no orientation'}`, + `${device.name} did not confirm rotation to ${rotation}: ${ + reading + ? `the lit panel shows ${reading.rotation}` + : 'the last panel table read named no single lit panel' + }`, { deviceId: device.id, requestedOrientation: rotation, - ...(observed === undefined ? {} : { observedOrientation: observed }), + ...(reading + ? { + observedOrientation: reading.rotation, + rawPanelOrientation: reading.panel.currentOrientation, + } + : {}), reason: 'orientation-unverified', hint: ORIENTATION_UNVERIFIED_HINT, }, ); } + +/** + * The one lit panel and the rotation it shows, from one fresh panel-table read, selected by the + * display inventory's {@link establishedLitPanel}. Anything less decided — an unreadable table, + * an incomplete one, no lit panel or several, a panel of unknown power, an unrecognized `rotN` — + * is no reading: + * the rotation loop must not confirm or dispatch from a panel that is not showing. + * On iPhone Duo the table reports both panels lit for a few seconds after a pose change in either + * direction, so the first reads after a fold or an unfold are no reading by design. + */ +async function readLitPanel( + device: DeviceInfo, + helper: FoldHelper, + options: { signal?: AbortSignal; requireTable?: boolean }, +): Promise { + const inventory = await queryAppleDisplayInventory(device, { + signal: options.signal, + source: 'guest', + foldHelper: helper, + }); + if (inventory.unresolved && options.requireTable) { + throw new AppError('COMMAND_FAILED', 'The guest panel table could not be read', { + deviceId: device.id, + reason: 'orientation-panel-table-unavailable', + hint: 'The fold helper could not provide a usable display table. Check the apple_display_inventory_unresolved diagnostic for the helper or display-table failure, then retry.', + }); + } + const panel = establishedLitPanel(inventory); + if (!panel) return undefined; + const rotation = panelContentRotation(panel.currentOrientation); + return rotation === undefined ? undefined : { panel, rotation }; +} + +/** The gap a dispatched picker event needs before the panel table shows its effect. */ +async function settleRotation(): Promise { + await new Promise((resolve) => { + setTimeout(resolve, IOS_FOLDABLE_ORIENTATION_SETTLE_MS); + }); +} diff --git a/packages/platform-apple/src/foldable/pose.test.ts b/packages/platform-apple/src/foldable/pose.test.ts index 60d3ef89f4..0b46b1b7c7 100644 --- a/packages/platform-apple/src/foldable/pose.test.ts +++ b/packages/platform-apple/src/foldable/pose.test.ts @@ -107,6 +107,10 @@ test('sends the simulator HID pose and reports the pose the guest hinge read bac screen: { display: 'LCD-1', coordinateSpace: 'native-panel', widthPt: 669, heightPt: 951 }, }); + expect(mockInventory.mock.calls.map(([, options]) => options.source)).toEqual([ + undefined, + 'guest', + ]); expect(mockSend).toHaveBeenCalledWith('open', undefined); expect(mockHinge).toHaveBeenCalledTimes(2); // Reads are spaced so two samples of a moving hinge cannot land together; the first is not delayed. diff --git a/packages/platform-apple/src/foldable/pose.ts b/packages/platform-apple/src/foldable/pose.ts index 5985e520ca..45aa4426bd 100644 --- a/packages/platform-apple/src/foldable/pose.ts +++ b/packages/platform-apple/src/foldable/pose.ts @@ -26,7 +26,7 @@ const FOLDABLE_REQUIRED_HINT = 'fold sets the hinge angle of a foldable simulator such as iPhone Duo; this simulator reports one integrated panel, so it has no hinge to pose.'; const INVENTORY_REQUIRED_HINT = - "fold needs the simulator's panel table to tell a foldable from a single-panel simulator: 'devicectl device info displays' in the default device set, or the fold helper's guest panel read in a scoped set. The apple_display_inventory_unresolved diagnostic names why the read failed."; + "fold needs the simulator's integrated displays to identify a foldable: the default set uses CoreDevice and a scoped set uses the fold helper. The apple_display_inventory_unresolved diagnostic names why the read failed."; const DISPATCHED_UNVERIFIED_HINT = 'The pose request was sent and the device may have followed it, but the first hinge-angle read returned no sample, so fold cannot confirm it. Take a screenshot or snapshot to see which panel is lit; hingeReadFailure names why the read failed.'; @@ -45,8 +45,7 @@ export async function setAppleFoldPose( const intent = input; const targetAngle = intent.keyframes?.at(-1)?.angle; const pose = intent.pose ?? foldPoseForHingeAngle(targetAngle!)!; - // One helper serves the scoped panel reads, the dispatch, and every hinge read. It builds on its - // first use, so a default-set device that cannot fold is refused before any build. + // One helper serves the panel reads, the dispatch, and every hinge read. const helper = createFoldHelper(device); const inventory = await queryAppleDisplayInventory(device, { signal: options.signal, @@ -276,7 +275,11 @@ async function readLitPanel( foldHelper: FoldHelper, signal: AbortSignal | undefined, ): Promise { - const inventory = await queryAppleDisplayInventory(device, { signal, foldHelper }); + const inventory = await queryAppleDisplayInventory(device, { + signal, + source: 'guest', + foldHelper, + }); if (inventory.unresolved || inventory.ambiguous) return undefined; return inventory.activeDisplay; } diff --git a/src/commands/schema/cli-help-foldable.ts b/src/commands/schema/cli-help-foldable.ts index decced21ea..2e754b9b0e 100644 --- a/src/commands/schema/cli-help-foldable.ts +++ b/src/commands/schema/cli-help-foldable.ts @@ -15,7 +15,7 @@ Changing the pose: Expect a fold to take 2-4 seconds: half-open waits for two agreeing hinge reads one second apart, and the first fold after a boot can take longer while the hinge settles. Re-snapshot after every fold; refs and coordinates from before it are stale, and the command's message says so. Timed motion: fold --keyframes '[{"atMs":0,"angle":0},{"atMs":5000,"angle":180}]'. Use 2–64 frames starting at 0ms, increasing integer timestamps up to 60000ms, and angles from 0 to 180. The last timestamp sets motion duration, excluding setup and verification. Equal angles hold; cancellation stops motion. See the fold examples in the command and Node API documentation for trajectories. - Requirements: an iOS simulator session on a foldable device and an Xcode toolchain with the iOS simulator SDK and foldable HID support (verified on Xcode 27.1). Device Hub and host Accessibility permission are not required. The command runs a small helper through simctl spawn for the session UDID; the helper is built once per Fold.m source hash and Xcode toolchain, cached under ~/.agent-device/fold-helper, and rebuilt only when the source or the toolchain changes. The same helper reads the hinge and, in a scoped set, the panels. Build or dispatch failures are reported without a UI fallback. The app under test reads the resulting pose as UIHinge.status. + Requirements: an iOS simulator session on a foldable device and an Xcode toolchain with the iOS simulator SDK and foldable HID support (verified on Xcode 27.1). Device Hub and host Accessibility permission are not required. The command runs a small helper through simctl spawn for the session UDID; the helper is built once per Fold.m source hash and Xcode toolchain, cached under ~/.agent-device/fold-helper, and rebuilt only when the source or the toolchain changes. The same helper reads the hinge and the panels in either simulator set. Build or dispatch failures are reported without a UI fallback. The app under test reads the resulting pose as UIHinge.status. Android emulators: fold sets the posture through the emulator console (adb emu posture 1|2|3), then reads adb shell cmd device_state print-state until the guest commits CLOSED, HALF_OPENED, or OPENED, and reports the angle read from the hinge sensor. It can differ from the AVD profile's per-posture angles: 0, 90 and 180 degrees on the Pixel folds, 15, 90 and 165 on the generic 7.6in Foldable. Folding to the cover display can raise Android's "Swipe up to continue" lock screen; fold dismisses it so the app stays reachable. An emulator profile without those device states (a phone) fails with UNSUPPORTED_OPERATION and reason single-panel-device, and a physical device is refused. Keyframes are not supported on Android, and the response carries no panel geometry: re-snapshot after every fold. If a task asserts behavior for more than one pose, fold to each pose and re-snapshot, and report which poses the run covered.`, } as const; diff --git a/test/integration/provider-scenarios/ios-fold.test.ts b/test/integration/provider-scenarios/ios-fold.test.ts index 4dc0f01229..61634d53e9 100644 --- a/test/integration/provider-scenarios/ios-fold.test.ts +++ b/test/integration/provider-scenarios/ios-fold.test.ts @@ -43,17 +43,19 @@ function guestHingeReading(angleDegrees: number): string { return `${JSON.stringify({ version: 1, angleDegrees, angleValid: true })}\n`; } -/** Writes the `devicectl … displays --json-output ` result both fold fakes below answer. */ -function writeDisplayInventoryFixture(jsonOutputPath: string): void { +/** The fold helper's `panels` mode output for a Duo with its inner panel lit. */ +function guestPanelReading(): string { const displays = [0, 1].map((displayId) => ({ name: `LCD-${displayId}`, displayId, nativeSize: [2007, 2853], pointScale: 3, type: { integrated: {} }, - active: displayId === 1, + primary: displayId === 0, + backlightState: displayId === 1 ? 'activeOn' : 'off', + currentOrientation: 'rot0', })); - fs.writeFileSync(jsonOutputPath, JSON.stringify({ result: { displays } })); + return `${JSON.stringify({ version: 1, displays })}\n`; } /** @@ -82,6 +84,11 @@ test('timed fold keyframes reach simulator HID through the public client and dae const ok = { stdout: '', stderr: '', exitCode: 0 }; let angle = 0; const tool = createRecordingAppleToolProvider({ + devicectl: async (args) => { + assert.deepEqual(args.slice(0, 3), ['device', 'info', 'displays']); + fs.writeFileSync(args.at(-1)!, JSON.stringify({ result: JSON.parse(guestPanelReading()) })); + return ok; + }, simctl: async (args, options) => { if (args[0] === 'list') { return { @@ -97,6 +104,7 @@ test('timed fold keyframes reach simulator HID through the public client and dae if (args[0] === 'launch') return ok; assert.equal(args[0], 'spawn'); assert.equal(args[1], PROVIDER_SCENARIO_IOS_SIMULATOR.id); + if (args[3] === 'panels') return { ...ok, stdout: guestPanelReading() }; if (args[3] === 'hinge') return { ...ok, stdout: guestHingeReading(angle) }; assert.equal(args[3], 'set'); assert.deepEqual(JSON.parse(args[4]!), trajectory); @@ -104,11 +112,6 @@ test('timed fold keyframes reach simulator HID through the public client and dae angle = 100; return ok; }, - devicectl: async (args) => { - assert.ok(args.includes('displays')); - writeDisplayInventoryFixture(args[args.indexOf('--json-output') + 1]!); - return ok; - }, }); let builds = 0; const daemon = await createProviderScenarioHarness({ @@ -144,6 +147,7 @@ test('timed fold keyframes reach simulator HID through the public client and dae }); assert.equal(result.pose, 'half-open'); assert.equal(result.hingeAngleDegrees, 100); + assert.equal(result.screen?.display, 'LCD-1'); const replayed = await daemon.client().command.fold({ platform: 'ios', udid: PROVIDER_SCENARIO_IOS_SIMULATOR.id, @@ -211,6 +215,12 @@ function createFoldLedgerAppleToolProvider(params: { // Built on the shared recording provider so any call this route does not script (macosHelper, // macosHost, plist, or an unexpected runCommand) throws instead of being silently answered. const recording = createRecordingAppleToolProvider({ + devicectl: async (args, options) => { + await record('devicectl', args, options); + assert.deepEqual(args.slice(0, 3), ['device', 'info', 'displays']); + fs.writeFileSync(args.at(-1)!, JSON.stringify({ result: JSON.parse(guestPanelReading()) })); + return ok; + }, simctl: async (args, options) => { if (args[0] === 'list') { if (params.isTracking()) await record('simctl', args, options); @@ -226,6 +236,9 @@ function createFoldLedgerAppleToolProvider(params: { } if (args[0] === 'launch') return ok; await record('simctl', args, options); + if (args[0] === 'spawn' && args[3] === 'panels') { + return { ...ok, stdout: guestPanelReading() }; + } if (args[0] === 'spawn' && args[3] === 'hinge') { const angle = params.hingeAngleAt(hingeReads); hingeReads += 1; @@ -233,12 +246,6 @@ function createFoldLedgerAppleToolProvider(params: { } return ok; }, - devicectl: async (args, options) => { - await record('devicectl', args, options); - assert.ok(args.includes('displays'), `unexpected devicectl call: ${args.join(' ')}`); - writeDisplayInventoryFixture(args[args.indexOf('--json-output') + 1]!); - return ok; - }, }); const provider: AppleToolProvider = { ...recording.provider, @@ -347,15 +354,23 @@ function assertFoldLedgerPhaseCoverage(ledger: readonly FoldLedgerCall[]): void const isHingeRead = (call: FoldLedgerCall) => call.tool === 'simctl' && call.args[0] === 'spawn' && call.args[3] === 'hinge'; const isDisplayInventory = (call: FoldLedgerCall) => - call.tool === 'devicectl' && call.args.includes('displays'); + call.tool === 'simctl' && call.args[0] === 'spawn' && call.args[3] === 'panels'; + assert.equal( + ledger.filter( + (call) => + call.tool === 'devicectl' && call.args.slice(0, 3).join(' ') === 'device info displays', + ).length, + 1, + 'ledger must record the set-source foldable classification', + ); assert.ok(ledger.some(isBuild), 'ledger is missing the helper-build phase'); assert.ok(ledger.some(isSpawn), 'ledger is missing the HID-dispatch (simctl spawn) phase'); assert.ok(ledger.some(isHingeRead), 'ledger is missing the hinge-settle-read phase'); assert.equal( ledger.filter(isDisplayInventory).length, - 2, - 'ledger must record both the foldable-check and the lit-panel display inventory reads', + 1, + 'ledger must record the lit-panel guest inventory read', ); } diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index 7f6edbff4e..cbfcf77765 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -132,7 +132,7 @@ Back, orientation, and hardware buttons: - A simulator sends the swipe as a screen-edge touch. A physical device has no such touch and sends a quick fling from the leading edge instead; if that fling does not pop the screen, `back` refuses with `back_swipe_no_change`. visionOS has no back gesture and refuses before dispatch with `UNSUPPORTED_OPERATION`. - An app with a left-edge drawer may open the drawer instead of going back. When a back or close control is visible, press it by its ref. - `back --system` sends system back input: the Android back keyevent, the iOS back swipe, or the tvOS Siri Remote Menu button. macOS has no system back, so it refuses before dispatch. -- `orientation portrait|portrait-upside-down|landscape-left|landscape-right` works on iOS and Android mobile targets only. On ADB-backed Android it turns auto-rotate off, and the lock persists after the session ends until something turns it back on. A foldable iPhone simulator (iPhone Duo) refuses `portrait-upside-down`, and refuses every rotation in a scoped simulator set (`--ios-simulator-device-set`). +- `orientation portrait|portrait-upside-down|landscape-left|landscape-right` works on iOS and Android mobile targets only. On ADB-backed Android it turns auto-rotate off, and the lock persists after the session ends until something turns it back on. In both default and scoped simulator sets, an iPhone Duo classified with more than one integrated panel refuses `portrait-upside-down` and rotates through the fold helper. It succeeds only when the lit panel shows the requested orientation; if it never does, or the simulator never reports a single lit panel, the command fails with `orientation-unverified`. Unresolved guest reads are retried inside the settle loop; an unresolved final read reports `orientation-panel-table-unavailable`. The picker offsets were measured on iPhone Duo; a new multi-panel model needs its own mapping. A simulator classified with one panel, or whose first classification read is unresolved, uses the standard runner rotation. - `orientation auto` hands rotation back to the sensor on ADB-backed Android, local or Limrun (`accelerometer_rotation=1`, `user_rotation` untouched), and reports `orientation: "auto"`. It does not wait for the display to rotate, because there is no target rotation: the sensor decides, and an emulator usually settles on portrait. Every other backend refuses `auto` with `UNSUPPORTED_OPERATION`. - `action-button` presses the iPhone or iPad Action Button once. It takes no arguments, no hold duration, and no `--settle`, and it doesn't activate the session app or drive Camera Control. Other platforms refuse it, and a model without the button (an iPhone SE, most iPad simulators) fails with `UNSUPPORTED_OPERATION`. - `action-button` reports that the press was dispatched, not what it triggered. Simulators run no Shortcuts or App Intents, so verify the outcome on a physical iPhone. Expect a press to take about five seconds on a Simulator.