Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
9fe6ad3
fix(ios): rotate a foldable in a scoped simulator set
sittingmongoose Nov 11, 2026
b35461b
fix(ios): confirm the final scoped dispatch and keep picker angles pr…
sittingmongoose Nov 11, 2026
48a90b6
refactor(ios): split scoped rotation state out of the dispatch loop
sittingmongoose Nov 11, 2026
4245f65
fix(ios): use the measured outer-panel picker offset on iPhone Duo
sittingmongoose Nov 11, 2026
47874e0
refactor(ios): one panel-content parser and one offset step for scope…
sittingmongoose Nov 11, 2026
a92989b
fix(ios): a scoped rotation learns nothing from a panel that did not …
sittingmongoose Nov 11, 2026
674f670
fix(ios): a scoped rotation needs one established lit panel
sittingmongoose Nov 11, 2026
64b4ea6
fix(ios): a panel of unknown power blocks the established lit panel
sittingmongoose Nov 11, 2026
84861b5
refactor(ios): confirm every foldable rotation from its lit panel
sittingmongoose Nov 12, 2026
5f3f6f2
docs(agents): keep foldable guidance within its context budget
sittingmongoose Nov 12, 2026
0b3e642
docs(ios): reconcile foldable rotation contract and fixtures
sittingmongoose Nov 12, 2026
ae7a602
fix(ios): keep foldable classification on the simulator set source
sittingmongoose Nov 12, 2026
07ce510
fix(ios): guard Duo picker mapping and report panel read failures
sittingmongoose Nov 12, 2026
28f1ac4
docs(ios): name the unreadable panel table reason in the orientation …
sittingmongoose Nov 12, 2026
c47ed58
fix(ios): retry transient foldable panel reads without a model guard
sittingmongoose Nov 12, 2026
92e5558
docs(ios): limit the orientation tool's fold-helper route to multi-pa…
sittingmongoose Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/foldable-scoped-rotation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
'agent-device': patch
---

Fix foldable iPhone simulator rotation in scoped sets and confirm the lit panel in both default and scoped sets. Rotation in both sets uses the fold helper, with no CoreDevice orientation polling. An already visible target dispatches nothing. Incomplete, unknown-power and multi-lit panel tables neither confirm nor dispatch. A target the panel never shows fails with `orientation-unverified`.

Use the fixed iPhone Duo picker offsets measured in both sets (outer 0 degrees, inner +90 degrees), replacing runtime calibration. Inner landscape-left sends `portrait-upside-down`, mapped to the HID wire value `pud`; it remains unsupported as a user target. Delayed reads wait without resending the same picker value to the same panel.
Comment thread
sittingmongoose marked this conversation as resolved.
Resolved

Unresolved guest reads inside the settle loop neither confirm nor dispatch and are retried. An unresolved final read fails with `orientation-panel-table-unavailable`. Multi-panel classification admits rotation without a model-name guard; a new multi-panel model needs its own picker mapping.
12 changes: 8 additions & 4 deletions apple/fold-helper/Fold.m
Original file line number Diff line number Diff line change
Expand Up @@ -106,14 +106,18 @@ static int runSet(NSString *argument) {
return dispatchEvents(^BOOL(CFTypeRef client) { return runFrames(client, frames); });
}

// The orientation control has no upside-down value.
// The upside-down picker value is spelled "pud" on the wire; "portrait-upside-down" is
// ignored by the guest. It matters because each picker event rotates the display by
// (new - previous) picker value: without a 180 value, one of the four display orientations
// can never be landed (measured on iPhone Duo).
static int runRotate(NSString *orientation) {
if (![@[@"portrait", @"landscape-left", @"landscape-right"] containsObject:orientation]) {
fprintf(stderr, "Expected orientation portrait, landscape-left or landscape-right\n");
NSString *value = [orientation isEqualToString:@"portrait-upside-down"] ? @"pud" : orientation;
if (![@[@"portrait", @"landscape-left", @"landscape-right", @"pud"] containsObject:value]) {
fprintf(stderr, "Expected orientation portrait, landscape-left, landscape-right or portrait-upside-down\n");
return 2;
}
return dispatchEvents(^BOOL(CFTypeRef client) {
return sendControl(client, @"orientation-picker-control", @"enum", orientation);
return sendControl(client, @"orientation-picker-control", @"enum", value);
});
}

Expand Down
38 changes: 22 additions & 16 deletions docs/adr/0025-foldable-apple-panels.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,9 @@ It is the only reader of panel tables, and it selects the source.
For a simulator in a scoped set, the fold helper's `panels` mode is the screen authority.
It emits the same entry shape from the guest sources that `dtdeviceinfod` relays to CoreDevice: `CADisplay` and `BLSBacklight`.
The guest lists every integrated panel and omits only the facts it cannot read.
Fold pose and rotation lit-panel reads explicitly request the guest table in either simulator set.
Panel-count classification keeps the set-based source: CoreDevice in the default set, guest in a scoped set.
Rotation confirms the lit panel because CoreDevice orientation can match the picker while the visible panel shows another orientation.
- **Completeness.**
Both sources list one entry per integrated panel. A panel that the parser drops is not single-panel evidence.
If two or more integrated entries exist and not all parse, the device is multi-panel without a capture panel. Capture refuses with `capture-panel-unidentified`.
Expand Down Expand Up @@ -104,7 +107,7 @@ It is the only reader of panel tables, and it selects the source.
`IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS` limits the probe to 5s, below the subsequent 20s screenshot deadline.
Thus, an unresponsive CoreDevice cannot consume the capture's own budget.
Measured cost is 0.16–0.26s, compared with 4.93s for the screenshot.
The guest probe costs about 0.3s. `IOS_SCOPED_DISPLAY_PROBE_BUDGET_MS` limits it to 15s, including the first helper build.
The guest probe costs about 0.3s. `IOS_GUEST_DISPLAY_PROBE_BUDGET_MS` limits it to 15s, including the first helper build.

## Why the default capture had to stop being implicit

Expand Down Expand Up @@ -370,19 +373,27 @@ Rejected alternatives:
`XCUIDevice.orientation` has no effect on an iPhone Duo simulator, and the runner readback repeats the requested value.
`devicectl device orientation set` reports success and has no effect on a simulator that `simctl boot` started.
`orientation` therefore rotates a multi-panel simulator through the guest device-state control: the HID event that Device Hub's orientation menu sends (`orientation-picker-control`, type `enum`).
The control has no upside-down value, so `portrait-upside-down` is refused with `unsupported-foldable-orientation`.
Routing uses the `fold` admission and the panel table. A resolved panel count is a hardware fact, so the daemon probes each simulator once.

The verdict comes from `devicectl device orientation get`, a process that never sent the HID event.
CoreDevice answers each query with the value that it sampled at the previous query, so the first read after a dispatch is discarded.
A rotation that no fresh read reports is refused with `orientation-unverified`.
A simulator in a scoped set is refused with `unsupported-device-scope` before dispatch, because no reader can verify its rotation.
The picker accepts `pud` for upside-down dispatch, which the inner panel needs to show landscape-left.
The user target `portrait-upside-down` remains refused with `unsupported-foldable-orientation`.
Routing uses the `fold` admission and the set-based panel-count source. A resolved panel count is a hardware fact, so the daemon probes each simulator once.
A supplied helper is a reuse handle; only an explicit guest-source request changes the default-set source.
Thus, classifying a single-panel default-set simulator does not prepare the fold helper.

The picker offsets were measured on iPhone Duo; a new multi-panel model needs its own mapping.
In both simulator sets, the helper reads the guest panel table and dispatches the fixed picker value for the established lit panel.
The table maps the outer panel at zero offset and the inner panel at +90 degrees.
Only a fresh `currentOrientation` from an established lit panel confirms the requested visible orientation.
An incomplete table, unknown-power sibling, or multiple lit panels cannot confirm or dispatch.
A repeated visible target dispatches nothing; delayed reads wait without resending to the same panel.
A target that never appears is refused with `orientation-unverified` in either set.
An unresolved guest read inside the settle loop neither confirms nor dispatches; the loop retries it.
An unresolved final read reports `orientation-panel-table-unavailable` and points to the inventory diagnostic.

Rejected alternatives:

- **The panel table's `orientation.currentDeviceOrientation`: rejected.** It is screen-relative. On the Duo inner panel, it is a quarter-turn from the device orientation.
- **A guest orientation read: rejected.** CoreMotion `CMDeviceOrientationManager` reports `Ambiguous`, and `BKSHIDServicesIsOrientationLockedWithOrientation` reports 0, after a rotation that CoreDevice and the screenshots confirm.
- **Dispatch without a verdict in a scoped set: rejected.** The runner path already reported a false success there.
- **CoreDevice orientation readback: rejected.** Its picker sample can lag and can match the request while the visible inner panel is a quarter-turn away.
- **Other guest device-orientation APIs: rejected.** CoreMotion `CMDeviceOrientationManager` reports `Ambiguous`, and `BKSHIDServicesIsOrientationLockedWithOrientation` reports 0 after a rotation.
- **Dispatch without a lit-panel verdict: rejected.** An acknowledgement alone can report false success.

## Pose is derived from panel power, and official control does not exist

Expand Down Expand Up @@ -678,11 +689,6 @@ The silent misses investigated here came from default outer-display routes and i
`AgentDeviceMacOSHelper` also reaches it through Screen Capture Kit.
Only iOS branches changed target.
Desktop behavior remains unmeasured here.
- **Quarter-turn detection.**
Both Duo panels report `currentOrientation: rot90`.
Inventory carries orientation without a test against changed orientation.
`orientation` now rotates a Duo simulator in the default set through guest HID (2026-10-26).
The changed `currentOrientation` values have no test yet.
- **A slowly settled hinge.**
Four reads consume twenty seconds of streams.
A Duo that takes longer to settle inside `half-open` is refused.
Expand Down
13 changes: 7 additions & 6 deletions docs/agents/device-foldable.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
# Foldable Apple verification

Before you change capture behavior on a multi-panel device, read ADR 0025.
To list the panels, run `xcrun devicectl device info displays --device <udid>`.
In a scoped set, run `xcrun simctl --set <set> spawn <udid> ~/.agent-device/fold-helper/<key>/fold-helper panels`.
Before capture changes, read ADR 0025.
For default-set capture, run `xcrun devicectl device info displays --device <udid>`.
For scoped capture or rotation, run `xcrun simctl --set <set> spawn <udid> ~/.agent-device/fold-helper/<key>/fold-helper panels`.
For default-set rotation, omit `--set <set>`.

- Panels: the list shows each integrated panel with `backlightState`. Only the lit panel permits capture.
A capture of the dark panel exits 0 and writes an all-black PNG.
A dark-panel capture exits 0 with a black PNG.
- Input routing: after each pose change, verify a fresh control.
A successful synthesis acknowledgement does not prove a hit.
Examine the runner and simulator `testmanagerd`/BackBoard logs for display identity and delivery.
Expand All @@ -14,13 +15,13 @@ In a scoped set, run `xcrun simctl --set <set> spawn <udid> ~/.agent-device/fold
The same helper reads the hinge back in `hinge` mode.
After a fold, take another snapshot. Folds invalidate refs and coordinates.
Verify locally with Device Hub stopped and the simulator booted through `simctl boot`.
Test these items in the default set and with `--ios-simulator-device-set`:
Test both sets (`--ios-simulator-device-set`):
- All three poses.
- Capture of the active panel.
- An app interaction.
Until GHA supports the runtime, Duo coverage remains local.
After a fresh boot, a closed hinge can keep the inner panel lit for more than 30 seconds.
The lit panel is evidence only. It cannot separate `half-open` from `open`.
Panel power cannot separate `half-open` from `open`.
- Touch overlays export at the captured track size.
The compositor checks its output size and non-black content against the raw capture.
If that check fails, `record stop` keeps the raw file. It reports `overlayWarning`.
Expand Down
2 changes: 1 addition & 1 deletion packages/command-surface/src/system/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,7 @@ const backCommandDescription =
const homeCommandDescription =
'Send the selected device to its home screen. This leaves the app session open but moves the foreground away from the app.';
const orientationCommandDescription =
'Set device orientation on iOS and Android. auto hands rotation back to the sensor on ADB-backed Android (local and Limrun); other backends refuse it. A foldable iPhone simulator (iPhone Duo) refuses portrait-upside-down, and refuses every rotation in a scoped simulator set.';
'Set device orientation on iOS and Android. auto hands rotation back to the sensor on ADB-backed Android (local and Limrun); other backends refuse it. In both default and scoped simulator sets, an iPhone Duo classified with more than one integrated panel refuses portrait-upside-down and rotates through the fold helper; it succeeds only when the lit panel shows the requested orientation, otherwise it fails with orientation-unverified, or orientation-panel-table-unavailable when the final guest panel table cannot be read. A simulator classified with one panel, or whose first classification read is unresolved, uses the standard runner rotation.';
const foldGuidance = {
hingeEvent: 'simulator HID hinge event',
permissions: 'Device Hub and host Accessibility permission are not required.',
Expand Down
100 changes: 98 additions & 2 deletions packages/platform-apple/src/core/__tests__/display-inventory.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,15 @@ vi.mock('@agent-device/host-kit/host-file', async (importOriginal) => {
});

import {
appleLitPanel,
appleSimulatorDisplayArgvFragment,
establishedLitPanel,
buildInventory,
parseCoreDeviceDisplays,
readPanelPower,
resolveAppleCaptureDisplay,
queryAppleDisplayInventory,
panelContentRotation,
resolveAppleCaptureTarget,
type AppleDeviceDisplay,
} from '../display-inventory.ts';
Expand Down Expand Up @@ -362,6 +365,75 @@ describe('resolveAppleCaptureDisplay', () => {
});
});

describe('panelContentRotation', () => {
test('parses each panel content rotation and nothing else', () => {
assert.equal(panelContentRotation('rot0'), 'portrait');
assert.equal(panelContentRotation('rot90'), 'landscape-right');
assert.equal(panelContentRotation('rot180'), 'portrait-upside-down');
assert.equal(panelContentRotation('rot270'), 'landscape-left');
for (const value of ['unknown', 'constructor', 'rot45', '']) {
assert.equal(panelContentRotation(value), undefined);
}
});
});

describe('establishedLitPanel', () => {
beforeEach(() => {
mockReadHostTextFile.mockReset();
mockReadHostTextFile.mockResolvedValue('');
});

function duoTable(mapEntry: (entry: Record<string, unknown>) => Record<string, unknown>) {
return JSON.stringify({
result: { displays: IPHONE_DUO_CLOSED.result.displays.map(mapEntry) },
});
}

test('a complete table with one lit panel establishes that panel for both selectors', () => {
const inventory = buildInventory([outerPanel, innerPanel]);
assert.equal(establishedLitPanel(inventory), outerPanel);
assert.deepEqual(appleLitPanel(inventory), { displayId: 1, widthPt: 466, heightPt: 678 });
});

test('two lit panels establish none', () => {
const inventory = buildInventory([outerPanel, { ...innerPanel, power: 'lit' }]);
assert.equal(establishedLitPanel(inventory), undefined);
assert.equal(appleLitPanel(inventory), undefined);
});

test('a lit panel beside a panel of unknown power establishes none', () => {
// Unknown power may be showing content, so the lit panel is not the only one that may be lit.
const inventory = buildInventory([outerPanel, { ...innerPanel, power: 'unknown' }]);
assert.equal(establishedLitPanel(inventory), undefined);
assert.equal(appleLitPanel(inventory), undefined);
});

test('an incomplete table establishes no panel even when one parsed panel is the only lit one', async () => {
// Two activeOn integrated entries; the inner one has no geometry and does not parse.
mockReadHostTextFile.mockResolvedValue(
duoTable((entry) => {
const lit: Record<string, unknown> = { ...entry, backlightState: 'activeOn' };
if (entry.name !== 'LCD-1') return lit;
const { nativeSize: _nativeSize, ...withoutGeometry } = lit;
return withoutGeometry;
}),
);
const inventory = await queryAppleDisplayInventory(IOS_TEST_SIMULATOR, {});
assert.equal(inventory.multiScreen, true);
assert.equal(inventory.displays.filter((display) => display.power === 'lit').length, 1);
assert.equal(establishedLitPanel(inventory), undefined);
assert.equal(appleLitPanel(inventory), undefined);
});

test('a single-panel or unresolved table establishes no foldable panel', () => {
assert.equal(establishedLitPanel(buildInventory([outerPanel])), undefined);
assert.equal(
establishedLitPanel({ displays: [], multiScreen: false, ambiguous: false, unresolved: true }),
undefined,
);
});
});

describe('resolveAppleCaptureTarget', () => {
beforeEach(() => {
mockRunXcrun.mockClear();
Expand Down Expand Up @@ -678,7 +750,10 @@ describe('scoped simulator panel table', () => {
assert.equal(mockRunXcrun.mock.calls.length, 0);
});

test('reads with the fold helper the caller already opened', async () => {
test.each([
['scoped', scopedDuo],
['default', { ...scopedDuo, simulatorSetPath: undefined }],
] as const)('reads with the supplied fold helper in the %s set', async (_set, device) => {
vi.mocked(ensureFoldHelperBinary).mockClear();
const readPanelPayload = vi.fn(async () => ({
result: { displays: (JSON.parse(GUEST_PANELS_OPEN) as { displays: unknown[] }).displays },
Expand All @@ -690,12 +765,33 @@ describe('scoped simulator panel table', () => {
readPanelPayload,
};

const inventory = await queryAppleDisplayInventory(scopedDuo, { foldHelper });
const inventory = await queryAppleDisplayInventory(device, { source: 'guest', foldHelper });

assert.equal(inventory.activeDisplay?.name, 'LCD-1');
assert.equal(readPanelPayload.mock.calls.length, 1);
assert.equal(vi.mocked(ensureFoldHelperBinary).mock.calls.length, 0);
assert.equal(mockRunSimctlForDevice.mock.calls.length, 0);
assert.equal(mockRunXcrun.mock.calls.length, 0);
});

test('a helper handle alone keeps the default-set CoreDevice source', async () => {
mockReadHostTextFile.mockResolvedValue(JSON.stringify(IPHONE_DUO_CLOSED));
const readPanelPayload = vi.fn(async () => {
throw new Error('guest must not classify');
});
const foldHelper = {
setPose: vi.fn(),
rotate: vi.fn(),
readHingeAngle: vi.fn(),
readPanelPayload,
};
const inventory = await queryAppleDisplayInventory(
{ ...scopedDuo, simulatorSetPath: undefined },
{ foldHelper },
);
assert.equal(inventory.multiScreen, true);
assert.equal(readPanelPayload.mock.calls.length, 0);
assert.equal(mockRunXcrun.mock.calls.length, 1);
});

test('applies the same rule to a CoreDevice table whose second panel does not parse', async () => {
Expand Down
15 changes: 6 additions & 9 deletions packages/platform-apple/src/core/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,10 @@ export const IOS_SIMULATOR_SCREENSHOT_TIMEOUT_MS = 20_000;
// request-level daemon reset. Measured probe cost is ~0.2s.
export const IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS = 5_000;
/**
* A scoped simulator's panel table comes from a guest helper that the first read on a host also
* A guest panel table comes from a helper that the first read on a host also
* builds. This ceiling on the whole read, build included, stays below the capture's own budget.
*/
export const IOS_SCOPED_DISPLAY_PROBE_BUDGET_MS = 15_000;
export const IOS_GUEST_DISPLAY_PROBE_BUDGET_MS = 15_000;
Comment thread
sittingmongoose marked this conversation as resolved.
Resolved
/**
* One guest hinge read, before its sampling window: the fold helper waits at most 3s for its first
* CoreMotion sample, so this exec deadline only guards a wedged `simctl spawn`.
Expand All @@ -55,14 +55,11 @@ export const IOS_FOLD_POSE_READ_INTERVAL_MS = 1_000;
/** Two consecutive reads this close together mean the hinge has stopped moving. */
export const IOS_FOLD_POSE_STABLE_DEGREES = 0.5;
/**
* How many fresh CoreDevice orientation reads a dispatched foldable rotation gets before it is
* refused. One read costs ~0.2s, and the iPhone Duo reported the new orientation on the first
* fresh read; the margin covers a loaded host. A read that times out counts as one that saw no
* orientation. With the routing probe, a cold fold-helper build, and the HID dispatch, the worst
* case stays inside the default 90s request envelope.
* Bounded lit-panel reads cover delayed HID effects and the two-lit interval after a fold.
* A fixed picker value is sent once per lit panel; the remaining reads only observe.
*/
export const IOS_FOLDABLE_ORIENTATION_SETTLE_ATTEMPTS = 8;
export const IOS_FOLDABLE_ORIENTATION_READ_TIMEOUT_MS = 2_000;
export const IOS_FOLDABLE_ORIENTATION_SETTLE_ATTEMPTS = 6;
export const IOS_FOLDABLE_ORIENTATION_SETTLE_MS = 600;

// CoreSimulator can briefly stall while it services the scale lookup immediately
// after a keyboard transition. Keep this bounded below the full capture budget.
Expand Down
Loading
Loading