Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions docs/adr/0025-foldable-apple-panels.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Only inference answers the second question.
| More than one integrated panel | Pass `--display=<panel name>` on every `simctl io screenshot`. Never use the implicit default. |
| Zero or multiple lit panels with a `primary` panel | Name the `primary` panel. Emit `apple_display_capture_ambiguous`. Report pose as `unknown`. |
| No sole lit panel and no `primary` panel | Refuse with `COMMAND_FAILED` and `details.reason: capture-panel-unidentified`. Never choose a panel by array order. |
| One integrated panel | Preserve pre-panel behavior exactly: no display flag, no pose, unchanged scale probe. |
| One integrated panel | Preserve pre-panel behavior exactly: no display flag, no pose, unchanged scale probe. Once a resolved table shows one panel, read the table beside the capture for `displayRotation` only. |
| Density normalization | Use the captured panel's `pointScale`. For runner fallback, use that capture's reported pixels-per-point. |
| Pose report | From panel power alone, report only `closed`, `fully-open`, or `unknown`, never a narrower pose. `fold` reports an exact pose only after hinge readings verify it. |
| Pose request | `agent-device fold <closed\|half-open\|open>` sends private HID inside the simulator. Then it reads the hinge angle back inside the simulator until the angle agrees. For `half-open`, two consecutive readings must agree. If the readings never agree, refuse the pose. |
Expand Down Expand Up @@ -100,8 +100,8 @@ It is the only reader of panel tables, and it selects the source.
The inventory reports measured `primary`, panel power, geometry, and orientation.
It leaves exact pose to the app under test through `UIHinge.status`.
If a caller needs pose, it asks the operator and takes another snapshot.
- **No cache.**
The probe executes per capture.
- **No cached lit-panel inventory.**
The probe executes per capture. The daemon caches only each device's multi-panel verdict, a hardware fact (see the 2026-11-16 amendment).
An operator changes the lit panel through a fold or open action.
Cached inventory could resume capture of the dark panel and recreate the failure.
`IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS` limits the probe to 5s, below the subsequent 20s screenshot deadline.
Expand Down Expand Up @@ -755,3 +755,19 @@ Consequences:
- `runner.log` prints `AGENT_DEVICE_RUNNER_LIT_PANEL_WINDOW lit=<id>(<w>x<h>) windows=<n> outcome=<reason> frame=(…)`. `SYNTHESIZED_DISPATCH reference=` must equal that frame. Snapshot points are translated by that frame's origin, as for a placed iPad window, and a synthesized turn is about the lit panel's screen in the app's interface orientation, not the window's shape: an app beside the inner panel's 80 pt status strip still maps into the whole screen. `SYNTHESIZED_RECORD displayID=` must equal `lit=`.
- A recording keeps the panel that its `recordStart` named. The runner cannot read panel power, so it does not notice a fold during the recording: SpringBoard keeps windows on the panel that went dark, and later frames show that dark panel. Start a new recording after a fold.
- Open gap: on the folded home screen, XCTest reports the page icons with the inner panel's layout in both panels' windows. The Dock and the page indicator report the outer panel correctly.

## Amendment: a single-panel table read runs beside the capture (2026-11-16)

This amendment narrows the capture rule "first resolve the panel table" to devices known to be single-panel.
The display inventory caches each device's multi-panel verdict from its latest resolved table for the daemon's lifetime; an unresolved read does not change it.
When the verdict is single-panel, the capture names no display, so the table read only supplies `displayRotation`.
That read then starts beside `simctl io screenshot` with a 2s budget, in both simulator sets.
A read still running 250ms after the image is written is aborted, and the capture reports no rotation, as for an unresolved table.
A multi-panel device or a device with no resolved verdict keeps the read before the capture, so a fold still names the lit panel.
If a read beside a capture resolves to a multi-panel table, including one that refuses with `capture-panel-unidentified`, that capture has already gone out without `--display`. It takes simctl's implicit default and can be the dark-panel black image.
The capture then emits `apple_display_single_panel_count_stale` with a retake hint, and the resolved read replaces the verdict, so the next capture reads the table before it captures and names the lit panel.

Rejected alternatives:

- **A cached panel table: rejected.** The lit panel changes on a fold, and rotation changes without an agent-device command.
- **Waiting for the read without a bound: rejected.** Rotation is optional metadata and must not delay the image.
4 changes: 4 additions & 0 deletions packages/capture-kit/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@
"types": "./src/capture-admission/audio-probe-session-resource.ts",
"default": "./src/capture-admission/audio-probe-session-resource.ts"
},
"./capture-fact-grace": {
"types": "./src/capture-fact-grace.ts",
"default": "./src/capture-fact-grace.ts"
},
"./durable-capture": {
"types": "./src/durable-capture/index.ts",
"default": "./src/durable-capture/index.ts"
Expand Down
21 changes: 21 additions & 0 deletions packages/capture-kit/src/capture-fact-grace.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
/**
* Waits at most `graceMs` for optional capture metadata that is read beside the image, such as
* the display rotation. A read still running at the deadline yields `undefined`; the caller then
* aborts it and reports the fact as absent.
*/
export async function settledWithin<T>(
pending: Promise<T>,
graceMs: number,
): Promise<T | undefined> {
let timer: NodeJS.Timeout | undefined;
try {
return await Promise.race([
pending,
new Promise<undefined>((resolve) => {
timer = setTimeout(() => resolve(undefined), graceMs);
}),
]);
} finally {
clearTimeout(timer);
}
}
15 changes: 1 addition & 14 deletions packages/platform-android/src/screenshot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { runAndroidExecOut, runAndroidShell, sleep } from './adb.ts';
import { emitAndroidAdbDiagnostic, requireAndroidAdbHost } from './adb-host.ts';
import { androidAdbResultError } from './adb-executor.ts';
import { probeAndroidDisplayRotation } from './input-actions.ts';
import { settledWithin } from '@agent-device/capture-kit/capture-fact-grace';

// PNG file signature: 0x89 P N G \r \n 0x1A \n
const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
Expand Down Expand Up @@ -123,20 +124,6 @@ async function captureAndroidScreenshotWithRotation(
}
}

async function settledWithin<T>(pending: Promise<T>, graceMs: number): Promise<T | undefined> {
let timer: NodeJS.Timeout | undefined;
try {
return await Promise.race([
pending,
new Promise<undefined>((resolve) => {
timer = setTimeout(() => resolve(undefined), graceMs);
}),
]);
} finally {
clearTimeout(timer);
}
}

async function captureAndroidScreenshot(device: DeviceInfo, outPath: string): Promise<void> {
const { runAndroidAutomationHelper } = await import('./automation-helper.ts');
const helper = await runAndroidAutomationHelper(device, { action: 'screenshot' });
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import {
establishedLitPanel,
buildInventory,
parseCoreDeviceDisplays,
knownAppleMultiScreen,
readPanelPower,
resolveAppleCaptureDisplay,
queryAppleDisplayInventory,
Expand Down Expand Up @@ -567,6 +568,82 @@ describe('resolveAppleCaptureTarget', () => {
});
});

describe('knownAppleMultiScreen', () => {
beforeEach(() => {
mockRunXcrun.mockClear();
mockEmitDiagnostic.mockClear();
mockReadHostTextFile.mockReset();
mockReadHostTextFile.mockResolvedValue('');
});

test('keeps the panel count of the latest resolved table, not of an unresolved one', async () => {
const device = { ...IOS_TEST_SIMULATOR, id: `${IOS_TEST_SIMULATOR.id}-panel-count` };
assert.equal(knownAppleMultiScreen(device), undefined);

mockReadHostTextFile.mockResolvedValueOnce(JSON.stringify(IPHONE_17_SINGLE_PANEL));
await queryAppleDisplayInventory(device, {});
assert.equal(knownAppleMultiScreen(device), false);

mockRunXcrun.mockResolvedValueOnce({
exitCode: 1,
stdout: '',
stderr: 'The specified device was not found',
} as Awaited<ReturnType<typeof runXcrun>>);
assert.equal((await queryAppleDisplayInventory(device, {})).unresolved, true);
assert.equal(knownAppleMultiScreen(device), false);

mockReadHostTextFile.mockResolvedValueOnce(JSON.stringify(IPHONE_DUO_CLOSED));
await queryAppleDisplayInventory(device, {});
assert.equal(knownAppleMultiScreen(device), true);
});

test('a refused multi-panel table still records the multi-panel verdict', async () => {
// A single-panel capture's read beside it relies on this to send the next capture through the
// table read, even when the table names no panel to capture.
const device = { ...IOS_TEST_SIMULATOR, id: `${IOS_TEST_SIMULATOR.id}-unidentified` };
mockReadHostTextFile.mockResolvedValueOnce(JSON.stringify(IPHONE_17_SINGLE_PANEL));
await queryAppleDisplayInventory(device, {});
assert.equal(knownAppleMultiScreen(device), false);

mockReadHostTextFile.mockResolvedValueOnce(
JSON.stringify({
result: {
displays: IPHONE_DUO_CLOSED.result.displays.map((entry) => ({
...entry,
primary: false,
backlightState: 'activeOn',
})),
},
}),
);
await assert.rejects(
resolveAppleCaptureTarget(device),
(error: unknown) =>
error instanceof AppError && error.details?.reason === 'capture-panel-unidentified',
);
assert.equal(knownAppleMultiScreen(device), true);
});

test('a read its caller aborted reports the cancellation, not a toolchain failure', async () => {
// Callers' hints (fold, orientation) point at this diagnostic, so it must fire and say why.
const device = { ...IOS_TEST_SIMULATOR, id: `${IOS_TEST_SIMULATOR.id}-aborted-read` };
const controller = new AbortController();
mockRunXcrun.mockImplementationOnce(async () => {
controller.abort();
throw new Error('aborted');
});
const inventory = await queryAppleDisplayInventory(device, { signal: controller.signal });
assert.equal(inventory.unresolved, true);
const unresolved = mockEmitDiagnostic.mock.calls
.map(([event]) => event)
.filter((event) => event.phase === 'apple_display_inventory_unresolved');
assert.equal(unresolved.length, 1);
assert.equal(unresolved[0]?.data?.reason, 'probe-canceled');
assert.equal(unresolved[0]?.data?.hint, undefined);
assert.equal(knownAppleMultiScreen(device), undefined);
});
});

describe('appleSimulatorDisplayArgvFragment', () => {
test('names the panel for capture commands that share the same implicit default', () => {
assert.deepEqual(appleSimulatorDisplayArgvFragment(outerPanel), ['--display=LCD']);
Expand Down Expand Up @@ -621,6 +698,7 @@ describe('captureSimulatorScreenshotWithFallback runner fallback', () => {
pixelDensity: 2,
deps: {
ensureBooted: async () => {},
isKnownSinglePanel: () => false,
resolveCaptureTarget: async () => ({ display: litPanel }),
captureWithRetry: async () => {
throw new Error('simctl screenshot failed');
Expand Down Expand Up @@ -653,6 +731,7 @@ describe('captureSimulatorScreenshotWithFallback runner fallback', () => {
pixelDensity: 2,
deps: {
ensureBooted: async () => {},
isKnownSinglePanel: () => false,
resolveCaptureTarget: async () => ({ display: outerPanel }),
captureWithRetry: async () => {
throw new Error('simctl screenshot failed');
Expand Down Expand Up @@ -680,6 +759,7 @@ describe('captureSimulatorScreenshotWithFallback runner fallback', () => {
pixelDensity: 2,
deps: {
ensureBooted: async () => {},
isKnownSinglePanel: () => false,
resolveCaptureTarget: async () => ({ display: outerPanel }),
captureWithRetry: async () => {},
normalizeDensity: async (_device, _path, _density, sourcePixelDensity) => {
Expand Down
Loading
Loading