Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions docs/evidence/ios-burst-capture-2026-11-13.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Screenshot burst on an iOS simulator, 2026-11-13

Live evidence for #4291 at head `7f3741906` (the format-only commit after the
burst fixes; the behavior this documents is unchanged since): the burst path
Comment on lines +3 to +4

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This says 7f3741906 was the format-only commit after the burst fixes, but that SHA is the simulator evidence commit; 77cc16f is the formatter commit and the current head is later. Describe this as the run commit, or cite the actual head, so readers can reproduce the evidence without ambiguous provenance.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/evidence/ios-burst-capture-2026-11-13.md, line 3:

<comment>This says `7f3741906` was the format-only commit after the burst fixes, but that SHA is the simulator evidence commit; `77cc16f` is the formatter commit and the current head is later. Describe this as the run commit, or cite the actual head, so readers can reproduce the evidence without ambiguous provenance.</comment>

<file context>
@@ -1,7 +1,12 @@
 
-Live evidence for #4291: the burst path (`--count`/`--interval-ms`) captured
-five frames of an animation on real simulator dispatch. The screen was a page
+Live evidence for #4291 at head `7f3741906` (the format-only commit after the
+burst fixes; the behavior this documents is unchanged since): the burst path
+(`--count`/`--interval-ms`) captured five frames of an animation on real
</file context>
Suggested change
Live evidence for #4291 at head `7f3741906` (the format-only commit after the
burst fixes; the behavior this documents is unchanged since): the burst path
Live evidence for #4291 from the simulator run at commit `7f3741906` (not the
current PR head): the burst path

(`--count`/`--interval-ms`) captured five frames of an animation on real
simulator dispatch. The gaps between the recorded `capturedAtMs` values run
724-909 ms — above the plan's 500 ms per-frame budget, because each frame also
pays the capture round trip the budget's doc comment excludes; the budget

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This evidence misstates the 500 ms term as a per-frame admission-wait bound that excludes capture time. perFrameBudgetMs is only the capture-cost estimate used by the whole-run admission calculation, and the burst loop imposes no 500 ms per-frame deadline. Describe the gaps as including the 200 ms wait and capture round trip, and call 500 ms an admission estimate.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/evidence/ios-burst-capture-2026-11-13.md, line 8:

<comment>This evidence misstates the 500 ms term as a per-frame admission-wait bound that excludes capture time. `perFrameBudgetMs` is only the capture-cost estimate used by the whole-run admission calculation, and the burst loop imposes no 500 ms per-frame deadline. Describe the gaps as including the 200 ms wait and capture round trip, and call 500 ms an admission estimate.</comment>

<file context>
@@ -3,10 +3,10 @@
-admission wait per frame, not the full round trip. The screen was a page
+simulator dispatch. The gaps between the recorded `capturedAtMs` values run
+724-909 ms — above the plan's 500 ms per-frame budget, because each frame also
+pays the capture round trip the budget's doc comment excludes; the budget
+bounds the admission wait per frame, not the full round trip. The screen was a page
 served to Safari on the booted iPhone 17 Pro simulator
</file context>

bounds the admission wait per frame, not the full round trip. The screen was a page
served to Safari on the booted iPhone 17 Pro simulator
(`C1C05FFF-29AD-40D8-94E5-D9CBB25EC6BB`, iOS 26.5) whose content updates every
50 ms (a millisecond clock and a progress bar), so consecutive frames cannot
hash the same.

## The run

`agent-device screenshot /tmp/ad-burst/frames/page.png --count 5 --interval-ms
200 --platform ios --json` at the pushed head:
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved

```json
{
"frames": [
{ "path": "/tmp/ad-burst/frames/page-0001.png", "capturedAtMs": 1794583109757 },
{ "path": "/tmp/ad-burst/frames/page-0002.png", "capturedAtMs": 1794583110561 },
{ "path": "/tmp/ad-burst/frames/page-0003.png", "capturedAtMs": 1794583111470 },
{ "path": "/tmp/ad-burst/frames/page-0004.png", "capturedAtMs": 1794583112194 },
{ "path": "/tmp/ad-burst/frames/page-0005.png", "capturedAtMs": 1794583112995 }
]
}
```

`capturedAtMs` strictly increases. The five files on disk:

```
-rw-r--r-- 20251 page-0001.png
-rw-r--r-- 20457 page-0002.png
-rw-r--r-- 19326 page-0003.png
-rw-r--r-- 20858 page-0004.png
-rw-r--r-- 20986 page-0005.png
```

Five distinct md5 hashes — every frame captured a different render of the
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
animation, and all five landed on disk beside the `--out` path:

```
4a943cceb3b7468c40bc438b110980d4 page-0001.png
7b9451fa6875aa2a1147ddeec78aee46 page-0002.png
bda33d6c1e3ec2eb9b49c77dd690d265 page-0003.png
df7a320a07e242ff281abea1d0358ec1 page-0004.png
eb77c0c4882b8d2093c70a23b65e2cda page-0005.png
```

The command's `--json` envelope (abridged to the identifying fields):

```json
{
"success": true,
"data": {
"path": "/tmp/ad-burst/frames/page-0001.png",
"frames": [ "... five frames as shown above ..." ],
"message": "Saved 5 screenshots starting at /tmp/ad-burst/frames/page-0001.png"
}
}
```
69 changes: 69 additions & 0 deletions packages/client/src/__tests__/screenshot-result.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import assert from 'node:assert/strict';
import { test } from 'vitest';
import { SCREENSHOT_BURST_LIMITS } from '@agent-device/contracts/capture';
import {
normalizeScreenshotCaptureResult,
pickScreenshotResultData,
Expand Down Expand Up @@ -57,3 +58,71 @@ test('normalizeScreenshotCaptureResult drops a display rotation outside the rota
);
assert.ok(!('displayRotation' in result));
});

// The default daemon payload reaches this normalizer on the SDK's default path and, through
// `pickScreenshotResultData`, on the CLI's default `--json` path. A burst whose frames the
// normalizer dropped would break the result's "reports every frame" promise at both boundaries.
test('normalizeScreenshotCaptureResult carries the burst frames the daemon reported', () => {
const result = normalizeScreenshotCaptureResult(
{
path: '/tmp/burst/page-0001.png',
width: 100,
height: 50,
frames: [
{ path: '/tmp/burst/page-0001.png', capturedAtMs: 1_000, displayRotation: 'portrait' },
{ path: '/tmp/burst/page-0002.png', capturedAtMs: 1_400 },
],
},
'qa',
);
assert.deepEqual(result.frames, [
{ path: '/tmp/burst/page-0001.png', capturedAtMs: 1_000, displayRotation: 'portrait' },
{ path: '/tmp/burst/page-0002.png', capturedAtMs: 1_400 },
]);
assert.equal(result.path, '/tmp/burst/page-0001.png');
assert.deepEqual(result.identifiers, { session: 'qa' });
});

test('normalizeScreenshotCaptureResult keeps only frames with a path and a wall clock, within the burst bound', () => {
const frames: unknown[] = [
{ path: '/tmp/burst/f-0.png', capturedAtMs: 0 },
// Malformed INSIDE the bound: a missing wall clock filters out and the
// frames behind it slide forward to fill the slot.
{ path: '/tmp/burst/f-1.png', capturedAtMs: Number.NaN },
...Array.from({ length: SCREENSHOT_BURST_LIMITS.maxCount }, (_, index) => ({
Comment thread
strzhao marked this conversation as resolved.
Resolved
path: `/tmp/burst/extra-${index}.png`,
capturedAtMs: 100 + index,
})),
{ capturedAtMs: 9_999 },
{ path: '/tmp/burst/extra.png' },
'frame',
];
const result = normalizeScreenshotCaptureResult({ path: '/tmp/burst/f-0.png', frames }, 'qa');
// The truncation cuts the raw list at the bound first; the malformed entry
// inside it then filters out, so the kept total is one below the bound.
assert.equal(result.frames?.length, SCREENSHOT_BURST_LIMITS.maxCount - 1);
assert.deepEqual(result.frames?.[0], { path: '/tmp/burst/f-0.png', capturedAtMs: 0 });
// The malformed inside-the-bound frame is filtered, and the frame behind it
// slid forward into the slot with its own path and a finite wall clock.
assert.equal(
result.frames?.some((frame) => frame.path === '/tmp/burst/f-1.png'),
false,
);
assert.equal(result.frames?.[1]?.path, '/tmp/burst/extra-0.png');
assert.equal(Number.isFinite(result.frames?.[1]?.capturedAtMs), true);
});

test('normalizeScreenshotCaptureResult omits empty and absent frames', () => {
assert.ok(!('frames' in normalizeScreenshotCaptureResult({ path: '/tmp/a.png' }, 'qa')));
assert.ok(
!('frames' in normalizeScreenshotCaptureResult({ path: '/tmp/a.png', frames: [] }, 'qa')),
);
});

test('pickScreenshotResultData keeps the frames a normalized result carried', () => {
const frames = [{ path: '/tmp/a-0001.png', capturedAtMs: 1 }];
assert.deepEqual(pickScreenshotResultData({ path: '/tmp/a-0001.png', frames }), {
path: '/tmp/a-0001.png',
frames,
});
});
35 changes: 34 additions & 1 deletion packages/client/src/screenshot-result.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
import type { ScreenshotResultData } from '@agent-device/contracts/capture';
import {
SCREENSHOT_BURST_LIMITS,
type ScreenshotBurstFrame,
type ScreenshotResultData,
} from '@agent-device/contracts/capture';
import type { CaptureScreenshotResult } from '@agent-device/contracts/client';
import { isDeviceRotation } from '@agent-device/contracts/device';
import { isRecord, parsePoint, parseRect, readRequiredString } from '@agent-device/kernel/record';
import { readResponseWarnings } from '@agent-device/kernel/success-text';
import type { ScreenshotOverlayRef } from '@agent-device/kernel/snapshot';

export function pickScreenshotResultData(value: ScreenshotResultData): ScreenshotResultData {
const frames = readScreenshotBurstFrames(value.frames);
return {
...(typeof value.path === 'string' ? { path: value.path } : {}),
...(typeof value.width === 'number' ? { width: value.width } : {}),
Expand All @@ -14,6 +19,7 @@ export function pickScreenshotResultData(value: ScreenshotResultData): Screensho
...(typeof value.logicalHeight === 'number' ? { logicalHeight: value.logicalHeight } : {}),
...(typeof value.pixelDensity === 'number' ? { pixelDensity: value.pixelDensity } : {}),
...(isDeviceRotation(value.displayRotation) ? { displayRotation: value.displayRotation } : {}),
...(frames ? { frames } : {}),
...(value.overlayRefs ? { overlayRefs: value.overlayRefs } : {}),
...(value.warnings && value.warnings.length > 0 ? { warnings: value.warnings } : {}),
};
Expand All @@ -33,6 +39,7 @@ export function normalizeScreenshotCaptureResult(
logicalHeight: screenshot?.logicalHeight,
pixelDensity: screenshot?.pixelDensity,
...(screenshot?.displayRotation ? { displayRotation: screenshot.displayRotation } : {}),
...(screenshot?.frames ? { frames: screenshot.frames } : {}),
overlayRefs: screenshot?.overlayRefs,
...(screenshot?.warnings ? { warnings: screenshot.warnings } : {}),
identifiers: { session },
Expand All @@ -58,11 +65,37 @@ function readScreenshotResultData(value: unknown): ScreenshotResultData | undefi
logicalHeight: readNumberField(value, 'logicalHeight'),
pixelDensity: readNumberField(value, 'pixelDensity'),
displayRotation: isDeviceRotation(value.displayRotation) ? value.displayRotation : undefined,
frames: readScreenshotBurstFrames(value.frames),
overlayRefs: readScreenshotOverlayRefs(value.overlayRefs),
...(warnings !== undefined ? { warnings } : {}),
});
}

/**
* The burst frames a payload may report: one plan admits at most `maxCount` frames, so a longer
* array is truncated, and a frame without a path and a wall clock is not evidence of a capture.
*/
function readScreenshotBurstFrames(value: unknown): ScreenshotBurstFrame[] | undefined {
if (!Array.isArray(value)) return undefined;
const frames = value
.slice(0, SCREENSHOT_BURST_LIMITS.maxCount)
.flatMap((frame) => readScreenshotBurstFrame(frame) ?? []);
return frames.length > 0 ? frames : undefined;
}

function readScreenshotBurstFrame(frame: unknown): ScreenshotBurstFrame | undefined {
if (!isRecord(frame)) return undefined;
if (typeof frame.path !== 'string' || frame.path.length === 0) return undefined;
if (typeof frame.capturedAtMs !== 'number' || !Number.isFinite(frame.capturedAtMs)) {
return undefined;
}
return {
path: frame.path,
capturedAtMs: frame.capturedAtMs,
...(isDeviceRotation(frame.displayRotation) ? { displayRotation: frame.displayRotation } : {}),
};
}

function readNumberField(value: Record<string, unknown>, key: string): number | undefined {
const field = value[key];
return typeof field === 'number' ? field : undefined;
Expand Down
6 changes: 4 additions & 2 deletions packages/command-registry/src/flag-definitions-action.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ export const ACTION_FLAG_DEFINITIONS: readonly FlagDefinition[] = [
min: 1,
max: 200,
usageLabel: '--count <n>',
usageDescription: 'Repeat count for press, swipe and gesture tap series',
usageDescription:
'Repeat count for press, swipe and gesture tap series; screenshot: burst frame count (2-100) captured sequentially into a numbered file sequence',
projectConfig: true,
recorded: true,
},
Expand Down Expand Up @@ -85,7 +86,8 @@ export const ACTION_FLAG_DEFINITIONS: readonly FlagDefinition[] = [
min: 0,
max: 10_000,
usageLabel: '--interval-ms <ms>',
usageDescription: 'Delay between press iterations',
usageDescription:
'Delay between press iterations; screenshot: wait between burst frames (100-10000, default 100)',
projectConfig: true,
recorded: true,
},
Expand Down
99 changes: 99 additions & 0 deletions packages/command-surface/src/capture/screenshot-options.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,12 @@ import {
SCREENSHOT_COMMAND_FLAG_KEYS,
SCREENSHOT_SPECIFIC_FLAG_DEFINITIONS,
readScreenshotScriptFlag,
screenshotBurstFramePath,
screenshotFlagsFromOptions,
screenshotFlagsFromPublicOptions,
screenshotOptionsFromFlags,
validateNoRetiredScreenshotMaxSize,
validateScreenshotBurst,
validateScreenshotScale,
type ScreenshotRequestFlags,
} from '@agent-device/contracts/capture';
Expand Down Expand Up @@ -173,6 +175,8 @@ test('screenshot script flags use the shared recorded flag contract', () => {
'screenshotNoStabilize',
'screenshotNormalizeStatusBar',
'screenshotCaptureBackend',
'count',
'intervalMs',
]);
});

Expand Down Expand Up @@ -200,3 +204,98 @@ test('explicit screenshot stabilization survives SDK projection and script recor
});
assert.deepEqual(screenshotOptionsFromFlags(recorded), { stabilize: true });
});

test('screenshot burst flags ride both projections', () => {
assert.deepEqual(screenshotOptionsFromFlags({ count: 5, intervalMs: 250 }), {
count: 5,
intervalMs: 250,
});
assert.deepEqual(screenshotFlagsFromOptions({ count: 5, intervalMs: 250 }), {
count: 5,
intervalMs: 250,
});
assert.deepEqual(screenshotFlagsFromPublicOptions({ count: 3 }), { count: 3 });
});

test('screenshot burst admission bounds the count, the interval, and their pair', () => {
assert.doesNotThrow(() => validateScreenshotBurst({ count: 2 }));
assert.doesNotThrow(() => validateScreenshotBurst({ count: 5, intervalMs: 250 }));
assert.doesNotThrow(() => validateScreenshotBurst({}));
assert.throws(() => validateScreenshotBurst({ intervalMs: 250 }), {
message: /--interval-ms requires --count/,
});
assert.throws(() => validateScreenshotBurst({ count: 1 }), {
message: /--count requires an integer from 2 to /,
});
assert.throws(() => validateScreenshotBurst({ count: 101 }), { code: 'INVALID_ARGS' });
assert.throws(() => validateScreenshotBurst({ count: 2.5 }), { code: 'INVALID_ARGS' });
assert.throws(() => validateScreenshotBurst({ count: 2, intervalMs: 50 }), {
message: /--interval-ms requires an integer from 100 to /,
});
assert.throws(() => validateScreenshotBurst({ count: 2, intervalMs: 20_000 }), {
code: 'INVALID_ARGS',
});
});

test('screenshot burst admission refuses crop and overlay combinations and request-budget overruns', () => {
assert.throws(() => validateScreenshotBurst({ count: 3, cropOn: 'role=button label=Save' }), {
message: /--count cannot be combined with --crop-on or --overlay-refs/,
});
assert.throws(() => validateScreenshotBurst({ count: 3, overlayRefs: true }), {
message: /--count cannot be combined with --crop-on or --overlay-refs/,
});
assert.doesNotThrow(() => validateScreenshotBurst({ count: 3, cropOn: undefined }));
// The run budget covers the waits plus one capture allowance per frame, so the admitted burst
// stays inside the request envelope even when every frame costs its full allowance.
assert.throws(
() => validateScreenshotBurst({ count: 100, intervalMs: 200 }),
/keep the burst within 60000 ms/,
);
assert.throws(
() => validateScreenshotBurst({ count: 100, intervalMs: 600 }),
/keep the burst within 60000 ms/,
);
assert.doesNotThrow(() => validateScreenshotBurst({ count: 100 }));
assert.throws(() => validateScreenshotBurst({ count: 150 }), {
message: /--count requires an integer from 2 to 100/,
});
});

test('screenshot burst frame paths number the user sequence in place', () => {
assert.equal(screenshotBurstFramePath('/tmp/page.png', 1), '/tmp/page-0001.png');
assert.equal(screenshotBurstFramePath('/tmp/page.png', 12), '/tmp/page-0012.png');
assert.equal(screenshotBurstFramePath('page.jpeg', 3), 'page-0003.jpeg');
assert.equal(screenshotBurstFramePath('/tmp/page', 2), '/tmp/page-0002');
assert.equal(screenshotBurstFramePath('/tmp/my.photos/page', 4), '/tmp/my.photos/page-0004');
assert.equal(screenshotBurstFramePath('/tmp/.hidden', 5), '/tmp/.hidden-0005');
});

test('screenshot burst flags round-trip through recorded scripts', () => {
const parts: string[] = [];
appendScreenshotScriptFlags(parts, { count: 5, intervalMs: 250 });
assert.deepEqual(parts, ['--count', '5', '--interval-ms', '250']);
const recorded: Partial<ScreenshotRequestFlags> = {};
assert.deepEqual(readScreenshotScriptFlag({ args: parts, index: 0, flags: recorded }), {
handled: true,
nextIndex: 1,
});
assert.deepEqual(readScreenshotScriptFlag({ args: parts, index: 2, flags: recorded }), {
handled: true,
nextIndex: 3,
});
assert.deepEqual(screenshotOptionsFromFlags(recorded), { count: 5, intervalMs: 250 });
assert.throws(() => readScreenshotScriptFlag({ args: ['--count', '1'], index: 0, flags: {} }), {
code: 'INVALID_ARGS',
});
assert.throws(
() => readScreenshotScriptFlag({ args: ['--interval-ms', '50'], index: 0, flags: {} }),
{ code: 'INVALID_ARGS' },
);
});

test('the scripted pixel-density flag keeps its released error text', () => {
assert.throws(
() => readScreenshotScriptFlag({ args: ['--pixel-density', '0'], index: 0, flags: {} }),
{ code: 'INVALID_ARGS', message: 'screenshot --pixel-density requires a positive integer' },
);
});
Loading
Loading