Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
e0f4723
feat(interaction): press/click --expect-screenshot-diff out-of-tree r…
strzhao Nov 6, 2026
565160e
docs(interaction): drop the workflow-card sentence for --expect-scree…
strzhao Nov 6, 2026
b6b63bd
refactor(interaction): move the screenshot bracket lifecycle out of t…
strzhao Nov 6, 2026
70043f9
refactor(interaction): keep the screenshot frame helpers module-private
strzhao Nov 6, 2026
c6f226d
fix(interaction): register --expect-screenshot-diff in the guarantee …
strzhao Nov 6, 2026
27429a3
refactor(interaction): load the evidence lane behind a dynamic import
strzhao Nov 6, 2026
584bcaf
fix(interaction): hoist the evidence import above its outer use
strzhao Nov 6, 2026
7a61db9
style(interaction): format after the import hoist
strzhao Nov 6, 2026
ffff797
fix(interaction): keep the screenshot readback from failing a deliver…
strzhao Nov 13, 2026
434e4ad
fix(daemon): refuse --expect-screenshot-diff on the touch routes unti…
strzhao Nov 14, 2026
59a058c
fix(daemon): bind the admitted screenshot capture into the touch backend
strzhao Nov 15, 2026
8ad2aee
fix(daemon): give the screenshot readback its own frame files
strzhao Nov 15, 2026
9aee85d
style(interaction): drop the unused catch bindings
strzhao Nov 15, 2026
e62ff68
docs(evidence): run the screenshot readback live on the daemon routes
strzhao Nov 15, 2026
f49ebe6
refactor(daemon): pass the readback capture without a second branch
strzhao Nov 15, 2026
f66aa4b
feat(daemon): bind the readback capture input from the dispatch conte…
strzhao Nov 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions docs/evidence/ios-screenshot-readback-2026-11-15.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Screenshot readback live on the daemon press/click routes, 2026-11-15

Evidence for #4297 at head `9aee85d6a` (the capture binding into the touch
backend plus the self-owned frame files). The PIN-pad rig again: a keypad page
served to Safari on the booted iPhone 17 Pro simulator
(`C1C05FFF-29AD-40D8-94E5-D9CBB25EC6BB`, iOS 26.5), whose digit keys are real
accessibility buttons.

## Effective key: `press 111 300 --expect-screenshot-diff --json`

```json
{
"success": true,
"screenshotDiff": {
"changedFromBefore": true,
"changedPixelRatio": 0.003685804387672621
},
"message": "Tapped (111, 300)"
}
```

## Runtime-ref route: `click @e16 --expect-screenshot-diff --json`

```json
{
"success": true,
"ref": "e16",
"screenshotDiff": {
"changedFromBefore": true,
"changedPixelRatio": 0.013402666302355499
}
}
```

## Dead tap: the any-pixel threshold question, on hardware

A tap into the page's dead area (below the pad) on a static page:

```json
{
"success": true,
"screenshotDiff": {
"changedFromBefore": true,
"changedPixelRatio": 0.0006062365517948017
}
}
```

The ratio is an order of magnitude below the effective key's, but still above

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The ratio comparison contradicts itself. 0.0006 is about six times below the effective key's 0.0037, not an order of magnitude. Replace 'an order of magnitude below' with 'about six times below' so the sentence matches the next sentence.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/evidence/ios-screenshot-readback-2026-11-15.md, line 49:

<comment>The ratio comparison contradicts itself. 0.0006 is about six times below the effective key's 0.0037, not an order of magnitude. Replace 'an order of magnitude below' with 'about six times below' so the sentence matches the next sentence.</comment>

<file context>
@@ -0,0 +1,55 @@
+}
+```
+
+The ratio is an order of magnitude below the effective key's, but still above
+zero: the status-bar clock ticks between the bracket's frames even on a page
+whose content is otherwise static. This is the any-pixel rule's live shape and
</file context>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This cause is asserted without evidence. Either record a rerun with the status bar excluded or masked, or phrase it as a hypothesis. Also list the dead-tap command and coordinates so the sample can be reproduced.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/evidence/ios-screenshot-readback-2026-11-15.md, line 49:

<comment>This cause is asserted without evidence. Either record a rerun with the status bar excluded or masked, or phrase it as a hypothesis. Also list the dead-tap command and coordinates so the sample can be reproduced.</comment>

<file context>
@@ -0,0 +1,55 @@
+}
+```
+
+The ratio is an order of magnitude below the effective key's, but still above
+zero: the status-bar clock ticks between the bracket's frames even on a page
+whose content is otherwise static. This is the any-pixel rule's live shape and
</file context>

zero: the status-bar clock ticks between the bracket's frames even on a page

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The dead-tap pair still contains a ticking status-bar clock even though the readback requests normalizeStatusBar: true and simulator captures pin the time to 9:41. Fix the daemon-bound capture normalization before treating this as only a threshold-policy question; otherwise status-bar changes mask no-op taps.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/evidence/ios-screenshot-readback-2026-11-15.md, line 50:

<comment>The dead-tap pair still contains a ticking status-bar clock even though the readback requests `normalizeStatusBar: true` and simulator captures pin the time to 9:41. Fix the daemon-bound capture normalization before treating this as only a threshold-policy question; otherwise status-bar changes mask no-op taps.</comment>

<file context>
@@ -0,0 +1,55 @@
+```
+
+The ratio is an order of magnitude below the effective key's, but still above
+zero: the status-bar clock ticks between the bracket's frames even on a page
+whose content is otherwise static. This is the any-pixel rule's live shape and
+the concrete case for the #4509 threshold question — the readback correctly
</file context>

whose content is otherwise static. This is the any-pixel rule's live shape and
the concrete case for the #4509 threshold question — the readback correctly
reports what it saw, and the threshold policy decides what a caller should
read from a 0.0006. The effective key on the same page reads 0.0036, six
times the dead tap.
4 changes: 4 additions & 0 deletions packages/capture-kit/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,10 @@
"types": "./src/png-rgb-difference.ts",
"default": "./src/png-rgb-difference.ts"
},
"./png-changed-pixel-ratio": {
"types": "./src/png-changed-pixel-ratio.ts",
"default": "./src/png-changed-pixel-ratio.ts"
},
"./png-size": {
"types": "./src/png-size.ts",
"default": "./src/png-size.ts"
Expand Down
20 changes: 20 additions & 0 deletions packages/capture-kit/src/png-changed-pixel-ratio.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,23 @@ describe('computePngChangedPixelRatio', () => {
).toEqual({ status: 'dimension_mismatch' });
});
});

test('computePngChangedPixelRatio ignores the top band share when asked', () => {
const first = { width: 4, height: 4, data: new Uint8Array(4 * 4 * 4).fill(10) };
const second = { width: 4, height: 4, data: new Uint8Array(4 * 4 * 4).fill(10) };
// Second frame: top half (rows 0-1) jumps to 200, bottom half (rows 2-3) to 30.
for (let y = 0; y < 2; y += 1) {
for (let x = 0; x < 4; x += 1) second.data[(y * 4 + x) * 4] = 200;
}
for (let y = 2; y < 4; y += 1) {
for (let x = 0; x < 4; x += 1) second.data[(y * 4 + x) * 4] = 30;
}

const full = computePngChangedPixelRatio(first, second);
expect(full).toEqual({ status: 'compared', changedPixelRatio: 1 });

// With the top half excluded, the ignored rows drop out of the comparison:
// the ratio counts only the bottom half's changed pixels over the full frame.
const half = computePngChangedPixelRatio(first, second, { ignoreTopHeightShare: 0.5 });
expect(half).toEqual({ status: 'compared', changedPixelRatio: 0.5 });
});
33 changes: 26 additions & 7 deletions packages/capture-kit/src/png-changed-pixel-ratio.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,17 @@ export type PngChangedPixelRatioResult =
*
* Dimensions must match. Callers comparing frames of different sizes are asking a different
* question, and returning a made-up ratio would hide it.
*
* `ignoreTopHeightShare` cuts the top share of the frame out of both the changed count and the
* denominator, for bands whose pixels move without the dispatch causing it — a status-bar clock
* ticks between the bracket's frames on every platform, and the readback must answer for what the
* dispatch changed, not for the wall clock. The excluded rows are zeroed in the comparison, so the
* share is a plain fraction of the full frame either way.
Comment on lines +21 to +25

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The docstring says ignoreTopHeightShare removes the top band from the denominator too, but the code keeps the full-frame totalPixels denominator. Reword the doc to match the code, which the test and the last docstring sentence already assume. The inline comment above the ignoredRows block ("contribute nothing to either side of the ratio") has the same error and needs the same fix.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/capture-kit/src/png-changed-pixel-ratio.ts, line 21:

<comment>The docstring says `ignoreTopHeightShare` removes the top band from the denominator too, but the code keeps the full-frame `totalPixels` denominator. Reword the doc to match the code, which the test and the last docstring sentence already assume. The inline comment above the `ignoredRows` block ("contribute nothing to either side of the ratio") has the same error and needs the same fix.</comment>

<file context>
@@ -17,10 +17,17 @@ export type PngChangedPixelRatioResult =
  * Dimensions must match. Callers comparing frames of different sizes are asking a different
  * question, and returning a made-up ratio would hide it.
+ *
+ * `ignoreTopHeightShare` cuts the top share of the frame out of both the changed count and the
+ * denominator, for bands whose pixels move without the dispatch causing it — a status-bar clock
+ * ticks between the bracket's frames on every platform, and the readback must answer for what the
</file context>
Suggested change
* `ignoreTopHeightShare` cuts the top share of the frame out of both the changed count and the
* denominator, for bands whose pixels move without the dispatch causing it — a status-bar clock
* ticks between the bracket's frames on every platform, and the readback must answer for what the
* dispatch changed, not for the wall clock. The excluded rows are zeroed in the comparison, so the
* share is a plain fraction of the full frame either way.
* `ignoreTopHeightShare` leaves the top share of the frame out of the changed count. Those rows
* are skipped rather than removed from the denominator, so the ratio stays a fraction of the full
* frame. The option is for bands whose pixels move without the dispatch causing it: a status-bar
* clock ticks between the bracket's frames on every platform, and the readback must answer for what
* the dispatch changed, not for the wall clock.

*/
export function computePngChangedPixelRatio(
first: PngRgbImage,
second: PngRgbImage,
options: Readonly<{ ignoreTopHeightShare?: number }> = {},
): PngChangedPixelRatioResult {
if (first.width !== second.width || first.height !== second.height) {
return { status: 'dimension_mismatch' };
Expand All @@ -32,14 +39,26 @@ export function computePngChangedPixelRatio(
return { status: 'dimension_mismatch' };
}

// Rows at or above the cut contribute nothing to either side of the ratio.
const ignoredRows =
options.ignoreTopHeightShare && options.ignoreTopHeightShare > 0
? Math.min(first.height, Math.floor(first.height * options.ignoreTopHeightShare))
: 0;

let changedPixels = 0;
for (let offset = 0; offset + 3 < first.data.length; offset += 4) {
if (
first.data[offset] !== second.data[offset] ||
first.data[offset + 1] !== second.data[offset + 1] ||
first.data[offset + 2] !== second.data[offset + 2]
) {
changedPixels += 1;
for (let y = 0; y < first.height; y += 1) {
const rowChangedBand = y < ignoredRows;
for (let x = 0; x < first.width; x += 1) {
const offset = (y * first.width + x) * 4;
if (offset + 3 >= first.data.length) break;
if (rowChangedBand) continue;
if (
first.data[offset] !== second.data[offset] ||
first.data[offset + 1] !== second.data[offset + 1] ||
first.data[offset + 2] !== second.data[offset + 2]
) {
changedPixels += 1;
}
}
}

Expand Down
10 changes: 10 additions & 0 deletions packages/command-registry/src/flag-definitions-action.ts
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,16 @@ export const ACTION_FLAG_DEFINITIONS: readonly FlagDefinition[] = [
projectConfig: true,
recorded: false,
},
{
key: 'expectScreenshotDiff',
names: ['--expect-screenshot-diff'],
type: 'boolean',
usageLabel: '--expect-screenshot-diff',
usageDescription:
'Press/click: compare lightweight screenshots taken right before the dispatch and after it settles briefly, and report screenshotDiff in the response; use it when the target lives outside the accessibility tree (a PIN keypad) and --verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect',
projectConfig: true,
recorded: false,
},
{
key: 'settle',
names: ['--settle'],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,7 @@ export function createInteractionDevice(
Pick<
AgentDeviceBackend,
| 'captureSnapshot'
| 'captureScreenshot'
| 'resolveGestureViewport'
| 'tap'
| 'tapTarget'
Expand All @@ -366,6 +367,9 @@ export function createInteractionDevice(
platform: overrides.platform ?? 'ios',
captureSnapshot: async (...args) =>
overrides.captureSnapshot ? await overrides.captureSnapshot(...args) : { snapshot },
captureScreenshot: overrides.captureScreenshot
? async (...args) => (await overrides.captureScreenshot?.(...args)) ?? {}
: undefined,
resolveGestureViewport: overrides.resolveGestureViewport
? async (...args) => await overrides.resolveGestureViewport?.(...args)
: undefined,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import { isFillableType } from '@agent-device/contracts/snapshot';
import { attachResolvedInteractionTarget } from '@agent-device/host-kit/interaction-outcome';
import { toBackendContext } from '../../runtime-common.ts';
import { toBackendResult, type RuntimeCommand } from '../../runtime-types.ts';
import type { ScreenshotPairOutcome } from './screenshot-diff-evidence.ts';
import {
applyPostActionObservation,
planPostActionObservation,
Expand Down Expand Up @@ -41,6 +42,14 @@ export type PressCommandOptions = CommandContext &
RepeatedInput & {
target: InteractionTarget;
button?: ClickButton;
/**
* Opt-in: bracket the dispatch with two lightweight screenshots and attach
* `screenshotDiff` to the result, so a tap on a surface the accessibility
* tree cannot describe (a PIN keypad) still answers "did the screen react
* at all". Best-effort — a capture failure drops the evidence; a matching
* pair adds a warning instead of failing the action.
*/
expectScreenshotDiff?: boolean;
/** ADR 0012 step 4: replay-only post-resolution guard; see resolution.ts. */
expectedResolvedTarget?: ExpectedResolvedTarget;
/** #1654: a mutating `find`'s already-resolved node; see resolution.ts. */
Expand Down Expand Up @@ -140,9 +149,12 @@ async function tapCommand(
action: 'click' | 'press',
): Promise<PressCommandResult> {
const observation = planPostActionObservation(options);
const nativeRefTap = observation.needsPreActionBaseline
? null
: await maybeTapRefTarget(runtime, options, action);
// --expect-screenshot-diff brackets the dispatch below, which the native ref
// fast path never reaches — route through the runtime path like verify/settle.
const nativeRefTap =
observation.needsPreActionBaseline || options.expectScreenshotDiff === true
? null
: await maybeTapRefTarget(runtime, options, action);
if (nativeRefTap) return nativeRefTap;

// One request object for the resolution and for the looks admission may spend, so a second look is
Expand All @@ -162,17 +174,28 @@ async function tapCommand(
const atRest = await admitResolvedTarget(runtime, options, resolutionRequest, resolved);
const point = requireResolvedPoint(atRest);
const pointObservation = pointObservationOf(atRest);
const dispatchTap = runtime.backend.tap;
// Dynamic import: the readback evidence lane stays off the command-runtime entry closure.
const { attachScreenshotPairOutcome, dispatchWithScreenshotPair } =
await import('./screenshot-diff-evidence.ts');
let backendResult;
let screenshotOutcome: ScreenshotPairOutcome | undefined;
try {
backendResult = await runtime.backend.tap(toBackendContext(runtime, options), point, {
button: options.button,
count: options.count,
intervalMs: options.intervalMs,
holdMs: options.holdMs,
jitterPx: options.jitterPx,
doubleTap: options.doubleTap,
...(pointObservation ? { pointObservation } : {}),
});
({ result: backendResult, outcome: screenshotOutcome } = await dispatchWithScreenshotPair(
runtime,
options,
options.expectScreenshotDiff === true,
Comment thread
strzhao marked this conversation as resolved.
Resolved
async () =>
await dispatchTap(toBackendContext(runtime, options), point, {
button: options.button,
count: options.count,
intervalMs: options.intervalMs,
holdMs: options.holdMs,
jitterPx: options.jitterPx,
doubleTap: options.doubleTap,
...(pointObservation ? { pointObservation } : {}),
}),
));
} catch (error) {
// Resolution is complete before the backend call. Preserve it out of
// band so a daemon-level failure corroboration can still record the same
Expand All @@ -185,10 +208,10 @@ async function tapCommand(
runtime,
options,
atRest,
{
...atRest,
...(formattedBackendResult ? { backendResult: formattedBackendResult } : {}),
},
attachScreenshotPairOutcome(
{ ...atRest, ...(formattedBackendResult ? { backendResult: formattedBackendResult } : {}) },
screenshotOutcome,
),
observation,
);
}
Expand Down
Loading
Loading