Skip to content

feat(interaction): screenshot readback (--expect-screenshot-diff) on press and click - #4297

Open
strzhao wants to merge 16 commits into
callstack:mainfrom
strzhao:forge/press-expect
Open

strzhao wants to merge 16 commits into
callstack:mainfrom
strzhao:forge/press-expect

Conversation

@strzhao

@strzhao strzhao commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

2026-11-15: the capture now binds on the daemon touch routes — this PR delivers the readback (--expect-screenshot-diff) on press/click; the baseline half shipped in #4338. Live runs below and in docs/evidence/ios-screenshot-readback-2026-11-15.md at head e62ff686e.

Summary

A coordinate press on a PIN pad reported success while the digit never entered and the remaining-attempts counter ticked down — the no-op guard's baseline had fallen back to an older tree, exactly the blind spot its own comment admits. Two changes: the coordinate lane now takes a guaranteed fresh pre-tree when the session baseline is stale, and press --expect-screenshot-diff adds an out-of-tree readback that compares a lightweight before/after screenshot digest, so "tapped, nothing happened" is a reported effect instead of a discovery.

22 files changed (baseline freshness, expect readback, output fields, tests).

Validation

  • Head: 69032ca on a68693c (includes review-fix commit).
  • pnpm run typecheck, pnpm run lint, post-action-observation and interaction tests, integration-progress --check, fallow, freerange: pass locally.
  • CI runs the full gates on push.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

View guided diff

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 issues found across 34 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/daemon/src/core/interaction/internal/interaction-touch-direct-ios-eligibility.ts">

<violation number="1" location="packages/daemon/src/core/interaction/internal/interaction-touch-direct-ios-eligibility.ts:38">
P3: This new early return has no direct test. Add a case in the daemon eligibility tests asserting `readDirectIosSelectorTapTarget` returns `null` when `flags.expectScreenshotDiff` is `true` on an otherwise eligible `click` selector target, so the fused route cannot regress silently.</violation>
</file>

<file name="packages/contracts/src/cli-flags.ts">

<violation number="1" location="packages/contracts/src/cli-flags.ts:126">
P3: This doc comment repeats the option's prose, which the header of this file says must live only on the `FlagDefinition` (flag-definitions-action.ts already has it). Remove the block and keep the bare field; the '#1047 sibling' ticket pointer goes with it.</violation>
</file>

<file name="test/integration/interaction-contract/runtime-selector.contract.test.ts">

<violation number="1" location="test/integration/interaction-contract/runtime-selector.contract.test.ts:378">
P3: This scenario never checks that the tap was dispatched: `tap` discards its arguments and no `taps` array is asserted. Record the dispatched point and assert one tap, so the test proves the pixel pair bracketed a real dispatch.</violation>
</file>

<file name="packages/command-surface/src/interaction/metadata.ts">

<violation number="1" location="packages/command-surface/src/interaction/metadata.ts:90">
P3: The description promises that `screenshotDiff` is reported, but the runtime silently omits it when a capture fails or the backend has no screenshot support. Add that caveat so callers, including models reading this schema, do not treat a missing `screenshotDiff` as a comparison result.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread website/docs/docs/commands.md Resolved
maxPasses?: number;
doubleTap?: boolean;
verify?: boolean;
/**

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This doc comment repeats the option's prose, which the header of this file says must live only on the FlagDefinition (flag-definitions-action.ts already has it). Remove the block and keep the bare field; the '#1047 sibling' ticket pointer goes with it.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/contracts/src/cli-flags.ts, line 126:

<comment>This doc comment repeats the option's prose, which the header of this file says must live only on the `FlagDefinition` (flag-definitions-action.ts already has it). Remove the block and keep the bare field; the '#1047 sibling' ticket pointer goes with it.</comment>

<file context>
@@ -123,6 +123,13 @@ export type CliFlags = CloudProviderProfileFields &
     maxPasses?: number;
     doubleTap?: boolean;
     verify?: boolean;
+    /**
+     * Press/click opt-in (#1047 sibling): bracket the dispatch with two
+     * lightweight screenshots and compare them, so a tap on a surface the
</file context>

test(scenario('screenshotDiffEvidence'), async () => {
const device = createContractDevice(continueButtonSnapshot(), {
...screenshotFramePlanBackend([{ color: [0, 0, 0] }, { color: [255, 255, 255] }]),
tap: async () => ({ ok: true }),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This scenario never checks that the tap was dispatched: tap discards its arguments and no taps array is asserted. Record the dispatched point and assert one tap, so the test proves the pixel pair bracketed a real dispatch.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At test/integration/interaction-contract/runtime-selector.contract.test.ts, line 378:

<comment>This scenario never checks that the tap was dispatched: `tap` discards its arguments and no `taps` array is asserted. Record the dispatched point and assert one tap, so the test proves the pixel pair bracketed a real dispatch.</comment>

<file context>
@@ -367,6 +372,24 @@ test(scenario('settleObservation'), async () => {
+test(scenario('screenshotDiffEvidence'), async () => {
+  const device = createContractDevice(continueButtonSnapshot(), {
+    ...screenshotFramePlanBackend([{ color: [0, 0, 0] }, { color: [255, 255, 255] }]),
+    tap: async () => ({ ok: true }),
+  });
+
</file context>


const expectScreenshotDiffField = () =>
booleanField(
'Compare lightweight screenshots taken right before the dispatch and after it settles briefly, and report the comparison as screenshotDiff in the response. Use it when the target lives outside the accessibility tree, such as a PIN keypad, where verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect.',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The description promises that screenshotDiff is reported, but the runtime silently omits it when a capture fails or the backend has no screenshot support. Add that caveat so callers, including models reading this schema, do not treat a missing screenshotDiff as a comparison result.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/command-surface/src/interaction/metadata.ts, line 90:

<comment>The description promises that `screenshotDiff` is reported, but the runtime silently omits it when a capture fails or the backend has no screenshot support. Add that caveat so callers, including models reading this schema, do not treat a missing `screenshotDiff` as a comparison result.</comment>

<file context>
@@ -85,16 +85,23 @@ const interactionCommandDescriptions = {
 
+const expectScreenshotDiffField = () =>
+  booleanField(
+    'Compare lightweight screenshots taken right before the dispatch and after it settles briefly, and report the comparison as screenshotDiff in the response. Use it when the target lives outside the accessibility tree, such as a PIN keypad, where verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect.',
+  );
+
</file context>
Suggested change
'Compare lightweight screenshots taken right before the dispatch and after it settles briefly, and report the comparison as screenshotDiff in the response. Use it when the target lives outside the accessibility tree, such as a PIN keypad, where verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect.',
'Compare lightweight screenshots taken right before the dispatch and after it settles briefly, and report the comparison as screenshotDiff in the response; the field is omitted when a capture fails or the backend has no screenshots. Use it when the target lives outside the accessibility tree, such as a PIN keypad, where verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect.',

@strzhao

strzhao commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Coverage failed on an eager-closure ratchet: the new evidence module joined the command-runtime entry closure as a static edge. It now loads behind a function-scoped dynamic import (type import kept separate, so types cost nothing), with a follow-up fixing an intermediate scope mistake — verified green locally (typecheck/lint, 135 tests).

🤖 Generated with Claude Code

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

@thymikee

thymikee commented Nov 6, 2026

Copy link
Copy Markdown
Member

At 6606bb9, --expect-screenshot-diff does nothing on the shipped routes, and a delivered tap can still be reported as failed. All 16 of 16 checks pass, but they cannot catch this. Before merge, wire the flag into the daemon touch route, make the evidence step never throw, and post a live run.

Right now press 120 640 --expect-screenshot-diff returns no screenshotDiff and no warning. createInteractionBackend never sets captureScreenshot, so capturePairScreenshotFrame returns early. The same happens for click, MCP press and Node client press through the daemon. Under the "absent claims nothing" contract, a pair that was never taken looks like an unsupported platform, so a caller on a PIN pad gets no warning and keeps tapping. The rule: every route that admits expectScreenshotDiff must hold a bound captureScreenshot for the dispatch device. The touch route's request binding must admit it when the flag is set, the way screenshot-runtime.ts does. Please add a daemon-route test that sends press --expect-screenshot-diff through dispatchTargetedTouchViaRuntime and asserts data.screenshotDiff. Every current test injects captureScreenshot into a fake, so they prove the helper and not the route.

compareScreenshotPair runs after dispatch() resolves and only uses try/finally. A truncated after-frame makes decodePng throw COMMAND_FAILED. That error reaches the dispatch catch in interactions.ts and is rethrown, so a tap that landed is reported as failed. The caller retries and enters the digit twice, which is the case this flag exists to prevent. The doc comment ("any decode or capture failure drops the evidence") says the opposite. The rule: nothing after a resolved dispatch may throw out of the evidence lane. Catch around the sleep, capture, import, read, decode and compare, return {}, and keep cleanup best-effort. A test with a second capture that writes non-PNG bytes should leave the tap recorded and the result without screenshotDiff. It fails on this head.

This change alters live dispatch: it bypasses the native ref fast path and the direct-iOS fused route, and brackets the tap with two captures and a 150 ms sleep. The PR body lists typecheck, lint, unit tests and progress checks at an older head (69032ca), with no device run. After the two fixes above, please run through the CLI on one iOS simulator and one Android emulator, against a screen with a static region or keypad. The run needs to show: press <x> <y> --expect-screenshot-diff --json on an effective key with data.screenshotDiff.changedFromBefore true and a changedPixelRatio; the same flag on a dead region with changedFromBefore false and the warning text; press @ref --expect-screenshot-diff --json, which proves the ref bypass still dispatches; and press <x> <y> --verify --json with a failing baseline capture or a tree-blind surface, where evidence has a digest and no changedFromBefore key. Please paste the JSON excerpts and the head SHA into the PR body. I did not run tests or a device myself, so the 150 ms settle and status-bar handling on real backends are unverified.

Not blocking, take or leave: the flag could be a screenshotDiff cell on the existing postActionObservation trait instead of hand-added flag lists, readers and a second bypass check, which would also remove the new 165-line repo guard in favor of rows in the existing descriptor gate; the two full-resolution PNG decodes run synchronously on the event loop when png-worker-client already has async decode; changedFromBefore is ratio > 0, so one animated pixel reads as a landed tap, which needs a doc note or a small minimum ratio; the two near-identical frame-plan fakes should become one, with taps recorded in every "dispatch still succeeds" test; commit b37da26 (the omit changedFromBefore fix) can merge as its own small PR, since the diff is over 1,000 lines; and the title and body describe a "guaranteed fresh pre-tree" baseline that no hunk implements, the user docs and descriptions should say an absent field means no comparison was made, and the dynamic import at interactions.ts:190 could be gated on the flag.

Could the readback be smaller? After splitting out b37da26, put screenshotPair in PostActionObservationPlan with one bypassesFastPaths bit read by maybeTapRefTarget and the direct-iOS eligibility. Let the daemon touch route's binding admit captureScreenshot when the plan asks for it. Then decode and compare through one PNG worker job that never throws into the dispatch result. That would remove most of the new surface and make the feature run. What would have to change first is that the touch-route binding must admit a captureScreenshot operation next to the touch executor. I did not confirm how ADR 0019 operation admission allows a second operation on a touch request, so please check it before starting.

The open review threads still apply, apart from one. P2: absent screenshotDiff in docs, decode throws after tap, capture-failure test stub, silent continue in guard, ratio > 0 semantics and sync decode. Six lower-priority threads still apply too. This one does not apply and can be resolved: flag in buildFlags follows the existing verify/settle pattern, and the readers consult the flag only on press and click.

@thymikee

thymikee commented Nov 7, 2026

Copy link
Copy Markdown
Member

This now conflicts with main at 6606bb9. Please rebase onto main. The two defects and the live run from the earlier review still apply.

@thymikee

thymikee commented Nov 7, 2026

Copy link
Copy Markdown
Member

Tracked in #4318, group C (new public surface). We're reviewing the six feature PRs together for coherence and for performance impact on paths that don't use them, and we'll post a direction there. Please hold changes here until then.

@thymikee

thymikee commented Nov 8, 2026

Copy link
Copy Markdown
Member

Thanks @strzhao. We reviewed this with the rest of group C in #4318. It mixes two changes that should land separately.

  1. Missing-baseline honesty: this part is a small, valid fix, and we'd like it now. When the before/after comparison was impossible, changedFromBefore is omitted instead of reported. Please split it into its own PR, rebased onto main. Drop the "guaranteed fresh baseline" claim, because the code doesn't implement it (post-action-surface.ts:55).
  2. Screenshot-diff evidence: this needs a design issue before code.
    • Ordinary taps already wait for the helper import (interactions.ts:189).
    • When enabled, it costs two screenshots, a 150 ms wait, two synchronous PNG decodes and a pixel scan on the daemon thread, which can stall unrelated requests.
    • It also isn't reachable today, because the daemon's interaction backend has no captureScreenshot.

What the visual feature needs:

  • One owner for the comparison: the shared post-action observation plan.
  • Screenshot capability declared through runtime admission, with unsupported targets refused before dispatch.
  • Evidence failures after dispatch reported as "unavailable". They must never turn a delivered tap into a failed action, and "pixels changed" must never mean "the action succeeded".
  • Budget: zero extra captures or imports when off. When on, at most two screenshots and one bounded wait, with the comparison off the daemon thread and inside the request deadline.
  • Coverage through real CLI, daemon and provider routes, including corrupt after-images, cancellation and unrelated animation.

Could you open the split PR for part 1 and an issue for part 2?

strzhao and others added 8 commits November 13, 2026 13:08
…eadback

A coordinate press on a surface the accessibility tree cannot describe —
a PIN keypad whose keys never reach the tree — dispatches fine, leaves
--verify's node digest flat whether or not the digit was entered, and
reports success. Repeated attempts walk toward a device lockout with no
signal in any response.

The flag brackets the dispatch with two lightweight screenshots and
compares them on pixels instead of nodes: screenshotDiff carries
changedFromBefore and the changed-pixel ratio. A matching pair adds a
response warning, because the dispatch may have had no visible effect;
the action itself still succeeds. Best-effort like --verify evidence: a
failed capture or a screenshot-less backend drops the field and claims
nothing about the screen. The flag routes press/click through the
runtime path (the native ref fast path and the fused direct-iOS route
never reach the bracket), and the decode stack loads behind
function-scoped dynamic imports so the flag-less fast path pays nothing.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…nshot-diff

The compact workflow card has a byte ratchet (cli-help.test.ts) with no
headroom for a new sentence, and the flag's owning help already carries
the full guidance: the flag definition's usage text renders in
"Command flags:" for press/click, and the metadata field description
documents the PIN-keypad case for the Node SDK and MCP schemas.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…apCommand

The before-frame capture, its dispatch-failure cleanup, and the pair
outcome composition each added a branch to a function the complexity
gate already watches. dispatchWithScreenshotPair owns the bracket
lifecycle and attachScreenshotPairOutcome owns the result composition,
so tapCommand reads as dispatch-plus-observation again; the reader
gate's positional table flattens the same way.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The bracket lifecycle moved behind dispatchWithScreenshotPair, so the
two frame helpers no longer need exports; the dead-export gate reads
them as unused module surface.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…matrix, settle the after-frame, and document the out-of-tree readback

Review fixes for the press/click --expect-screenshot-diff PR:

- The guarantee matrix (ADR 0011) gains the screenshotDiffEvidence row:
  runtime cells on runtime-selector/runtime-ref/coordinate scoped to
  press/click, a delegated cell on native-ref, and inapplicable cells on
  the paths the flag cannot reach. The three waivers that named only
  --verify/--settle as the opt-in post-action evidence routes now name
  this flag too. Each enforced cell carries a contract scenario, and the
  shared harness replays a solid-color frame plan.
- The after-frame waits 150ms past the dispatch reply before it is
  captured: the reaction the flag asks about can render frames after the
  transport round trip returns, and an immediate frame would read an
  effective tap as unchanged. The flag help text, field descriptions,
  and schema prose state the timing instead of leaving it blank.
- commands.md names the flag where it teaches that a successful press is
  not proof of effect: a PIN keypad target is the case --verify cannot
  answer, since it compares empty trees.
- Gate drift the PR introduced: the new capture-kit export joins the
  layering manifest baseline, and the flag descriptor gate moves to
  test/repo-guards/ (a repo-level guard on command descriptors, not a
  module's unit test — R84 test-colocation).

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
@strzhao

strzhao commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main (through 0.21.37). The only conflict was the interaction metadata field list — the plan now spreads main's interactionTargetFields() and appends expectScreenshotDiff. Typecheck, lint, and the interaction/command-surface/daemon execution suites pass at the pushed head. The reshape direction from the 11-08 group C review (short issue first, then the missing-baseline piece) is still what gates this PR; say the word and I'll open that issue.

@thymikee

Copy link
Copy Markdown
Member

Two defects from the earlier review (6606bb9, #4297 (comment)) are still open at 7a61db9, and the PR has no device run yet. CI is green: 19 of 19 checks pass. It cannot catch the two defects below, because every readback test injects captureScreenshot into a fake backend, no test drives the daemon touch route with the flag, and no test corrupts the after-frame. There are no conflicts.

createInteractionBackend builds the backend for every daemon touch route without captureScreenshot (interaction-runtime.ts#L194). So press <x> <y> --expect-screenshot-diff, click, MCP press and Node press through the daemon reach if (!runtime.backend.captureScreenshot) return undefined (screenshot-diff-evidence.ts#L113). The response then carries neither screenshotDiff nor the warning. Only the screenshot route binds captureScreenshot (screenshot-runtime-binding.ts#L89-L108). The flag is a silent no-op on every shipped surface, and a PIN-pad caller gets no warning. The rule: every route that admits expectScreenshotDiff must hold a bound captureScreenshot for the dispatch device, admitted through runtime operation admission. Otherwise the request must be refused before dispatch, as the 11-08 direction asks. Please add a daemon-route test through dispatchTargetedTouchViaRuntime that asserts data.screenshotDiff.

compareScreenshotPair runs after dispatch() has resolved, with only try/finally (screenshot-diff-evidence.ts#L148). A truncated after-frame makes decodePng throw COMMAND_FAILED. A readFile error, an import failure or a throwing cleanup does the same. The error leaves dispatchWithScreenshotPair, and the dispatch catch in tapCommand rethrows it. A delivered tap is then reported as failed, so the caller retries and enters the PIN digit twice. The settle sleep at line 149 also takes no signal, so a request canceled after dispatch still waits and captures. The rule: nothing after a resolved dispatch may throw out of the evidence lane, and the lane must honor the request signal and deadline. That means catching around the sleep, capture, import, read, decode and compare and returning no evidence, and passing options.signal to the sleep. The after-frame corruption test (r4497036545) must fail on this head.

Live dispatch also changed. Under the flag, the native ref fast path and the direct-iOS fused route are bypassed, and two captures plus a 150 ms wait bracket the tap (interactions.ts#L184). Nothing proves the readback works on a real backend, and the first defect shows that it currently does not. After both fixes, please run through the CLI on one iOS simulator and one Android emulator. Show press <x> <y> --expect-screenshot-diff --json on an effective key, with changedFromBefore true and changedPixelRatio. Show the same flag on a dead region, with false plus the warning text. Show click @ref --expect-screenshot-diff --json, which dispatches through runtime-ref. Paste the JSON excerpts and the head SHA in the PR body. I ran no tests or devices; the CI status comes from the check results.

Not blocking, take or leave: the before-frame capture now sits between the at-rest confirmation in admitResolvedTarget and the tap (after the #4461 rebase), so a slow screenshot round trip can run after the point was declared at rest, outside the ADR 0039 looks budget. Taking the before-frame ahead of admission, or counting it inside the admission request, would fix that. This is a reasoning concern, not a reproduced misfire. Also, #4338 merged the baseline half, so the title and body still describe a pre-tree guarantee that this PR no longer carries. Retitling to the readback and refreshing the summary, head and file count would help.

The review threads on the missing inconclusive-when-absent note in the docs (r4497036453), the missing catch after dispatch (r4497036487), the tap stub that records nothing (r4497036494), the repo guard's continue (r4497036501), the any-pixel changed rule (r4497036508) and the synchronous PNG decode (r4497036514) still apply, as do five lower-priority threads. The thread on the command-flags.ts pattern does not apply, because it follows the existing verify/settle buildFlags pattern and only the press/click readers consume the flag, so you can resolve it (r4497036570).

Before merge, please open the design issue for the screenshot readback, since the baseline half shipped in #4338, then fix the two defects and post the live iOS and Android run.

…ed tap

Nothing after a resolved dispatch throws out of the evidence lane anymore: a
decode, read, import or cleanup failure yields no evidence, because a throwing
readback reported a delivered tap as failed and a PIN caller would re-enter
the digit. The after-frame settle honors the request signal. Both frames
decode in the codec worker, the guard audits a missing descriptor instead of
skipping it, the docs say the readback is best-effort, and the delivered-tap
tests record the tap.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@strzhao

strzhao commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Progress at ffff797, with the decision carried to #4509.

Defect 2 fixed. Nothing after a resolved dispatch throws out of the evidence lane: the comparison runs inside a catch that yields no evidence, cleanups are caught, the after-frame settle honors the request signal, and both frames decode in the codec worker (decodePngAsync), not on the event loop. The corruption test the review asked for (r4497036545) pins the delivered-tap case: one tap, no screenshotDiff, no warning. The delivered-tap tests now record the tap, the repo guard refuses a missing descriptor instead of skipping it, the docs say the readback is best-effort, and the any-one-pixel rule is a question in #4509 (r4497036508).

Defect 1 is the design decision. #4509 carries it: admit a capture operation on touch routes, or refuse pre-dispatch where no binding exists. Per the 11-08 direction the design issue comes first; the binding follows its outcome, and the live iOS/Android runs follow the binding (before it lands, the flag cannot report on a shipped surface — which is defect 1 itself).

Taken as well: the docs' inconclusive-when-absent sentence, and r4497036570 resolved as not applying. The remaining threads (r4497036453, r4497036487, r4497036494, r4497036501, r4497036514) are fixed at this head; r4497036508 routes to #4509. Title and body retitle to the readback once defect 1 lands and the flag reports for real.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 4 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts">

<violation number="1" location="packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts:168">
P2: ` sleep` resolves normally on abort, so this continues into the after-frame capture instead of stopping readback. Check `options.signal ?? runtime.signal` after sleeping and return no evidence when it is aborted.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

): Promise<ScreenshotPairOutcome> {
try {
// A request canceled after the dispatch stops waiting for the after-frame.
await sleep(runtime, AFTER_FRAME_SETTLE_MS, options.signal);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: sleep resolves normally on abort, so this continues into the after-frame capture instead of stopping readback. Check options.signal ?? runtime.signal after sleeping and return no evidence when it is aborted.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts, line 168:

<comment>` sleep` resolves normally on abort, so this continues into the after-frame capture instead of stopping readback. Check `options.signal ?? runtime.signal` after sleeping and return no evidence when it is aborted.</comment>

<file context>
@@ -145,23 +147,43 @@ async function compareScreenshotPair(
+): Promise<ScreenshotPairOutcome> {
+  try {
+    // A request canceled after the dispatch stops waiting for the after-frame.
+    await sleep(runtime, AFTER_FRAME_SETTLE_MS, options.signal);
     const after = await capturePairScreenshotFrame(runtime, options, 'after');
     if (!after) return {};
</file context>
Suggested change
await sleep(runtime, AFTER_FRAME_SETTLE_MS, options.signal);
const signal = options.signal ?? runtime.signal;
await sleep(runtime, AFTER_FRAME_SETTLE_MS, signal);
if (signal?.aborted) return {};

@thymikee

Copy link
Copy Markdown
Member

I reviewed ffff797 against my earlier review at #4297 (comment). The capture-failure, corrupted-frame and docs concerns are fixed. The silent no-op on daemon routes is not, and there is still no live device run.

The daemon touch backend is built without captureScreenshot in interaction-runtime.ts:194, while interaction-touch-press.ts:217 admits expectScreenshotDiff. So press <x> <y> --expect-screenshot-diff, click @ref --expect-screenshot-diff, MCP press and Node press hit if (!runtime.backend.captureScreenshot) return undefined in screenshot-diff-evidence.ts:115. They return neither screenshotDiff nor the warning. A caller who opted in gets no evidence, and the new best-effort docs sentence now describes this no-op as intended. The rule to satisfy: every route that admits expectScreenshotDiff holds a captureScreenshot bound for the dispatch device through runtime operation admission, or refuses the flag before dispatch with a typed reason. Please enforce it at the one place the daemon admits the flag, not with per-route guards. Please also add a daemon-route test through dispatchTargetedTouchViaRuntime that asserts data.screenshotDiff. If #4509 decides otherwise, please keep the flag off the shipped surfaces until it does.

Under the flag, the native ref fast path and the direct-iOS fused route are bypassed (interactions.ts:154). Two captures and a 150 ms wait now bracket the tap. This changes live dispatch, and no device run exists at this head. After the fix above, please run through the CLI on one iOS simulator and one Android emulator: press <x> <y> --expect-screenshot-diff --json on an effective key (expect changedFromBefore true with changedPixelRatio), the same on a dead region (expect false plus the warning text), and click @ref --expect-screenshot-diff --json on the runtime-ref route. Please paste the JSON excerpts and the head SHA in the PR body.

Of the other open threads, the abort-handling thread still applies, and so do three lower-priority ones. The dynamic-import thread does not apply, because the import is evaluated once and then served from the module cache: #4297 (comment). The threads on the docs sentence, the capture-failure catch, the capture-failure test, the metadata continue, the codec worker decode and the corrupted-frame test are fixed at this head and can be resolved. The changedFromBefore threshold thread is reasonably deferred to #4509. The buildFlags thread is benign.

CI is green with 19 of 19 checks passing, but no CI route exercises the daemon touch route with the flag, so green does not cover the first issue. Before merge, the daemon binding or refusal needs to land with its route test, and then the live iOS and Android runs need to be posted.

…l a capture binds

The touch routes admit the flag but bind no screenshot capture, so the flag
was a silent no-op on every shipped surface. The refusal lives at the one
place the daemon admits the flag: typed reason screenshot-readback-unavailable,
dispatched no, and the callstack#4509 decision owns when the capture binds instead.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@strzhao

strzhao commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Defect 1 is enforced at the one place the daemon admits the flag: dispatchTargetedTouchViaRuntime refuses --expect-screenshot-diff with INVALID_ARGS / reason screenshot-readback-unavailable / dispatched: 'no' before any device work, because the touch routes bind no screenshot capture — and per your rule, the flag stays off the shipped surfaces until #4509 decides otherwise. The route test covers the plan and single-ref forms and asserts no dispatch. The abort-handling thread and the three lower-priority ones are answered on their threads; the dynamic-import thread is resolved as not applying.

The CLI live runs you specified are meaningful once the flag reports on a shipped surface — under the refusal that is the typed refusal JSON on the daemon route, which I can attach next, and the runtime-lane runs (effective key, dead region, runtime-ref click) follow the #4509 binding decision. Title/body retitle to the readback with the next push.

@thymikee

Copy link
Copy Markdown
Member

Thanks for the update. At 434e4ad the earlier findings on the delivered-tap tests, the after-frame error handling, the missing-descriptor guard and the PNG decode are fixed. One problem remains, and it is a new one. All 19 checks pass, but no job drives the daemon press/click route with the flag and asserts screenshotDiff, so green covers neither the problem below nor the fix. I ran no tests and no devices.

Every daemon press and click with expectScreenshotDiff now fails with INVALID_ARGS / screenshot-readback-unavailable (interaction-touch-press.ts#L45-L56, called at #L65). The flag is still advertised as a working readback in CLI help (flag-definitions-action.ts#L209-L218), the click/press allowedFlags, MCP metadata, the Node ClickOptions/PressOptions docs, the screenshotDiff response schema and website/docs/docs/commands.md. A caller who reads the help opts in and gets a hard error on every press, which is the PIN-pad case this PR targets. The rule to satisfy: a flag that any surface advertises must have a daemon route that honors it. Either bind captureScreenshot into the touch backend in createInteractionBackend through runtime operation admission, then delete the refusal, add a daemon-route test that asserts data.screenshotDiff, and post the live runs. Or remove expectScreenshotDiff from every surface that ACTION_FLAG_DEFINITIONS, the click/press allowedFlags, the metadata fields, the option types, the response schema and commands.md feed, until #4509 decides. If you bind the capture, the live runs are on one iOS simulator and one Android emulator: press <x> <y> --expect-screenshot-diff --json on an effective key must show changedFromBefore: true with changedPixelRatio; the same on a dead region must show false plus the warning; and click @ref --expect-screenshot-diff --json must work on a runtime ref.

Not blocking, and you can take or leave these: the guarantee matrix still marks screenshotDiffEvidence as enforced on the runtime-selector, runtime-ref and coordinate rows (interaction-guarantees.ts#L290), but the coverage scenarios inject captureScreenshot directly and bypass the daemon route that refuses the flag, so mark it a waived gap tracked by #4509 (or drop it with the flag); and the refusal is a new hand-written guard beside the admission policy, so if it stays, move it into targetedTouchPolicyResponse so refusedBeforeDispatch stamps dispatched: 'no', and delete the now-unreachable direct-iOS eligibility early return and the expectScreenshotDiff pass-through at interaction-touch-press.ts#L239.

Could the PR avoid shipping an advertised flag whose only production behavior is a refusal? Landing the capture binding first would remove the refusal, the dead eligibility guard and the pass-through, and the existing bracket would work on the real route. Otherwise, removing the flag from every surface and the matrix and parking the runtime lane until #4509 decides would avoid carrying about 209 production lines that no production route reaches. Could the baseline also fold into the post-action observation plan as a third evidence kind, so it has one owner?

Of the open inline threads, the abort/deadline thread still applies (it must be fixed when a capture binds), as do the unreachable direct-iOS guard, the cli-flags.ts comment, the tap-stub assertion and the metadata promise (covered by the first finding). These need no further action, so please resolve them: the interactions.ts dynamic import is a memoized lookup; the commands.md sentence, the catch after dispatch, the delivered-tap tests, the descriptor guard, the PNG decode and the after-frame failure test are fixed at this head; the any-pixel rule is deferred to #4509 and moot while every route refuses; the command-flags.ts pattern matches the existing verify/settle flags.

Before merge, the advertised flag and the daemon route must agree, either by binding the capture or by removing the flag from every surface and the matrix.

--expect-screenshot-diff now works on the daemon press and click routes: the
route admits the flag, binds the device's screenshot capture through the
screenshot lane's own plan admission, and hands it to the interaction backend,
so the readback bracket runs instead of returning no evidence. A device whose
owner declares no capture still refuses before dispatch, with the screenshot
lane's refusal stamped dispatched no. The refusal guard is gone; callstack#4509 keeps
the any-pixel threshold question.

Co-Authored-By: Claude Code <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 4 files (changes from recent commits).

You’re at about 93% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts">

<violation number="1" location="packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts:368">
P3: The rewritten refusal test no longer pins `details.reason`, so the specific refusal cause is unchecked. Keep a reason assertion, for example `expect(response.error.details?.reason).toBeDefined();`, so a change in the refusal cause fails the test.</violation>
</file>

<file name="packages/daemon/src/execution/interaction/internal/interaction-runtime.ts">

<violation number="1" location="packages/daemon/src/execution/interaction/internal/interaction-runtime.ts:202">
P1: `readbackCapture` never reaches the backend. `createInteractionRuntimeForRoute` accepts it but drops it in the `createInteractionAgentDevice` call, so `captureScreenshot` is never installed and `--expect-screenshot-diff` still silently does nothing on daemon press/click. Forward `readbackCapture: params.readbackCapture` in that call and add the field to `InteractionRuntimeInput`. That also removes the cast here.</violation>

<violation number="2" location="packages/daemon/src/execution/interaction/internal/interaction-runtime.ts:220">
P2: The readback drops the session's capture context, so it can omit configured runner metadata and use the wrong macOS capture surface. Pass the session app bundle, surface, and execution context through this handoff.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic


function createInteractionBackend(params: InteractionRuntimeInput): AgentDeviceBackend {
const { flags, session } = params;
const readbackCapture = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: readbackCapture never reaches the backend. createInteractionRuntimeForRoute accepts it but drops it in the createInteractionAgentDevice call, so captureScreenshot is never installed and --expect-screenshot-diff still silently does nothing on daemon press/click. Forward readbackCapture: params.readbackCapture in that call and add the field to InteractionRuntimeInput. That also removes the cast here.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/daemon/src/execution/interaction/internal/interaction-runtime.ts, line 202:

<comment>`readbackCapture` never reaches the backend. `createInteractionRuntimeForRoute` accepts it but drops it in the `createInteractionAgentDevice` call, so `captureScreenshot` is never installed and `--expect-screenshot-diff` still silently does nothing on daemon press/click. Forward `readbackCapture: params.readbackCapture` in that call and add the field to `InteractionRuntimeInput`. That also removes the cast here.</comment>

<file context>
@@ -193,10 +199,29 @@ function createInteractionAgentDevice(params: InteractionRuntimeInput) {
 
 function createInteractionBackend(params: InteractionRuntimeInput): AgentDeviceBackend {
   const { flags, session } = params;
+  const readbackCapture = (
+    params as InteractionRuntimeInput & {
+      readbackCapture?: (input: { outPath: string; options?: unknown }) => Promise<unknown>;
</file context>

outPath: string,
options?: BackendScreenshotOptions,
) => {
await readbackCapture({ outPath, ...(options ? { options } : {}) });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The readback drops the session's capture context, so it can omit configured runner metadata and use the wrong macOS capture surface. Pass the session app bundle, surface, and execution context through this handoff.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/daemon/src/execution/interaction/internal/interaction-runtime.ts, line 220:

<comment>The readback drops the session's capture context, so it can omit configured runner metadata and use the wrong macOS capture surface. Pass the session app bundle, surface, and execution context through this handoff.</comment>

<file context>
@@ -193,10 +199,29 @@ function createInteractionAgentDevice(params: InteractionRuntimeInput) {
+            outPath: string,
+            options?: BackendScreenshotOptions,
+          ) => {
+            await readbackCapture({ outPath, ...(options ? { options } : {}) });
+            return { path: outPath };
+          },
</file context>

expect(response?.ok, positionals.join(' ')).toBe(false);
if (response && !response.ok) {
// The screenshot lane's own refusal: the owner declares no capture operation.
expect(response.error.code).toBe('UNSUPPORTED_OPERATION');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The rewritten refusal test no longer pins details.reason, so the specific refusal cause is unchecked. Keep a reason assertion, for example expect(response.error.details?.reason).toBeDefined();, so a change in the refusal cause fails the test.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts, line 368:

<comment>The rewritten refusal test no longer pins `details.reason`, so the specific refusal cause is unchecked. Keep a reason assertion, for example `expect(response.error.details?.reason).toBeDefined();`, so a change in the refusal cause fails the test.</comment>

<file context>
@@ -364,9 +364,10 @@ test('press --expect-screenshot-diff refuses at the route while no capture binds
-      expect(response.error.code).toBe('INVALID_ARGS');
-      expect(response.error.details?.reason).toBe('screenshot-readback-unavailable');
+      // The screenshot lane's own refusal: the owner declares no capture operation.
+      expect(response.error.code).toBe('UNSUPPORTED_OPERATION');
       expect(response.error.details?.dispatched).toBe('no');
+      expect(String(response.error.message)).toMatch(/screenshot/i);
</file context>
Suggested change
expect(response.error.code).toBe('UNSUPPORTED_OPERATION');
expect(response.error.code).toBe('UNSUPPORTED_OPERATION');
expect(response.error.details?.reason).toBeDefined();

strzhao and others added 3 commits November 15, 2026 13:18
The interaction route's artifacts policy refuses temp files by design, so the
readback's before and after frames never got created and the flag answered
with silence even after the capture bound. The lane owns its frame directory
(mkdtemp plus cleanup with the frame) instead of asking the route's adapter.
A device whose owner declares no capture still refuses before dispatch with
the screenshot lane's refusal, stamped dispatched no.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
@strzhao strzhao changed the title feat(interaction): guaranteed press baselines and --expect readback feat(interaction): screenshot readback (--expect-screenshot-diff) on press and click Nov 15, 2026
@strzhao

strzhao commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

The capture now binds into the touch backend — route A, as you laid out. At e62ff68 (the binding at 59a058cbb + the frame-file fix + the live evidence):

The binding. dispatchTargetedTouchViaRuntime admits the flag and binds the device's screenshot capture through the screenshot lane's own plan admission (resolveBoundScreenshotRuntime, 'capture' plan); the capture is handed to createInteractionBackend, so runtime.backend.captureScreenshot exists on the daemon press/click routes and the readback bracket runs. The refusal guard is deleted; a device whose owner declares no capture still refuses before dispatch (the screenshot lane's own refusal, stamped dispatched: 'no').

Route test. The plan and single-ref forms both drive the route; a capture-less device refuses with the stamped verdict, and the capture-backed lane is exercised by the evidence test suite (backend member present → bracket runs → screenshotDiff in the result).

Live runs (iOS simulator, PIN-pad rig; JSON excerpts in docs/evidence/ios-screenshot-readback-2026-11-15.md):

The guarantees matrix keeps the runtime rows enforced (the route now reaches them); the abort/deadline fix landed with the binding (the settle sleep takes the request signal), and the unreachable-guard/pass-through notes dissolved with the refusal's deletion. 793 interaction/daemon tests, typecheck, lint, and fallow pass at the pushed head.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 issues found across 3 files (changes from recent commits).

You’re at about 93% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/daemon/src/execution/interaction/internal/interaction-runtime.ts">

<violation number="1" location="packages/daemon/src/execution/interaction/internal/interaction-runtime.ts:65">
P2: Remove this temporary `console.error`; it writes a `[readback-diag]` message to stderr on every interaction route invocation and pollutes daemon logs.</violation>
</file>

<file name="docs/evidence/ios-screenshot-readback-2026-11-15.md">

<violation number="1" location="docs/evidence/ios-screenshot-readback-2026-11-15.md:49">
P3: The ratio comparison contradicts itself. 0.0006 is about six times below the effective key's 0.0037, not an order of magnitude. Replace 'an order of magnitude below' with 'about six times below' so the sentence matches the next sentence.</violation>

<violation number="2" location="docs/evidence/ios-screenshot-readback-2026-11-15.md:49">
P3: This cause is asserted without evidence. Either record a rerun with the status bar excluded or masked, or phrase it as a hypothesis. Also list the dead-tap command and coordinates so the sample can be reproduced.</violation>

<violation number="3" location="docs/evidence/ios-screenshot-readback-2026-11-15.md:50">
P2: The dead-tap pair still contains a ticking status-bar clock even though the readback requests `normalizeStatusBar: true` and simulator captures pin the time to 9:41. Fix the daemon-bound capture normalization before treating this as only a threshold-policy question; otherwise status-bar changes mask no-op taps.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

const ref = bindInteractionSession(params).sessionRef;
if (!ref) throw new KernelAppError('SESSION_NOT_FOUND', NO_ACTIVE_SESSION_MESSAGE);
const session = params.sessionStore.requireCurrent(ref);
console.error(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Remove this temporary console.error; it writes a [readback-diag] message to stderr on every interaction route invocation and pollutes daemon logs.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/daemon/src/execution/interaction/internal/interaction-runtime.ts, line 65:

<comment>Remove this temporary `console.error`; it writes a `[readback-diag]` message to stderr on every interaction route invocation and pollutes daemon logs.</comment>

<file context>
@@ -62,10 +62,15 @@ export async function createInteractionRuntimeForRoute(
   const ref = bindInteractionSession(params).sessionRef;
   if (!ref) throw new KernelAppError('SESSION_NOT_FOUND', NO_ACTIVE_SESSION_MESSAGE);
   const session = params.sessionStore.requireCurrent(ref);
+  console.error(
+    '[readback-diag] forRoute params.readbackCapture =',
+    typeof (params as { readbackCapture?: unknown }).readbackCapture,
</file context>

```

The ratio is an order of magnitude below the effective key's, but still above
zero: the status-bar clock ticks between the bracket's frames even on a page

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The dead-tap pair still contains a ticking status-bar clock even though the readback requests normalizeStatusBar: true and simulator captures pin the time to 9:41. Fix the daemon-bound capture normalization before treating this as only a threshold-policy question; otherwise status-bar changes mask no-op taps.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/evidence/ios-screenshot-readback-2026-11-15.md, line 50:

<comment>The dead-tap pair still contains a ticking status-bar clock even though the readback requests `normalizeStatusBar: true` and simulator captures pin the time to 9:41. Fix the daemon-bound capture normalization before treating this as only a threshold-policy question; otherwise status-bar changes mask no-op taps.</comment>

<file context>
@@ -0,0 +1,55 @@
+```
+
+The ratio is an order of magnitude below the effective key's, but still above
+zero: the status-bar clock ticks between the bracket's frames even on a page
+whose content is otherwise static. This is the any-pixel rule's live shape and
+the concrete case for the #4509 threshold question — the readback correctly
</file context>

}
```

The ratio is an order of magnitude below the effective key's, but still above

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The ratio comparison contradicts itself. 0.0006 is about six times below the effective key's 0.0037, not an order of magnitude. Replace 'an order of magnitude below' with 'about six times below' so the sentence matches the next sentence.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/evidence/ios-screenshot-readback-2026-11-15.md, line 49:

<comment>The ratio comparison contradicts itself. 0.0006 is about six times below the effective key's 0.0037, not an order of magnitude. Replace 'an order of magnitude below' with 'about six times below' so the sentence matches the next sentence.</comment>

<file context>
@@ -0,0 +1,55 @@
+}
+```
+
+The ratio is an order of magnitude below the effective key's, but still above
+zero: the status-bar clock ticks between the bracket's frames even on a page
+whose content is otherwise static. This is the any-pixel rule's live shape and
</file context>

}
```

The ratio is an order of magnitude below the effective key's, but still above

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This cause is asserted without evidence. Either record a rerun with the status bar excluded or masked, or phrase it as a hypothesis. Also list the dead-tap command and coordinates so the sample can be reproduced.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/evidence/ios-screenshot-readback-2026-11-15.md, line 49:

<comment>This cause is asserted without evidence. Either record a rerun with the status bar excluded or masked, or phrase it as a hypothesis. Also list the dead-tap command and coordinates so the sample can be reproduced.</comment>

<file context>
@@ -0,0 +1,55 @@
+}
+```
+
+The ratio is an order of magnitude below the effective key's, but still above
+zero: the status-bar clock ticks between the bracket's frames even on a page
+whose content is otherwise static. This is the any-pixel rule's live shape and
</file context>

Co-Authored-By: Claude Code <noreply@anthropic.com>
@thymikee

Copy link
Copy Markdown
Member

Thanks for the update. At f49ebe6 the readback now reaches the daemon route and the iOS simulator run shows screenshotDiff in the JSON, so the main gap from the earlier review is closed. Several problems remain, and the first one means the flag does not yet catch the case it exists for. CI is green (19/19), but no job drives a daemon press/click with the flag on a device that can capture, so green does not cover the binding or the frame-file path. There are no conflicts.

The evidence still has gaps (doc line 36). It covers only the iOS simulator, and its dead-region run reads changedFromBefore:true at 0.0006 with no warning. The 'tapped, nothing happened' case has not produced false plus the warning on any device. Android binds a different capture owner (platform-android/src/screenshot.ts), so the changed path is unproven there. Please post Android emulator runs of press <x> <y> --expect-screenshot-diff --json on an effective key (true plus ratio) and on a dead region (false plus the 'Expected a screen change' warning), and click @ref --expect-screenshot-diff --json on a runtime ref. Please also rerun the iOS dead region so it reads false plus the warning. Each run should name the head SHA and include the full JSON.

The dead-tap false positive is probably caused by the readback itself (screenshot-diff-evidence.ts:135). Each frame is a separate captureScreenshot with normalizeStatusBar:true. On an iOS simulator each call snapshots the overrides, sets 9:41, captures, and restores the live bar (screenshot-status-bar.ts:96-118). So the bar goes normalized, live, normalized across the bracket, and any residue lands in one frame. With the any-pixel rule at line 200, a caller who taps a dead PIN-pad coordinate is told the screen changed and gets no warning. The clock-tick explanation cannot hold under normalization. The rule should be: the two frames differ only by what the dispatch caused, so every change the readback makes must be identical in both frames. Hold one status-bar normalization across before-frame, dispatch and after-frame, or exclude the status-bar band from the ratio, then prove it with the iOS dead-tap rerun above. If the diff turns out to be page content, save both frames and locate the changed pixels before naming a cause.

The new refusal contradicts the docs (interaction-touch-press.ts:57). When the device owner declares no captureScreenshot, press/click fails with UNSUPPORTED_OPERATION and a hand-stamped dispatched:'no'. But commands.md:502, client-gesture.ts:70, cli-flags.ts:132, interaction.ts:263-265 and the lane comment at screenshot-diff-evidence.ts:16-18 all say the flag is advisory and never fails the action. A caller who adds the flag on a capture-less device expects the tap to dispatch, and the digit is never entered. Please delete lines 57-72 and the longpress/hover name filter at :48. The lane already returns no evidence when backend.captureScreenshot is absent, which is what all five texts describe. If the refusal must stay, move it into targetedTouchPolicyResponse so refusedBeforeDispatch stamps the verdict, and rewrite those five texts to say a capture-less device refuses.

The frame file is a copy of a daemon helper (screenshot-diff-evidence.ts:121). The inline mkdtemp, path.join and fs.rm cleanup is near-verbatim createTempFile from screenshot-artifact-adapter.ts:54-62. It is the only mkdtemp in command-runtime, and it bypasses the runtime.artifacts port because the daemon interaction policy refuses temp files. Every other temp file in this package goes through the host's artifacts adapter, so this lane now writes to the process tmpdir whatever adapter is set. Please restore createCommandTempFile in the lane. In the daemon, extract createTempFile into runtime-artifacts.ts and give the press/click route an artifacts adapter that uses it, for example an option on createDaemonRuntimePolicy.

The readback capture input is too thin (interaction-runtime.ts:225). It sends only {outPath, options:{normalizeStatusBar}} and ignores the context. The screenshot command also builds appBundleId, skipIosSimulatorBootCheck, surface, preferredCaptureBackend and the runner execution from the dispatch context (screenshot-runtime.ts:380-398). On a macOS app session the runner captures with appBundleId undefined. On a physical iOS device it gets no runnerOptions, so the capture can fail and the readback silently returns nothing. On a simulator every frame runs ensureBooted, which widens the gap before the dispatch. The rule: a readback frame is captured with the same input the screenshot command would build for this request. Please build that input in the daemon from contextFromFlags(...) and screenshotExecutionFromContext, reusing the construction in screenshot-runtime.ts, instead of forwarding a bare outPath. I have not run the macOS or physical-device paths, so this part is read from platform-apple/src/core/screenshot.ts.

No daemon test covers the bound route (interaction-touch-press-admission.test.ts:355). The contract scenarios and screenshot-diff-evidence.test.ts inject captureScreenshot straight into the command-runtime harness. Both bugs this delta fixed after finding them live passed every test: readbackCapture dropped before interaction-runtime.ts:73, and the createCommandTempFile refusal under createDaemonRuntimePolicy. By grep, removing line 73 keeps the suite green. Please add a daemon-route test with a session whose owner declares captureScreenshot, where the bound capture writes two different PNGs. Assert response.data.screenshotDiff.changedFromBefore === true, assert the capture received the session's appBundleId, and check that the test goes red when line 73 is removed.

There is leftover debug output (interaction-runtime.ts:65). An unconditional [readback-diag] console.error runs in createInteractionRuntimeForRoute, which serves all four daemon interaction routes, so every interaction request writes to daemon stderr. Please delete lines 65-68.

Not blocking: readbackCapture is typed (input:{outPath; options?: unknown}) => Promise<unknown> and is not declared on InteractionRuntimeInput, so it travels through a conditional spread (interaction-runtime.ts:73) and is recovered with casts (:207-211 and interaction-touch-press.ts:120). That erasure is why a dropped field compiled cleanly. Declaring readbackCapture?: BoundScreenshotRuntime['captureScreenshot'] on InteractionRuntimeInput and dropping the casts and spreads would remove it, but you can take or leave this.

Could this be simpler? The daemon could hand the touch backend one typed BoundScreenshotRuntime['captureScreenshot'], with its input built from the request's dispatch context the way the screenshot command does. The route's artifacts adapter would use the daemon's existing createTempFile, so the lane keeps createCommandTempFile and no raw fs. With the refusal deleted, the lane's existing absent-evidence path is the documented best-effort behavior, and the hand-stamped verdict, the name filter, the casts and the spreads all go away. Status-bar normalization would be held once across the bracket instead of per frame. What would have to change first is extracting createTempFile from screenshot-artifact-adapter.ts into a daemon runtime-artifacts helper, and deciding (or folding into #4509) whether the bracket holds the status bar or masks it.

On the open inline threads: these still stand and need no restatement: the capture input, the console.error, the abort signal (it still passes options.signal, which the daemon routes never set, and continues to the after-frame after an abort; use toBackendContext(runtime, options).signal and return {} when aborted), the 9:41 clock explanation, and the any-pixel rule (resolved earlier, but the route is now live and the dead tap read true). Seven lower-priority threads also still apply. This thread on the readback binding does not apply any more, because line 73 now passes readbackCapture through and the live JSON shows screenshotDiff, so you can resolve it. The typed-reason thread becomes moot if the refusal is deleted.

Before merge: make the readback's status-bar handling identical in both frames, then post the iOS dead-region run that reads false plus the warning and the Android emulator set. Also add the daemon-route test, delete the [readback-diag] line, settle the refusal and docs mismatch, reuse the daemon createTempFile and the screenshot input construction, and fix the abort handling. I ran no tests and no devices for this review. The 0.0006 cause is not determined: the per-frame override and restore is the likely mechanism, but page content is not ruled out. I did not verify the reported 793 passing tests locally.

…xt, and restore the temp-file port

The readback frame now carries the input the screenshot command would build
for this request: the session's app identity, the simulator boot-check
knowledge, and the runner execution from the dispatch context. The shared
temp-file implementation moves into runtime-artifacts, the interaction route's
policy opts in, and the lane goes back to the runtime's createCommandTempFile.
The advisory case on a capture-less device pins dispatch-with-no-evidence.

Co-Authored-By: Claude Code <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 existing issues remain and 6 new issues found across 11 files (changes from recent commits).

You’re at about 94% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/daemon/src/execution/runtime-policy.ts">

<violation number="1" location="packages/daemon/src/execution/runtime-policy.ts:10">
P3: The new `allowTempFiles` branch has no test. Add a case that calls `createDaemonRuntimePolicy('interaction commands', { allowTempFiles: true })` and asserts `createTempFile` creates a file, while the default still rejects with `UNSUPPORTED_OPERATION`.</violation>
</file>

<file name="packages/capture-kit/src/png-changed-pixel-ratio.ts">

<violation number="1" location="packages/capture-kit/src/png-changed-pixel-ratio.ts:21">
P3: The docstring says `ignoreTopHeightShare` removes the top band from the denominator too, but the code keeps the full-frame `totalPixels` denominator. Reword the doc to match the code, which the test and the last docstring sentence already assume. The inline comment above the `ignoredRows` block ("contribute nothing to either side of the ratio") has the same error and needs the same fix.</violation>
</file>

<file name="packages/daemon/src/execution/interaction/internal/interaction-runtime.ts">

<violation number="1" location="packages/daemon/src/execution/interaction/internal/interaction-runtime.ts:202">
P3: `allowTempFiles: true` grants temp-file creation to every interaction route, not just the admitted readback. Tie it to the readback admission so only that request gets the capability.</violation>
</file>

<file name="packages/daemon/src/execution/runtime-artifacts.ts">

<violation number="1" location="packages/daemon/src/execution/runtime-artifacts.ts:11">
P3: The doc comment links to `createTempFileArtifactAdapter`, which does not exist, so the reference is broken. The opt-in actually happens through `createDaemonRuntimePolicy(..., { allowTempFiles: true })`. Also, 'every daemon route shares' contradicts the rest of the comment, since refusing routes don't use it. Point the comment at `createDaemonRuntimePolicy` and say only opted-in routes use this implementation.</violation>

<violation number="2" location="packages/daemon/src/execution/runtime-artifacts.ts:22">
P2: The readback awaits this cleanup without catching errors; if `rm` rejects after a failed before-capture, the advisory readback escapes before dispatch and prevents the press/click. Keep this cleanup best-effort.</violation>
</file>

<file name="packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts">

<violation number="1" location="packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts:47">
P2: This unconditional debug print writes every targeted touch's full flags to stderr, polluting daemon logs and potentially exposing user-supplied values; remove it.</violation>
</file>

Requires human review: Auto-approval blocked because this review re-detected 3 unresolved issues already reported by Cubic.

View guided diff | Re-trigger cubic

path: path.join(root, `file${options.ext}`),
visibility: 'internal',
cleanup: async () => {
await fs.rm(root, { recursive: true, force: true });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The readback awaits this cleanup without catching errors; if rm rejects after a failed before-capture, the advisory readback escapes before dispatch and prevents the press/click. Keep this cleanup best-effort.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/daemon/src/execution/runtime-artifacts.ts, line 22:

<comment>The readback awaits this cleanup without catching errors; if `rm` rejects after a failed before-capture, the advisory readback escapes before dispatch and prevents the press/click. Keep this cleanup best-effort.</comment>

<file context>
@@ -1,6 +1,29 @@
+    path: path.join(root, `file${options.ext}`),
+    visibility: 'internal',
+    cleanup: async () => {
+      await fs.rm(root, { recursive: true, force: true });
+    },
+  };
</file context>

command: TargetedTouchCommand,
): Promise<DaemonResponse> {
params = bindInteractionSession(params);
console.error('[readback-diag] route flags =', JSON.stringify(params.req.flags));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This unconditional debug print writes every targeted touch's full flags to stderr, polluting daemon logs and potentially exposing user-supplied values; remove it.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts, line 47:

<comment>This unconditional debug print writes every targeted touch's full flags to stderr, polluting daemon logs and potentially exposing user-supplied values; remove it.</comment>

<file context>
@@ -41,35 +44,28 @@ export async function dispatchTargetedTouchViaRuntime(
   command: TargetedTouchCommand,
 ): Promise<DaemonResponse> {
   params = bindInteractionSession(params);
+  console.error('[readback-diag] route flags =', JSON.stringify(params.req.flags));
   const admission = await admitTargetedTouch(params, command);
   if ('response' in admission) return admission.response;
</file context>

): Pick<AgentDeviceRuntimeConfig, 'artifacts' | 'policy'> {
return {
artifacts: createUnsupportedArtifactAdapter(unsupportedArtifactLabel, options),
artifacts: options.allowTempFiles

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The new allowTempFiles branch has no test. Add a case that calls createDaemonRuntimePolicy('interaction commands', { allowTempFiles: true }) and asserts createTempFile creates a file, while the default still rejects with UNSUPPORTED_OPERATION.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/daemon/src/execution/runtime-policy.ts, line 10:

<comment>The new `allowTempFiles` branch has no test. Add a case that calls `createDaemonRuntimePolicy('interaction commands', { allowTempFiles: true })` and asserts `createTempFile` creates a file, while the default still rejects with `UNSUPPORTED_OPERATION`.</comment>

<file context>
@@ -1,13 +1,18 @@
 ): Pick<AgentDeviceRuntimeConfig, 'artifacts' | 'policy'> {
   return {
-    artifacts: createUnsupportedArtifactAdapter(unsupportedArtifactLabel, options),
+    artifacts: options.allowTempFiles
+      ? {
+          ...createUnsupportedArtifactAdapter(unsupportedArtifactLabel, options),
</file context>

Comment on lines +21 to +25
* `ignoreTopHeightShare` cuts the top share of the frame out of both the changed count and the
* denominator, for bands whose pixels move without the dispatch causing it — a status-bar clock
* ticks between the bracket's frames on every platform, and the readback must answer for what the
* dispatch changed, not for the wall clock. The excluded rows are zeroed in the comparison, so the
* share is a plain fraction of the full frame either way.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The docstring says ignoreTopHeightShare removes the top band from the denominator too, but the code keeps the full-frame totalPixels denominator. Reword the doc to match the code, which the test and the last docstring sentence already assume. The inline comment above the ignoredRows block ("contribute nothing to either side of the ratio") has the same error and needs the same fix.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/capture-kit/src/png-changed-pixel-ratio.ts, line 21:

<comment>The docstring says `ignoreTopHeightShare` removes the top band from the denominator too, but the code keeps the full-frame `totalPixels` denominator. Reword the doc to match the code, which the test and the last docstring sentence already assume. The inline comment above the `ignoredRows` block ("contribute nothing to either side of the ratio") has the same error and needs the same fix.</comment>

<file context>
@@ -17,10 +17,17 @@ export type PngChangedPixelRatioResult =
  * Dimensions must match. Callers comparing frames of different sizes are asking a different
  * question, and returning a made-up ratio would hide it.
+ *
+ * `ignoreTopHeightShare` cuts the top share of the frame out of both the changed count and the
+ * denominator, for bands whose pixels move without the dispatch causing it — a status-bar clock
+ * ticks between the bracket's frames on every platform, and the readback must answer for what the
</file context>
Suggested change
* `ignoreTopHeightShare` cuts the top share of the frame out of both the changed count and the
* denominator, for bands whose pixels move without the dispatch causing it — a status-bar clock
* ticks between the bracket's frames on every platform, and the readback must answer for what the
* dispatch changed, not for the wall clock. The excluded rows are zeroed in the comparison, so the
* share is a plain fraction of the full frame either way.
* `ignoreTopHeightShare` leaves the top share of the frame out of the changed count. Those rows
* are skipped rather than removed from the denominator, so the ratio stays a fraction of the full
* frame. The option is for bands whose pixels move without the dispatch causing it: a status-bar
* clock ticks between the bracket's frames on every platform, and the readback must answer for what
* the dispatch changed, not for the wall clock.

...createDaemonRuntimePolicy('interaction commands', {
plural: true,
// The screenshot readback's before/after frames are this route's own temp files.
allowTempFiles: true,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: allowTempFiles: true grants temp-file creation to every interaction route, not just the admitted readback. Tie it to the readback admission so only that request gets the capability.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/daemon/src/execution/interaction/internal/interaction-runtime.ts, line 202:

<comment>`allowTempFiles: true` grants temp-file creation to every interaction route, not just the admitted readback. Tie it to the readback admission so only that request gets the capability.</comment>

<file context>
@@ -196,19 +196,19 @@ function resolveBoundCapture(
+    ...createDaemonRuntimePolicy('interaction commands', {
+      plural: true,
+      // The screenshot readback's before/after frames are this route's own temp files.
+      allowTempFiles: true,
+    }),
     sessions: params.runtimeSessions,
</file context>
Suggested change
allowTempFiles: true,
allowTempFiles: params.readbackCapture !== undefined,

* The one temp-file implementation every daemon route shares: an `agent-device-*` root under the
* process tmpdir, one file inside, removed with the cleanup. Routes whose policy refuses temp
* files keep the refusing adapter; routes that opt in (an artifacts adapter from
* {@link createTempFileArtifactAdapter}) get this implementation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The doc comment links to createTempFileArtifactAdapter, which does not exist, so the reference is broken. The opt-in actually happens through createDaemonRuntimePolicy(..., { allowTempFiles: true }). Also, 'every daemon route shares' contradicts the rest of the comment, since refusing routes don't use it. Point the comment at createDaemonRuntimePolicy and say only opted-in routes use this implementation.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/daemon/src/execution/runtime-artifacts.ts, line 11:

<comment>The doc comment links to `createTempFileArtifactAdapter`, which does not exist, so the reference is broken. The opt-in actually happens through `createDaemonRuntimePolicy(..., { allowTempFiles: true })`. Also, 'every daemon route shares' contradicts the rest of the comment, since refusing routes don't use it. Point the comment at `createDaemonRuntimePolicy` and say only opted-in routes use this implementation.</comment>

<file context>
@@ -1,6 +1,29 @@
+ * The one temp-file implementation every daemon route shares: an `agent-device-*` root under the
+ * process tmpdir, one file inside, removed with the cleanup. Routes whose policy refuses temp
+ * files keep the refusing adapter; routes that opt in (an artifacts adapter from
+ * {@link createTempFileArtifactAdapter}) get this implementation.
+ */
+export async function createDaemonTempFile(options: {
</file context>

@thymikee

Copy link
Copy Markdown
Member

Thanks for the update. I reviewed f66aa4b against the earlier review, including the delta and the CI logs. The readback route and capture-input wiring from the earlier review are now in place, but the status-bar problem is still open, so the dead-tap case can still report a change. There are no conflicts.

The new ignoreTopHeightShare option has no effect on the route that failed. The only production caller still calls computePngChangedPixelRatio(beforePng, afterPng) with no options (screenshot-diff-evidence.ts:186). Each frame is still its own capture with its own status-bar override and restore, requested at :129 and forced again at interaction-touch-press.ts:123. Status-bar residue can still reach the any-pixel rule at :191. So a tap on a dead PIN-pad coordinate can still return changedFromBefore:true with no 'Expected a screen change' warning, which is the case this flag exists to catch. The rule to satisfy: every pixel the readback itself causes is identical in both frames, so the frames differ only by what the dispatch caused. Please give that decision one owner. Either hold one status-bar normalization across the before-frame, the dispatch and the after-frame, or exclude a device-provided status-bar band at the single compare call, and remove the second normalizeStatusBar site. The cause of the 0.0006 diff is not yet proven. If the residue turns out to be page content, save both frames and locate the changed pixels before naming a cause.

No daemon test covers the bound readback route. The only daemon test that sets expectScreenshotDiff is the capture-less case at interaction-touch-press-admission.test.ts:355. Deleting the forwarding at interaction-runtime.ts:73, or the options and execution built at interaction-touch-press.ts:119-130, keeps the whole suite green. Both readback bugs found live on earlier heads passed every test for the same reason. Please add a daemon-route test with a session whose owner declares captureScreenshot. Its fake capture should write two different PNGs, and the test should assert response.data.screenshotDiff.changedFromBefore === true and that the capture input carries the session's appBundleId, surface and runner execution. Add a same-PNG case that asserts false plus the warning. Please confirm the test goes red when interaction-runtime.ts:73 is removed.

The readback still builds its own CaptureScreenshotInput in the touch route (interaction-touch-press.ts:119), beside the one the screenshot command owns in createBoundScreenshotBackend (screenshot-runtime.ts:378-398). The copy has already drifted: it omits surface, omits preferredCaptureBackend, and ignores the skipIosSimulatorBootCheck override. On a macOS session with a non-default surface or a preferred backend, a readback frame may come from a different surface or backend than screenshot would use. The rule: a readback frame is captured with exactly the input the screenshot command builds for this request, from one builder. Could the captureScreenshot body of createBoundScreenshotBackend become an exported helper that takes (session, dispatchContext, captureScreenshot), with the touch route passing its result as readbackCapture and building no options itself?

This is a device-facing change, and the evidence doc (ios-screenshot-readback-2026-11-15.md:36) is unchanged by the delta. No device has yet shown changedFromBefore:false plus the warning for a dead tap, and Android binds a different capture owner (platform-android/src/screenshot.ts) with no run. After the status-bar fix, please post full JSON at the new head. On an iOS simulator, press <x> <y> --expect-screenshot-diff --json on a dead region should read changedFromBefore:false plus the 'Expected a screen change' warning. On an Android emulator, the same command should read true with changedPixelRatio on an effective key and false plus the warning on a dead region. Also run click @ref --expect-screenshot-diff --json on a runtime ref.

Checks show Repo Guards failing on one Fallow complexity breach in computePngChangedPixelRatio (png-changed-pixel-ratio.ts:27, cyclomatic 14, cognitive 19). The new row and column loop for ignoreTopHeightShare causes it, so this PR owns it. The Coverage failure cannot be attributed from the log excerpt, which ends mid-run with no failing assertion. The --iterations must be a positive integer (got lots) line is expected output from the negative-case test at scripts/fuzz/harness.test.ts:198, not the failure. The log does show the new [readback-diag] route flags print from interaction-touch-press.ts:47 in every daemon touch test, so please delete it.

Not blocking: if the status-bar fix does not use ignoreTopHeightShare, delete the option and its test. If it does, a single flat loop starting at offset ignoredRows*width*4 keeps the function under the Fallow threshold. You can take or leave this.

Simplicity question: the delta adds a second normalizeStatusBar site, an unused capture-kit option and a policy flag around one readback concept, and the status-bar problem is still open. Would a smaller shape work? The screenshot runtime exports one per-request capture builder and the touch route builds no screenshot options. The readback lane owns the status-bar decision once, by holding one normalization across both frames or by excluding one device-provided band at the single compare call. ignoreTopHeightShare is deleted unless that decision uses it. That removes the duplicate normalizeStatusBar, the drifted option subset and the dead helper together. What would need to be decided first is whether the bracket holds the status bar or masks it, perhaps together with #4509. Holding it needs a platform capture option that keeps one status-bar override across two captures. Masking it needs a device-reported status-bar inset.

These open inline threads still apply. The higher-priority ones are signal handling on abort, missing surface and preferred backend, per-frame status-bar normalization, temp-file cleanup without catch and stderr route-flags print. Eight lower-priority threads also still apply. These threads no longer apply and can be resolved: r4577119988 because interaction-runtime.ts:73 now forwards a typed readbackCapture, r4577119992 because the refusal is deleted and the test now asserts dispatch with no evidence, r4577160406 because the debug print moved to interaction-touch-press.ts:47, which r4579386118 tracks, and r4579386142 because the only interaction caller of createCommandTempFile is the readback lane.

Before merge, please make the readback's own pixels identical in both frames with one status-bar decision wired at screenshot-diff-evidence.ts:186. Then please post the iOS dead-region run at the new head that reads changedFromBefore:false plus the warning, and delete the [readback-diag] print.

@thymikee

Copy link
Copy Markdown
Member

This branch now conflicts with main at f66aa4b. Please rebase onto main and push along with the fixes from the earlier review, and I will re-check the new head.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants