Repository navigation
feat(interaction): screenshot readback (--expect-screenshot-diff) on press and click #4297
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
e0f4723
565160e
b6b63bd
70043f9
c6f226d
27429a3
584bcaf
7a61db9
ffff797
434e4ad
59a058c
8ad2aee
9aee85d
e62ff68
f49ebe6
f66aa4b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| # Screenshot readback live on the daemon press/click routes, 2026-11-15 | ||
|
|
||
| Evidence for #4297 at head `9aee85d6a` (the capture binding into the touch | ||
| backend plus the self-owned frame files). The PIN-pad rig again: a keypad page | ||
| served to Safari on the booted iPhone 17 Pro simulator | ||
| (`C1C05FFF-29AD-40D8-94E5-D9CBB25EC6BB`, iOS 26.5), whose digit keys are real | ||
| accessibility buttons. | ||
|
|
||
| ## Effective key: `press 111 300 --expect-screenshot-diff --json` | ||
|
|
||
| ```json | ||
| { | ||
| "success": true, | ||
| "screenshotDiff": { | ||
| "changedFromBefore": true, | ||
| "changedPixelRatio": 0.003685804387672621 | ||
| }, | ||
| "message": "Tapped (111, 300)" | ||
| } | ||
| ``` | ||
|
|
||
| ## Runtime-ref route: `click @e16 --expect-screenshot-diff --json` | ||
|
|
||
| ```json | ||
| { | ||
| "success": true, | ||
| "ref": "e16", | ||
| "screenshotDiff": { | ||
| "changedFromBefore": true, | ||
| "changedPixelRatio": 0.013402666302355499 | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Dead tap: the any-pixel threshold question, on hardware | ||
|
|
||
| A tap into the page's dead area (below the pad) on a static page: | ||
|
|
||
| ```json | ||
| { | ||
| "success": true, | ||
| "screenshotDiff": { | ||
| "changedFromBefore": true, | ||
| "changedPixelRatio": 0.0006062365517948017 | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The ratio is an order of magnitude below the effective key's, but still above | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3: This cause is asserted without evidence. Either record a rerun with the status bar excluded or masked, or phrase it as a hypothesis. Also list the dead-tap command and coordinates so the sample can be reproduced. Prompt for AI agents |
||
| zero: the status-bar clock ticks between the bracket's frames even on a page | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: The dead-tap pair still contains a ticking status-bar clock even though the readback requests Prompt for AI agents |
||
| whose content is otherwise static. This is the any-pixel rule's live shape and | ||
| the concrete case for the #4509 threshold question — the readback correctly | ||
| reports what it saw, and the threshold policy decides what a caller should | ||
| read from a 0.0006. The effective key on the same page reads 0.0036, six | ||
| times the dead tap. | ||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -17,10 +17,17 @@ export type PngChangedPixelRatioResult = | |||||||||||||||||||||
| * | ||||||||||||||||||||||
| * Dimensions must match. Callers comparing frames of different sizes are asking a different | ||||||||||||||||||||||
| * question, and returning a made-up ratio would hide it. | ||||||||||||||||||||||
| * | ||||||||||||||||||||||
| * `ignoreTopHeightShare` cuts the top share of the frame out of both the changed count and the | ||||||||||||||||||||||
| * denominator, for bands whose pixels move without the dispatch causing it — a status-bar clock | ||||||||||||||||||||||
| * ticks between the bracket's frames on every platform, and the readback must answer for what the | ||||||||||||||||||||||
| * dispatch changed, not for the wall clock. The excluded rows are zeroed in the comparison, so the | ||||||||||||||||||||||
| * share is a plain fraction of the full frame either way. | ||||||||||||||||||||||
|
Comment on lines
+21
to
+25
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3: The docstring says Prompt for AI agents
Suggested change
|
||||||||||||||||||||||
| */ | ||||||||||||||||||||||
| export function computePngChangedPixelRatio( | ||||||||||||||||||||||
| first: PngRgbImage, | ||||||||||||||||||||||
| second: PngRgbImage, | ||||||||||||||||||||||
| options: Readonly<{ ignoreTopHeightShare?: number }> = {}, | ||||||||||||||||||||||
| ): PngChangedPixelRatioResult { | ||||||||||||||||||||||
| if (first.width !== second.width || first.height !== second.height) { | ||||||||||||||||||||||
| return { status: 'dimension_mismatch' }; | ||||||||||||||||||||||
|
|
@@ -32,14 +39,26 @@ export function computePngChangedPixelRatio( | |||||||||||||||||||||
| return { status: 'dimension_mismatch' }; | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| // Rows at or above the cut contribute nothing to either side of the ratio. | ||||||||||||||||||||||
| const ignoredRows = | ||||||||||||||||||||||
| options.ignoreTopHeightShare && options.ignoreTopHeightShare > 0 | ||||||||||||||||||||||
| ? Math.min(first.height, Math.floor(first.height * options.ignoreTopHeightShare)) | ||||||||||||||||||||||
| : 0; | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| let changedPixels = 0; | ||||||||||||||||||||||
| for (let offset = 0; offset + 3 < first.data.length; offset += 4) { | ||||||||||||||||||||||
| if ( | ||||||||||||||||||||||
| first.data[offset] !== second.data[offset] || | ||||||||||||||||||||||
| first.data[offset + 1] !== second.data[offset + 1] || | ||||||||||||||||||||||
| first.data[offset + 2] !== second.data[offset + 2] | ||||||||||||||||||||||
| ) { | ||||||||||||||||||||||
| changedPixels += 1; | ||||||||||||||||||||||
| for (let y = 0; y < first.height; y += 1) { | ||||||||||||||||||||||
| const rowChangedBand = y < ignoredRows; | ||||||||||||||||||||||
| for (let x = 0; x < first.width; x += 1) { | ||||||||||||||||||||||
| const offset = (y * first.width + x) * 4; | ||||||||||||||||||||||
| if (offset + 3 >= first.data.length) break; | ||||||||||||||||||||||
| if (rowChangedBand) continue; | ||||||||||||||||||||||
| if ( | ||||||||||||||||||||||
| first.data[offset] !== second.data[offset] || | ||||||||||||||||||||||
| first.data[offset + 1] !== second.data[offset + 1] || | ||||||||||||||||||||||
| first.data[offset + 2] !== second.data[offset + 2] | ||||||||||||||||||||||
| ) { | ||||||||||||||||||||||
| changedPixels += 1; | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
|
|
||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
P3: The ratio comparison contradicts itself. 0.0006 is about six times below the effective key's 0.0037, not an order of magnitude. Replace 'an order of magnitude below' with 'about six times below' so the sentence matches the next sentence.
Prompt for AI agents