Repository navigation
Description
Current execution status — 2026-11-13
Open; adopted fourth-wave scope. Renderer-refusal source is reviewed, but rendered-frame execution and the native producer/canvas basis remain incomplete. Planning-viewport projection is rejected; the positive no-snapshot route still needs proof.
Latest detailed Current state.
Related: #4447/#4428 and audit umbrella #4368. Adopted into the authorized fourth-wave continuation through the existing issue; no duplicate implementation.
Observed trigger
On candidate 2640216c2eff655a3374130f4451b8e383096736, owned iPhone18ProMax/iOS27.0 (1C2C9FD4-B720-4160-BAEB-7D4724D7B5F3), a fresh session ran open Settings without a foreground/snapshot capture, then record start ... --scope device, swipe 220 700 220 300 --count 3 --pause-ms 500, and record stop. The resulting1320x2868/30fps video contains three timed swipe markers. The telemetry has x=220 and y=700→300, but no referenceWidth/referenceHeight; rendered markers appear at x≈220video pixels instead of scaling point-space input to the3x canvas. Settings visibly scrolls. Cleanup completed.
The timing candidate changes no coordinate-reference or Swift-scaling policy. Source on the named base also has the same snapshot-only fallback and Swift1.0 scaling for absent dimensions; no separate baseline live run is claimed.
Existing owner and prevention contract
packages/daemon/src/core/recording-gestures.ts:resolveEventReferenceFrame uses explicit result dimensions or getSnapshotReferenceFrame(session.snapshot). If both are absent, the event lacks a coordinate reference. apple/runner/AgentDeviceRunner/RecordingScripts/recording-overlay.swift:overlayPoint/scaledAxis treats missing reference size as1.0, silently interpreting point coordinates as video pixels.
Use the existing admitted touch/viewport coordinate owner to construct unambiguous recording telemetry for valid commands with no prior snapshot. Define coordinate-space/reference dimensions in the owning type and construction boundary. Do not infer the space from a magic3x ratio or invent a second device-size cache. For truly unknown coordinates, choose an explicit truthful refusal/omission policy rather than a plausible wrong overlay. Preserve the documented no-snapshot command flow; do not require an unrelated screenshot merely to repair hidden recording state.
Completion evidence
Exercise actual action→recording construction before any snapshot, at point/pixel scaling factors including1x/3x, and verify burned-in marker position from rendered frames against native touch coordinates. Include existing snapshot-backed and explicit-reference controls, rotation/scoped target behavior as applicable, and independent app/video evidence. Old producer or a missing-reference mutant must fail. Require current overlap check, independent review, full format, exact-head validation and applicable CI. Keep #4050 and budgets; no blanket test deletion, fallback framework, merge/release/tags.
This issue needs source/compatibility triage before implementation. The separate repeated-timing regression remains #4428; this report does not claim a new timing regression or a ready fix.
Activity
apex-sentinel-staging commentedon Nov 11, 2026
Apex Sentinel · Issue review
Suggested label:
bugThe issue reports incorrect existing behavior where recording telemetry lacks reference dimensions before the first snapshot and the Swift overlay scales missing references as 1.0, causing point-space coordinates to be rendered as video pixels. It includes a concrete repro command sequence, version/commit, and source owners, making it actionable.
Required sections
thymikee commentedon Nov 12, 2026
Decision: fourth-wave adoption: This live-discovered issue is now one of the next ten authorized scopes under #4368 (comment). Reuse the existing coordinate construction owner and this issue. Source/compatibility triage and an explicit unknown-reference policy precede implementation. Preserve no-snapshot commands; do not infer a magic display scale or add another viewport cache. The required regression and rendered live evidence remain part of completion.
thymikee commentedon Nov 13, 2026
Decision: recording coordinate basis and unknown-marker policy:
Adopt the following finite policy for #4448 at baseline 80075b6 after the source trace and independent review. A gesture planning viewport is a movement constraint, not proof of the recording canvas. Do not derive a recording basis from its width/height or x+width/y+height; keyboard clipping, window placement and scoped recording invalidate that shortcut.
A marker reference must come from the existing platform producer that actually knows the executed coordinates' space. Keep the planning rect distinct. An additive same-operation basis must name origin, units/extent and orientation/scope sufficiently to establish the transform into the captured canvas. Propagate it through the existing gesture runtime/result and recording construction; do not guess from display scale, take an unrelated snapshot, create another cache, or remove the Apple adapter's intentional coordinate stripping. Where that same-space/canvas relationship is not proven, do not attach a plausible reference. Existing explicit/snapshot-backed paths require their existing validity guarantees; an old snapshot is not authority merely because it exists.
When an event lacks a finite, positive paired reference, omit only its rendered marker/layers. Preserve action success, the video and the telemetry event. Remove Swift's missing-reference scale=1 fallback on all marker paths. Do not make optional released telemetry fields required or silently change public command responses. This omission policy is intentional observable behavior for unknown geometry.
The first source slice implements that renderer refusal and authors its direct proof cases. Full #4448 completion still requires a producer-owned reference on the supported no-snapshot flow, exact producer/wire tracing, scope/rotation and keyboard controls, request-path regression/mutation proof and actual native-touch versus rendered-frame evidence at 1x/3x. A renderer-only patch is incomplete. The fabricated 1000x1000 scroll basis remains a separate recorded umbrella residual; this fix does not validate or expand it. No coding against unproven viewport projection is authorized. Keep #4447 repetition timing, #4050 fallback and unrelated PRs intact.
thymikee commentedon Nov 13, 2026
Current state: This is an adopted fourth-wave scope with one canonical implementation record. The renderer-refusal component is local at
ac2d6982244625406fd4d198e38376c8a486f917(renderer sourceeb2f4bd8ae9f2103e0de5c497b796347d4154998plus documentation correction). Independent source review found no P1/P2 in that bounded component. Rendered-frame cases are authored but remain unexecuted; no exact-gate or native coordinate proof is claimed.The producer/canvas component is still incomplete. The latest source-grounded plan identifies native display geometry, but the daemon-to-recording propagation and its relationship to the captured canvas still need proof. A separately named basis must survive the planned-gesture response boundary before the daemon can consume it; adding a field before a fixed whitelist and assuming it survives is insufficient.
Next: measure the native producer basis against an actual captured canvas in the relevant orientations, complete the private basis propagation and exercise the positive no-snapshot route plus unknown-basis omission. Then run rendered-pixel, request-path, isolated regression, exact-head and live evidence. Planning-viewport projection remains rejected. Missing basis omits the marker while retaining the action, telemetry and video. The fabricated 1000×1000 scroll reference remains a separate residual.
No published PR, readiness or whole-issue completion. No new cache, guessed scale or weakened visual assertion.
thymikee commentedon Nov 16, 2026
Current state: producer/canvas trace advanced; omission candidate remains unpublished and native evidence unrun
Apex verified on main cfe0850 that the renderer missing-reference scale1 fallback remains; candidate ac2d6982 is not merged. The source trace identified a telemetry consequence: reference width also affects swipe/back-swipe classification, so basis changes need classification assertions. The recording inspector decodes its own gesture shape and must observe any new private basis field. Existing post-canonical response evidence attachment offers a graft point; a planning viewport is still not a basis.
Native measurement scripts are authored, not executed or approved as proof. Their hardcoded canvas dimensions and touch coordinates are not accepted evidence for the reserved device. The execution owner must first read actual producer/display bounds and independent video-track geometry, derive safe asymmetric points, and record units/orientation/scope. Baseline track geometry can be measured without assuming the renderer patch proves the transform. Separate omission tests from positive native geometry, and rendered1x fixtures from actual native3x evidence. Stop on contradictions; unknown geometry still omits markers.
No source edit, build, test, gate or device action ran in this assignment. The reserved owned Apple target remains untouched. A later execution phase must use T3 device_open and its pinned command, retain keyboard/rotation/scope controls and request-route/mutation proof. No new cache, viewport projection, guessed scale, fabricated scroll-basis expansion, public response change, PR or readiness claim.
Resource update: fresh T3 inventory now shows the previously reserved Apple target booted with ownership unknown to this phase. Its old reservation is not authority to open or stop it. No action was taken on that target. Revalidate ownership or select a fresh owned target before recording measurements; Android native currently owns execution.