Skip to content

Preserve recording overlay coordinate references before the first snapshot #4448

Open

Description

@thymikee

Current execution status — 2026-11-13

Open; adopted fourth-wave scope. Renderer-refusal source is reviewed, but rendered-frame execution and the native producer/canvas basis remain incomplete. Planning-viewport projection is rejected; the positive no-snapshot route still needs proof.

Latest detailed Current state.

Related: #4447/#4428 and audit umbrella #4368. Adopted into the authorized fourth-wave continuation through the existing issue; no duplicate implementation.

Observed trigger

On candidate 2640216c2eff655a3374130f4451b8e383096736, owned iPhone18ProMax/iOS27.0 (1C2C9FD4-B720-4160-BAEB-7D4724D7B5F3), a fresh session ran open Settings without a foreground/snapshot capture, then record start ... --scope device, swipe 220 700 220 300 --count 3 --pause-ms 500, and record stop. The resulting1320x2868/30fps video contains three timed swipe markers. The telemetry has x=220 and y=700→300, but no referenceWidth/referenceHeight; rendered markers appear at x≈220video pixels instead of scaling point-space input to the3x canvas. Settings visibly scrolls. Cleanup completed.

The timing candidate changes no coordinate-reference or Swift-scaling policy. Source on the named base also has the same snapshot-only fallback and Swift1.0 scaling for absent dimensions; no separate baseline live run is claimed.

Existing owner and prevention contract

packages/daemon/src/core/recording-gestures.ts:resolveEventReferenceFrame uses explicit result dimensions or getSnapshotReferenceFrame(session.snapshot). If both are absent, the event lacks a coordinate reference. apple/runner/AgentDeviceRunner/RecordingScripts/recording-overlay.swift:overlayPoint/scaledAxis treats missing reference size as1.0, silently interpreting point coordinates as video pixels.

Use the existing admitted touch/viewport coordinate owner to construct unambiguous recording telemetry for valid commands with no prior snapshot. Define coordinate-space/reference dimensions in the owning type and construction boundary. Do not infer the space from a magic3x ratio or invent a second device-size cache. For truly unknown coordinates, choose an explicit truthful refusal/omission policy rather than a plausible wrong overlay. Preserve the documented no-snapshot command flow; do not require an unrelated screenshot merely to repair hidden recording state.

Completion evidence

Exercise actual action→recording construction before any snapshot, at point/pixel scaling factors including1x/3x, and verify burned-in marker position from rendered frames against native touch coordinates. Include existing snapshot-backed and explicit-reference controls, rotation/scoped target behavior as applicable, and independent app/video evidence. Old producer or a missing-reference mutant must fail. Require current overlap check, independent review, full format, exact-head validation and applicable CI. Keep #4050 and budgets; no blanket test deletion, fallback framework, merge/release/tags.

This issue needs source/compatibility triage before implementation. The separate repeated-timing regression remains #4428; this report does not claim a new timing regression or a ready fix.

Activity

  1. apex-sentinel-staging commented on Nov 11, 2026

    @apex-sentinel-staging

    Apex Sentinel · Issue review

    Suggested label: bug

    The issue reports incorrect existing behavior where recording telemetry lacks reference dimensions before the first snapshot and the Swift overlay scales missing references as 1.0, causing point-space coordinates to be rendered as video pixels. It includes a concrete repro command sequence, version/commit, and source owners, making it actionable.

    Required sections

    • Steps to reproduce (missing)
    • Expected and actual behavior (missing)
    • Version or environment (missing)
  2. added
    bugSomething isn't working
    and removed on Nov 12, 2026
  3. thymikee commented on Nov 12, 2026

    @thymikee
    MemberAuthor

    Decision: fourth-wave adoption: This live-discovered issue is now one of the next ten authorized scopes under #4368 (comment). Reuse the existing coordinate construction owner and this issue. Source/compatibility triage and an explicit unknown-reference policy precede implementation. Preserve no-snapshot commands; do not infer a magic display scale or add another viewport cache. The required regression and rendered live evidence remain part of completion.

  4. thymikee commented on Nov 13, 2026

    @thymikee
    MemberAuthor

    Decision: recording coordinate basis and unknown-marker policy:

    Adopt the following finite policy for #4448 at baseline 80075b6 after the source trace and independent review. A gesture planning viewport is a movement constraint, not proof of the recording canvas. Do not derive a recording basis from its width/height or x+width/y+height; keyboard clipping, window placement and scoped recording invalidate that shortcut.

    A marker reference must come from the existing platform producer that actually knows the executed coordinates' space. Keep the planning rect distinct. An additive same-operation basis must name origin, units/extent and orientation/scope sufficiently to establish the transform into the captured canvas. Propagate it through the existing gesture runtime/result and recording construction; do not guess from display scale, take an unrelated snapshot, create another cache, or remove the Apple adapter's intentional coordinate stripping. Where that same-space/canvas relationship is not proven, do not attach a plausible reference. Existing explicit/snapshot-backed paths require their existing validity guarantees; an old snapshot is not authority merely because it exists.

    When an event lacks a finite, positive paired reference, omit only its rendered marker/layers. Preserve action success, the video and the telemetry event. Remove Swift's missing-reference scale=1 fallback on all marker paths. Do not make optional released telemetry fields required or silently change public command responses. This omission policy is intentional observable behavior for unknown geometry.

    The first source slice implements that renderer refusal and authors its direct proof cases. Full #4448 completion still requires a producer-owned reference on the supported no-snapshot flow, exact producer/wire tracing, scope/rotation and keyboard controls, request-path regression/mutation proof and actual native-touch versus rendered-frame evidence at 1x/3x. A renderer-only patch is incomplete. The fabricated 1000x1000 scroll basis remains a separate recorded umbrella residual; this fix does not validate or expand it. No coding against unproven viewport projection is authorized. Keep #4447 repetition timing, #4050 fallback and unrelated PRs intact.

  5. thymikee commented on Nov 13, 2026

    @thymikee
    MemberAuthor

    Current state: This is an adopted fourth-wave scope with one canonical implementation record. The renderer-refusal component is local at ac2d6982244625406fd4d198e38376c8a486f917 (renderer source eb2f4bd8ae9f2103e0de5c497b796347d4154998 plus documentation correction). Independent source review found no P1/P2 in that bounded component. Rendered-frame cases are authored but remain unexecuted; no exact-gate or native coordinate proof is claimed.

    The producer/canvas component is still incomplete. The latest source-grounded plan identifies native display geometry, but the daemon-to-recording propagation and its relationship to the captured canvas still need proof. A separately named basis must survive the planned-gesture response boundary before the daemon can consume it; adding a field before a fixed whitelist and assuming it survives is insufficient.

    Next: measure the native producer basis against an actual captured canvas in the relevant orientations, complete the private basis propagation and exercise the positive no-snapshot route plus unknown-basis omission. Then run rendered-pixel, request-path, isolated regression, exact-head and live evidence. Planning-viewport projection remains rejected. Missing basis omits the marker while retaining the action, telemetry and video. The fabricated 1000×1000 scroll reference remains a separate residual.

    No published PR, readiness or whole-issue completion. No new cache, guessed scale or weakened visual assertion.

  6. thymikee commented on Nov 16, 2026

    @thymikee
    MemberAuthor

    Current state: producer/canvas trace advanced; omission candidate remains unpublished and native evidence unrun

    Apex verified on main cfe0850 that the renderer missing-reference scale1 fallback remains; candidate ac2d6982 is not merged. The source trace identified a telemetry consequence: reference width also affects swipe/back-swipe classification, so basis changes need classification assertions. The recording inspector decodes its own gesture shape and must observe any new private basis field. Existing post-canonical response evidence attachment offers a graft point; a planning viewport is still not a basis.

    Native measurement scripts are authored, not executed or approved as proof. Their hardcoded canvas dimensions and touch coordinates are not accepted evidence for the reserved device. The execution owner must first read actual producer/display bounds and independent video-track geometry, derive safe asymmetric points, and record units/orientation/scope. Baseline track geometry can be measured without assuming the renderer patch proves the transform. Separate omission tests from positive native geometry, and rendered1x fixtures from actual native3x evidence. Stop on contradictions; unknown geometry still omits markers.

    No source edit, build, test, gate or device action ran in this assignment. The reserved owned Apple target remains untouched. A later execution phase must use T3 device_open and its pinned command, retain keyboard/rotation/scope controls and request-route/mutation proof. No new cache, viewport projection, guessed scale, fabricated scroll-basis expansion, public response change, PR or readiness claim.

    Resource update: fresh T3 inventory now shows the previously reserved Apple target booted with ownership unknown to this phase. Its old reservation is not authority to open or stop it. No action was taken on that target. Revalidate ownership or select a fresh owned target before recording measurements; Android native currently owns execution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions