Skip to content

fix(apple): use typed screenshot timeouts and observed shutdown recovery - #4544

Merged
thymikee merged 10 commits into
mainfrom
t3/audit-4442-test-gates
Nov 15, 2026
Merged

thymikee merged 10 commits into
mainfrom
t3/audit-4442-test-gates

Conversation

@thymikee

@thymikee thymikee commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes #4442. Recognize the first unindented simctl error header after diagnostics and recover skipped boot checks only from a fresh Shutdown observation capped at 5 seconds. Preserve capture and probe-failure diagnostics with consistent device fields.

Changes 15 files. Scope expansion: isolate process fixtures from live Linux /proc; retain assertions. The feedback correction narrows the scripted state-read matcher and clarifies API docs. Boot memo, capture/boot signal and provider-callback limitations remain disclosed.

Validation

Base d78be6c88715ad80fc658497f23f91c86bf9489b; head d4bc0556139fa4078b4a79a28b81fdb493655dbb.

  • One pinned pnpm 11.17.0 exact affected gate passed: exactCommit=true, checkoutChanged=false, failOpen=false. Eight checks passed; full unit 19,835 passed/10 skipped, related 377 passed. Provider integration deduplicated; coverage/replay checks skipped.
  • Focused control: 80 passed. Each matcher, missing-probe-event and device-key mutant caused 2 intended failures/78 passes; restores passed. Bounded independent source review found no findings. Earlier parser/recovery evidence retains its original SHA.
  • Owned native recovery at b0cc3743: Shutdown probe, one boot, one re-capture and PNG; resources cleaned. The new diagnostics are fixture-tested.
  • Maintainer acceptance before merge: accept the retained 5-second boot-memo limitation and the unobserved real simctl code-60 header/stamp or retry above attempt 1. Diagnostic stderr is redacted/trimmed. Head CI is green (19 successful checks, one skipped); human review is clean. External feat(screenshot): --native-density keeps the display's own pixels #4111 overlaps screenshot paths.

…oot on observed state

The screenshot retry, the runner fallback and the skipped-boot-check recovery each decided
from the tool's prose. `screenshot.ts` matched "timeout waiting for screen surfaces", a
POSIX-domain string pair, and six spellings of a shutdown state, so any CoreSimulator wording
change could silently retire all three decisions at once.

`tool-provider.ts` is the one adapter that runs `xcrun simctl` itself, so it is the one place
allowed to read simctl's error protocol. It takes the top-level header's `domain=
NSPOSIXErrorDomain, code=60` and stamps `details.reason`; a nested `Underlying error` line,
another domain or code, stdout, and a caller-supplied command or simctl callback stay
unattributed. The reason is stamped, never host-kit's `timeoutMs`, which is that layer's own
evidence that it killed a child at its deadline.

Recovery now asks the tool what state the simulator is in. `simulator.ts` owns the one
boot-state read and takes the caller's ceiling, so the probe runs under a 5s screenshot-owned
budget instead of the 60s inventory budget. Only the literal `Shutdown` authorizes a boot:
`Booted`, `Booting`, `Creating` and `Shutting Down` describe a simulator that is already on its
way, and a listing that failed, omits the simulator, or names no state proves nothing. A
canceled request is terminal and never reaches the probe; a probe that is itself canceled
travels on; every other probe failure answers `false` and leaves the capture's own error,
details and hint, as the reported failure. The boot memo policy is unchanged, so a simulator
turned off outside this process is still not seen until the memo expires.

Refs #4442
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.28 MB 6.28 MB +939 B
Package (unpacked) 6.28 MB 6.28 MB +939 B
Package (download) 1.90 MB 1.90 MB +425 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 22.7 ms 23.0 ms +0.3 ms
CLI --help 45.6 ms 48.2 ms +2.6 ms

@thymikee

Copy link
Copy Markdown
Member Author

Coverage is base-blocked at 855aa4a, before coverage thresholds: host-process-liveness.test.ts:182 fails in the fleet identity case. Expected fabricated PID 202 to have the fixture's start time and zombie:true; Linux returned a real /proc/202 identity and zombie:false.

The same test, location and assertion already fail in unchanged-main CI at d78be6c8, with the same Node 24.13 and coverage command. The source and test are byte-identical between that base and this head. This PR's failed job is retained as failed.

The fixture mocks ps but leaves /proc live, so a real process can replace its synthetic facts. I accept ownership of a bounded fixture-isolation repair, retaining all assertions and the dedicated Linux /proc tests. The patch is prepared but unrun; #4426 currently owns local execution. It will need focused proof and a new exact-head gate before a push, followed by Linux CI. No CI retry, test deletion or timeout change was made. Native recovery and real code-60 evidence remain separate draft blockers.

@thymikee

Copy link
Copy Markdown
Member Author

Pushed the fixture correction as 50034fe. All existing identity, zombie, command, ordering and call-count assertions remain; the dedicated Linux /proc tests retain their own overrides.

Regression proof in an isolated simulated Linux process/filesystem boundary, with the original assertions unchanged: corrected fixture 21 passed; removing only the seven-line isolation 1 failed / 20 passed, at the exact PID-202 start-time/zombie assertion from the failed Coverage jobs; restored fixture 21 passed. The harness was removed and source hashes restored. This models the boundary; it is not a Linux-host run.

One pinned pnpm 11.17.0 exact affected gate passed at 5003: exactCommit=true, checkoutChanged=false, failOpen=false. Eight checks passed, including 19,824 unit tests / 10 skipped and 377 related tests. Provider integration was deduplicated; coverage and replay checks were skipped, not passed. Independent review found no small-delta source findings, with unread policy/evidence inputs disclosed.

Real Linux CI is now running on the pushed fix. The original 855a Coverage job remains failed; this was a source correction, not a retry of unchanged CI. The PR stays draft: native Shutdown recovery and the real code-60 header remain unproven.

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the typed timeout work. At 855aa4a I found one likely regression in the new retry rule, and the live evidence the PR body says is missing is still missing.

SIMCTL_ERROR_HEADER only matches at the start of stderr, and a test fixes "header after a diagnostic preamble" as not recognized. The failure this retry exists for (#167, still visible in main's screenshot-simulator-capture.test.ts) has simctl print Detected file type from extension: PNG on stderr before the An error was encountered processing the command (domain=NSPOSIXErrorDomain, code=60) header. If that ordering is real, no reason is stamped, shouldRetryIosSimulatorScreenshot returns false, and the capture fails on the first attempt with no runner fallback. Before this PR the same failure retried and then fell back to the runner. The PR moved the PNG line to stdout in the existing fixtures, so no test covers the stderr ordering. Could the producer recognize the top-level header as an unindented line anywhere in stderr (a multiline anchor), while still excluding indented Underlying error lines? Please add a protocol test with the preamble ordering on stderr. I could not confirm real simctl output, so if it writes that line to stdout this does not apply.

This change rewires native screenshot retry, runner fallback and skipped-boot recovery on the local simulator route, and every test feeds hand-written stderr. If the typed reason never fires on real hardware, the retries disappear and the tests still pass. Please run once on an owned simulator. First, capture with skipIosSimulatorBootCheck against a simulator that was shut down after its boot memo was written, and show a probe returning Shutdown, one ensureBooted, one re-capture and a written PNG. Second, attach the raw stderr of a real xcrun simctl io <udid> screenshot failure (ideally code=60) with the stamped details.reason, or the ios_simulator_screenshot retry diagnostics with attempt greater than 1. That also settles the header position question above.

Not blocking, take or leave: shouldRetryIosSimulatorScreenshot is also the runner fallback predicate and runs on errors from captureAndNormalize, so any exec timeout in density normalization (such as the scale probe) now selects the runner fallback; scoping it to the capture step would keep the old behavior. The doc blocks in tool-provider.ts could each shrink to one line, and the runLocalSimctlCommand link names SIMCTL_SCREENSHOT_TIMEOUT_CODE where it means SIMCTL_SCREENSHOT_TIMEOUT_REASON. The four new test files add about 1,000 lines, so folding the retry-policy cases into the protocol suite with one negative table would keep it smaller.

The Coverage failure looks unrelated: the diff only touches packages/platform-apple, and the failing test named in the PR is a host-kit Linux /proc fixture. I could not see the failing assertion in the log, so that is not confirmed. No conflicts. Before merge, the header match needs to handle a stderr preamble, the simulator evidence above needs to be attached, and Coverage needs to pass once the host-kit fixture is repaired on main.

This review covers 855aa4a. The newer head 50034fe only adds a host-kit liveness test fixture, which does not touch the findings above and may address the Coverage failure.

@thymikee

Copy link
Copy Markdown
Member Author

I own #4544 and am addressing this review.

The preamble finding is accepted. Local commit cec1726bb03383a3c678d8ea7313713280a5412e recognizes the first unindented protocol header anywhere on stderr. The new positive cases put Detected file type from extension: PNG before the timeout header, with LF and CRLF. Indented/nested/stdout-only headers remain negative; an earlier unrelated top-level error is not skipped to select a later timeout. The comment blocks and incorrect reason link are also corrected. Regression discrimination and the new exact gate are assigned; this commit is not pushed yet.

The native requests remain open. The earlier ordinary Settings screenshot did not prove recovery or a real timeout header. I am checking a deterministic supported T3-bound route before another owned-simulator attempt, including whether a running batch can retain the session across T3 shutdown without admission booting first. I will attach actual command/probe/capture evidence and raw adapter stderr if reachable, and will not substitute scripted stderr for it.

The separate Linux process-fixture correction is already pushed at 50034fe3, with its passing exact gate and isolated regression evidence in the previous comment. The size-report edit is informational. The PR remains a blocked draft, with no merge planned.

@thymikee

Copy link
Copy Markdown
Member Author

The source trace corrected my earlier explanation: an open-session screenshot does not run daemon device readiness, so admission is not what blocked the previous attempt. That attempt still provides no recovery proof—the returned CLI file was removed before the screenshot could be sent.

A supported route is now planned: start one bound batch with a duration wait and screenshot, then use T3 shutdown while that client is already running. Whether T3 preserves that client and private session is an assumption the single attempt must verify.

Successful recovery currently discards the first error and records no probe state. I recorded the bounded observation decision to preserve the observed state, original stderr and a separate typed failure reason in the existing redacted diagnostics, plus retry-reason telemetry. This changes evidence, not recovery policy. The patch is prepared but remains unapplied while the current CEC gate owns the frozen checkout. It will be tested and reviewed before the native run.

A natural non-60 failure will remain unstamped; I will not present it as code-60 evidence or close that request without evidence or a waiver.

@thymikee

Copy link
Copy Markdown
Member Author

Pushed review-round changes at b0cc374 include: the PNG-stderr-preamble correction plus diagnostics needed to observe recovery. The parser matches the first unindented protocol header anywhere on stderr; nested/indented/stdout-only and earlier-unrelated-header controls remain. Removing /m fails both new LF/CRLF positives. Removing the new observation or retry-reason callback fails the intended assertions. Independent source reviews found no findings within their documented bounds.

Exact local validation: one pinned pnpm 11.17.0 affected gate at b0cc passed, exactCommit=true, checkoutChanged=false, failOpen=false. Eight checks passed; unit 19,830 passed / 10 skipped, related 377 passed. Provider integration was deduplicated; coverage and three replay entries skipped. Real Linux CI was green at 50034fe; b0cc needs its own CI.

One owned native run, 2026-11-14, iPhone 17 Pro / iOS 27.0, candidate daemon 0.21.38-dev, codeOrigin: checkout, signature graph:124:b72e1956bbfcd453fd5b0d395d873e100f773629. All device commands used the T3-returned CLI plus a private config. Its private session/request logs prove the candidate served the run. Settings control and selector passed first.

The already-running bound batch waited 15 seconds, then took a screenshot. T3 shutdown returned during the wait and removed the CLI file, but the running client/session survived. Batch exited 0: 2/2 steps, 45,726 ms, screenshot step 30,718 ms, within the 60-second work budget. The memo had expired: the control completed >30 seconds before the failing capture.

Screenshot request 6e2a67a82db92103, UTC:

Time Observed event
13:38:59.125–13:39:19.131 First capture hit host-kit's 20,000 ms deadline and was killed with SIGKILL.
13:39:19.322 observedState: "Shutdown", recoveryAuthorized: true, after one fresh 5,000 ms-capped probe.
13:39:28.174 Boot recovery phase succeeded on attempt 1 of 3.
13:39:29.423 One re-capture succeeded, followed by request/batch success.

recovered.png: 402×874 RGBA PNG, 534,741 bytes, SHA256 84af709dd161909267b05b76a37c8e00614ffcf19970dddadb82674d1783ff4c. I inspected it: freshly booted home screen; the subsequent T3 screenshot matches. Boot count is from the owner’s retry phase; exec trace records basenames, not each boot subcommand argv.

Actual failure diagnostic excerpt (existing redaction, bounds and outer-whitespace normalization apply; not byte-exact stderr):

Detected file type from extension: PNG
Note: No display specified. Defaulting to display: 41DF04CA-7262-439C-832A-CFB80894071C (screenID: 1, name: LCD)

The event has reason: "xcrun timed out after 20000ms", timeoutMs: 20000. captureFailureReason is absent, and retry diagnostics have no reason key. This is a real host-owned deadline failure, not simctl NSPOSIXErrorDomain code 60. It confirms the PNG preamble is on stderr, but no real error header, code-60 stamp or retry attempt >1 occurred. The existing exhausted.attempt:5 field reports the configured maximum; it is not evidence that five attempts ran.

The requested Shutdown → observed probe → one boot recovery → one re-capture → PNG sequence is proven. The real simctl code-60/stamped-reason or actual attempt>1 item remains unobserved, not resolved by fixtures. Please keep that item open, or explicitly confirm whether this native host-timeout proof plus the adapter protocol regression tests satisfies the review requirement; I will not force a timeout or fabricate native output.

Cleanup verified: session closed with shutdown; private daemon stopped gracefully, PID gone; config/token/state removed; token scan clean; assigned simulator stopped and T3 open list empty. Pre-dispatch authentication/quoting mistakes were excluded from the one recovery attempt. The checkout is clean; no merge.

@thymikee

Copy link
Copy Markdown
Member Author

This PR is ready at b0cc374: I found nothing that must change in the code. CI is green, with 0 of 18 checks failing, and the earlier Coverage failure is fixed by the host-process-liveness test isolation in this diff. There are no conflicts. No real simctl code 60 failure was observed, so the header format rests on scripted stderr and simctl's known format. If a real stall prints 'Timeout waiting for screen surfaces' under another domain, or with no header, it now ends the run and skips the retry and runner fallback. #4442 asks for that, but we do not know how often it happens, so a maintainer should accept this explicitly. Before merge, please take the PR out of draft and have a maintainer accept that gap. Not blocking, and fine to take or leave: (1) the fallback check at https://github.com/callstack/agent-device/blob/b0cc374/packages/platform-apple/src/core/screenshot.ts#L213 now also runs on errors from normalizeDensity, so a timeout in the scale probe picks the runner re-capture; should it only apply to the capture step's error? (2) The fake-xcrun tests, such as screenshot-simulator-capture.test.ts line 100, return a pre-stamped AppError, so they no longer prove stderr leads to a retry; screenshot-simctl-retry.test.ts covers that path. (3) The long design comments in screenshot.ts, simulator.ts, config.ts and the test fixtures could shrink to one line each. (4) The new test files overlap, since the retry-policy negatives repeat the protocol negatives and the protocol positives overlap the simctl-retry positives; could they fold into one table?

@thymikee
thymikee marked this pull request as ready for review November 14, 2026 14:55
@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 14, 2026
@thymikee

thymikee commented Nov 14, 2026 •

Copy link
Copy Markdown
Member Author

Marked ready for review at b0cc374 after the clean review above. The existing CI completed green, and there are no inline review threads. Marking the PR ready started fresh iOS/macOS smoke checks and automated reviews; the existing watch stays active until they settle.

The owned Shutdown recovery sequence is proven. A real simctl code-60 header/stamped reason or actual retry above attempt 1 remains unobserved; that gap still needs explicit maintainer acceptance before merge. The PR description now names this as a maintainer decision rather than a draft blocker. The optional cleanup suggestions remain optional. No merge was performed.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 15 files

You’re at about 91% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 1 new · 0 still open · 0 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
packages/platform-apple/src/core/screenshot.ts:497 🆕 Boot recovery reads cached Booted state, so the re-capture is guaranteed to fail

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of b0cc3743 · 14.11.2026 15:04 UTC

Apex Sentinel · ✅ Approve

No open findings.

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on d4bc055. The verdict and the full list are in the summary comment.

@thymikee

Copy link
Copy Markdown
Member Author

I found no problems in d4bc055. The earlier review of b0cc374 found none either, and the changes since then only touch screenshot diagnostics and tests in platform-apple.

All 20 checks pass, and there are no conflicts. I read the new diagnostic assertions but did not re-run the focused test file or your mutation. I also did not trace the diagnostic redaction into the shared emit path, and I relied on the PR body for the native recovery run on #4442. A real simctl code-60 header, or a retry above attempt 1, is still unobserved on a device, as the PR body says.

The fixed items are the devices check in isSimctlListDevices (#4544 (comment)), the readSimulatorBootState doc comments (#4544 (comment) and #4544 (comment)), the deviceId field in the skipped-recovery diagnostic (#4544 (comment)), and the probe-failure event (#4544 (comment)). You can resolve those threads. You can also resolve #4544 (comment): the runner fallback uses the typed-timeout predicate by design, and the density probe is not part of that decision.

The boot-memo thread still stands (#4544 (comment)). A capture that fails within 5s of the memo being seeded is probed as Shutdown, so the boot is skipped and the retry repeats the failure. The typed-timeout route outlives that memo, so I don't see it as blocking this PR.

Before merge, the maintainer decides whether to accept that disclosed gap and the unobserved code-60 header.

@thymikee
thymikee merged commit 3f9ee58 into main Nov 15, 2026
20 checks passed
@thymikee
thymikee deleted the t3/audit-4442-test-gates branch November 15, 2026 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use typed Apple screenshot recovery reasons

1 participant