Skip to content

feat(recovery): bound remote health and runner-busy recovery by the command's deadline - #4541

Merged
thymikee merged 9 commits into
mainfrom
feat/deadline-aware-read-recovery
Nov 15, 2026
Merged

thymikee merged 9 commits into
mainfrom
feat/deadline-aware-read-recovery

Conversation

@thymikee

@thymikee thymikee commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Before, callers wrapped read-only commands in their own retry loops because recovery used fixed counts. This PR bounds two recovery paths by the command's own deadline. 15 files.

  • Remote /health gate (daemon-client): when the probe before the first send gets a 5xx or times out, it probes again every second, for up to the command's request timeout. Nothing was sent, so this is safe for UI actions too. A refused connection still fails at once, so a URL with nothing listening never waits. The wait does not shorten the dispatched command's budget, because a command sent with too little time left could time out after reaching the device. remote_daemon_unavailable now reports details.attempts.
  • Apple RUNNER_BUSY (apple-runner): read-only commands resend until 2 s before the request deadline (getRequestDeadlineMs, added to the runner host port), instead of 8 attempts (≈5.4 s). Transport resends keep 3 attempts. Mutations are sent once. The final refusal reports details.attempts. A busy read now takes up to about 58 s to fail instead of about 5 s.

Not in this PR (follow-ups for #4522):

  • Android: a typed device not found observation and deadline-bounded waits after boot or helper install. This stays after Drive Android boot polling with typed observations instead of error text #4426, which owns the typed adb boot observations (still unpublished).
  • The re-probe on the instance-mismatch restart retry.
  • Known edge: inside batch/replay (60 s execution deadline), a busy resend the runner executes in the last ~1 s can end as the request-deadline error without attempts. Plain reads are unaffected.

Part of #4522 (not closing: Android remains). Related: #4426.

Validation

At head 43fd323c2a: focused vitest, all passing: apple-runner project (963), daemon-client remote-health/index/explicit-lifecycle/transport + CLI refusal (70), platform-apple core + test-kit (467). pnpm typecheck, pnpm format, and oxlint are clean. Mutation check: reverting each production change fails the new tests. CI owns check:affected. No device run: these are host-side retry decisions, tested at each owner without devices.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

View guided diff Turn on auto-fix

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.29 MB 6.29 MB +1.9 kB
Package (unpacked) 6.29 MB 6.29 MB +1.9 kB
Package (download) 1.91 MB 1.91 MB +924 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 22.1 ms 21.3 ms -0.8 ms
CLI --help 43.2 ms 43.5 ms +0.3 ms

@blacksmith-sh

This comment has been minimized.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 14 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread website/docs/docs/commands.md Resolved
@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (7848a1a9283f).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of ce1cead8 · 14.11.2026 12:34 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

2 findings below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

Review of 9ea971bb · 14.11.2026 12:14 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

Review of 6ef40ee1 · 14.11.2026 11:49 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

Review of 3ec9ada0 · 14.11.2026 11:33 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 1 no longer raised

No open findings.

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

Review of 43fd323c · 14.11.2026 11:15 UTC

Apex Sentinel · ✅ Approve

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
packages/daemon-client/src/daemon-client-remote-health-recovery.ts:15 A dead remote daemon now hangs every command up to its full request timeout

2 findings below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on 43fd323. The verdict and the full list are in the summary comment.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files (changes from recent commits).

Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

View guided diff | Turn on auto-fix | Re-trigger cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed 6ef40ee. One change is needed before this can merge: the new deadline window for runner-busy resends also applies to read-only steps inside mutating actions.

In packages/apple-runner/src/index.ts:178, resolveReadOnlyResendBudget keys the window on the runner command, but getRequestDeadlineMs returns the deadline of the whole request. A swipe or scroll calls gestureViewport first, and a selector press calls querySelector. Both are read-only runner commands sent under the action's request id. A RUNNER_BUSY drain on that step is now resent until 2s before the action's deadline (before: 8 attempts, about 5.4s), and the gesture is then dispatched with 2s or less left. That action can reach the device and then hit the deadline, so it reports dispatched unknown where it used to fail fast with RUNNER_BUSY dispatched no. This is the hazard the PR cites for not shortening the remote send, and it widens the "read-only commands" scope of #4522 to reads inside UI actions. The rule: a deadline-bounded busy window may spend only budget that no later dispatch in the same request needs. The smallest fix is to apply the window only when the request itself is a read-only command (pass that fact in AppleRunnerCommandOptions from the daemon dispatch, or reserve the mutation's budget) and keep the 8-attempt count for read-only steps inside mutating requests. Please add a test with a busy gestureViewport step under a 30s deadline. I read the code paths and did not run them. The dispatched-unknown outcome depends on gesture time against the remaining 2s, so the mechanism is confirmed but the end-to-end result is not reproduced. I also did not trace read-only runner commands beyond gestureViewport and querySelector.

Smoke Tests (ios), Smoke Tests (macos) and one review check are still in progress, not failed. The diff is on their route, since every read-only runner command now goes through resolveReadOnlyResendBudget and the new appleRunnerHost wiring, so please wait for Smoke Tests (ios) to pass at 6ef40ee. A live forced RUNNER_BUSY is not needed. The busy branch is covered at its owner with the real retryWithPolicy. The smoke jobs use a local daemon, so they do not exercise the remote gate. There are no conflicts. The next step is to scope the Apple busy window to read-only requests, then get a green iOS smoke run at the head.

Not blocking, take or leave: the loop in daemon-client-remote-health-recovery.ts:38 re-implements host-kit pollUntilSettled; retryWithPolicy at apple-runner/src/index.ts:180 already accepts a deadline, so the attempt arithmetic and the 2x delay reserve could go; at daemon-client/src/index.ts:111 the health gate and the send each get the full requestTimeoutMs, so wall time can reach about twice the envelope unless you share one deadline or state that bound in the docs; no test aborts the signal while the gate re-probes (lifecycle-remote-health test); commands.md:595 says it probes every second, but a timed-out probe takes up to 3s first, and healthRecoveryBudgetMs is undocumented; and the health cache in daemon-client-remote-health-cache.ts:44 has a bounded/unbounded split whose in-flight sharing serves only the gate without a request budget.

Could this be simpler? The remote gate could be pollUntilSettled over the cached health probe, with a Deadline created before the first probe and settled = reachable or non-transient, caching only settled answers. The Apple side could pass the deadline to retryWithPolicy and apply it only for read-only requests, keeping the 8-attempt count inside actions. That would remove ReadOnlyResendBudget, the attempt arithmetic, the in-flight cache mode and the hand-rolled loop. What would have to change first is that AppleRunnerCommandOptions needs the request's read-only fact, or a reserved mutation budget.

The inline thread on a refused connection no longer applies, because a refused connection is non-transient and refuses at once with attempts 1 (covered by the "is refused at once" test), while a blackholed host waits the budget by design as documented in commands.md. Please resolve it: #4541 (comment)

@thymikee

Copy link
Copy Markdown
Member Author

Thanks. Addressed in 9ea971b.

Blocking: the busy window inside actions. Fixed with the fact passed from daemon dispatch, as you suggested:

  • The transports now stamp readOnly: isReadOnlyRequest(req) on each admitted request (packages/daemon/src/core/daemon-command-registry.ts). It reads the descriptor's recording effect, the same classification isHumanControlMutation already used, which now delegates to it.
  • host-kit's new getRequestReadOnlyDeadlineMs returns the request deadline only for a read-only request, and an attempt registered under a parent inherits the parent's fact.
  • The Apple runner reads that through its host port instead of getRequestDeadlineMs. So gestureViewport before a swipe and querySelector before a press keep the 8-attempt window, and the rest of the budget stays with the action.
  • Tests:
    • "a busy read inside a mutating request keeps the fixed window under a 30s deadline": a busy gestureViewport gives 8 sends, RUNNER_BUSY, dispatched: no, attempts: 8.
    • host-kit: a read-only request, its child attempt, and a mutating request with the same deadline (the last answers undefined).
    • daemon registry: read/action classification pairs.

Could this be simpler? Partly yes. Taken:

  • The health cache now holds only settled usable answers. The in-flight sharing mode is gone, and each caller probes under its own bound.
  • New test that aborts the signal while the gate re-probes: it answers as a cancellation, and no probe follows.
  • commands.md now gives the probe timing (a probe that gets no answer takes up to 3 s before the 1 s wait), documents healthRecoveryBudgetMs, and states the bound: up to about twice the command timeout while the budget decision is pending.

Kept, with reasons:

  • retryWithPolicy's deadline on the Apple side. That option clamps every retry delay, transport resends included. Near a deadline, a fetch failed would then get one attempt instead of three. An earlier review found this (round 1, bug: fill on search does not always work #4), and a test pins it: "under a request deadline too close for a busy resend, transport failures keep their three attempts". So the busy window stays in shouldRetry. The attempt bound is only there because retryWithPolicy needs a finite maxAttempts.
  • pollUntilSettled for the gate. It always takes its first sample at once, but the gate has already taken that sample before deciding to recover. Using it would add an immediate duplicate probe, or move the hot-path probe into the lazily loaded recovery module. The loop is about 15 lines, and I'd keep it unless you prefer the duplicate probe.

Pending: whether the /health wait counts against the command's own timeout is still waiting on your decision, and I haven't changed it. Resolving the Apex thread as you asked.

Local at 9ea971b:

  • focused vitest: daemon-client + daemon registry + host-kit + daemon server + CLI refusal 1643 passing; apple-runner 965 passing.
  • check:layering, check:daemon-wire-compat, eager-closure-budgets, typecheck and lint are clean.
  • CI is running.

@thymikee

Copy link
Copy Markdown
Member Author

@cubic-dev-ai review

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review

@thymikee I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 24 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread website/docs/docs/commands.md Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files (changes from recent commits).

You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@thymikee

Copy link
Copy Markdown
Member Author

This PR is ready at ce1cead. The code review found no problems, and the findings from the earlier review at 6ef40ee (#4541 (comment)) are fixed. All 22 checks pass at this commit, including the iOS smoke run on the changed apple-runner resend route that the earlier review asked for. Nothing else must happen before merge.

Not blocking, and you can take or leave these: no test admits a request through a transport and reads the stamped read-only fact, so changing readOnly: isReadOnlyRequest(req) to readOnly: true at transport.ts:97 would keep every test green, and one test that admits swipe up and snapshot with the same budget would close that gap. The sentence at commands.md:461 says a read-only command resends until shortly before its own timeout, but a read run as a step of batch, replay or test keeps the fixed 8-send window, so it could say so, and the PR body still describes the old batch edge, the old host-port name, 15 files and validation at 43fd323. isHumanControlMutation could be written as its exemptions plus !isReadOnlyRequest(req) at daemon-command-registry.ts:121, since both resolve the same observe/mutate split.

None of the earlier inline threads still apply. These are fixed at this commit and can be resolved: the observability branch is gone (#4541 (comment)); the generation counter rejects earlier probes (#4541 (comment)); the busy-resend test comment now says 8 sends (#4541 (comment)); commands.md now says at most 8 sends (#4541 (comment)); the recovery-budget test loop is bounded and asserts at least 3 probes (#4541 (comment)); and the registry doc no longer claims a special case (#4541 (comment)). This one is benign and can be resolved: a refused connection fails at once, and only a 5xx or timed-out probe waits the budget, as commands.md:595 documents (#4541 (comment)).

I did not run any test; I read the code at head and relied on the 22/0 check count, and I did not see the iOS smoke job by name. I did not trace every read-only request for a mutating runner command sent after a long busy read in the same request. I found no such route for snapshot, screenshot, alert get or find read actions, but that search was not exhaustive. I did not re-read daemon-client/src/index.ts for the twice-the-timeout bound, because this change does not touch it.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 14, 2026
@thymikee

Copy link
Copy Markdown
Member Author

This branch now conflicts with main at ce1cead. Please rebase onto main and push; the earlier clean review still applies to the code, and I will re-check the rebased head.

@thymikee thymikee removed the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 15, 2026
thymikee and others added 9 commits November 15, 2026 09:52
…mmand's deadline

A remote `/health` gate that finds a daemon there but not ready (a 5xx answer or a probe
timeout) now probes again every second for the command's own request budget before it
refuses, since nothing was sent while the gate is closed. A connection refused outright
still refuses at once. The refusal reports `details.attempts`.

A read-only Apple runner command refused with RUNNER_BUSY is now resent until shortly before
the request's own deadline instead of a fixed eight attempts, and the final refusal reports
`details.attempts`. Mutations are still sent once. The runner host port gains
getRequestDeadlineMs to read that deadline.

Refs #4522

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Transport resends keep their three attempts near a deadline, and the attempt bound stays
finite in retry diagnostics. The docs state that the remote health wait leaves the dispatched
command its full timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on a refused probe

- The first /health probe runs under the command's budget and signal, and the recovery
  deadline starts with it.
- The re-probe loop stops at once when a re-probe is not transient (a refused connection).
- Concurrent callers share one in-flight probe, but each applies its own timeout and abort to
  its wait. The cache moves to daemon-client-remote-health-cache.ts.
- remote_daemon_unavailable always carries attempts, and healthRecoveryBudgetMs when the
  budget ran out.
- Tests: busy refusal after the window closed, refused re-probe, shared-probe abort, and
  stricter dispatched-press assertions. Docs: a closed busy window still answers runner_busy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lify the health cache

- RemoteDaemonHealth and RemoteDaemonHealthLink are declared in daemon-client-transport.ts
  again, where the wire-compat manifest expects them.
- The remote health cache module loads lazily on the remote path, so the CLI and client entries
  evaluate no extra module.
- The cache shares one in-flight probe among unbounded callers, as on main. A bounded caller
  reads only a settled answer, or runs its own probe under its own timeout and signal. It never
  joins another caller's probe, so a stale probe cannot spend its budget, nothing is left
  unobserved, and an aborted or budgetless caller sends no request.
- The budget test asserts 90 probes exactly one second apart on the faked clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed local request no longer loads the remote health cache: invalidation returns at once
when the daemon has no base URL. A transport test pins both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elf is read-only

A read-only runner step inside an action (the viewport read before a swipe, the selector query
before a press) runs under the action's request id, so the busy window could spend the budget
the action needs and turn a fast RUNNER_BUSY into a dispatch with no time left. The daemon now
stamps whether each admitted request is read-only (isReadOnlyRequest, from the descriptor's
recording effect), host-kit answers getRequestReadOnlyDeadlineMs only for such requests, and the
runner reads that through its host port. Reads inside actions keep the 8-attempt window.

The remote health cache now holds only settled usable answers; each caller probes under its own
bound. Adds an abort-while-re-probing test and documents the probe timing, the wall-time bound
and healthRecoveryBudgetMs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rd stale health writes

- isReadOnlyRequest no longer counts observability commands (logs start/stop/mark, logs clear
  --restart, audio probe start/stop) as reads, since they change state; isHumanControlMutation
  keeps its original classification.
- A /health answer that lands after an invalidation is returned but not cached.
- Wording: a busy read inside an action is sent at most 8 times in total; the budget test's wait
  loop is bounded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… effect

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee

Copy link
Copy Markdown
Member Author

Rebased onto main at 7848a1a. New head: 0e2a0c6, pushed with --force-with-lease from ce1cead.

Conflicts: only website/docs/docs/commands.md, on the iOS runner refusals line. The PR's commits rewrite its runner_busy clause, and main changed the same line elsewhere. I kept main's line and dropped in the PR's final runner_busy wording at each step. That clause is now byte-identical to the one at ce1cead, and the docs diff against main is still only the PR's two lines.

No other file conflicted. git diff origin/main..HEAD outside website/ is identical to the PR's diff before the rebase, so behavior is unchanged.

Re-run at the new head:

  • check:layering, check:daemon-wire-compat and typecheck are clean.
  • eager-closure-budgets, wire-compat and wire-mutations: 954 passing.
  • Focused daemon-client, daemon registry, host-kit request-cancel and CLI refusal tests: 366 passing.
  • Apple busy-resend tests: 13 passing.

The /health budget decision is still on hold. Not merging.

@thymikee
thymikee force-pushed the feat/deadline-aware-read-recovery branch from ce1cead to 0e2a0c6 Compare November 15, 2026 08:55
@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the rebase. The new head 0e2a0c6 has the same code as the reviewed ce1cead. The only change is the resolved conflict on the iOS runner refusals line in website/docs/docs/commands.md, which keeps main's text and the PR's runner_busy wording. The earlier clean review still applies. The checks pass and there are no conflicts.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 15, 2026
@thymikee
thymikee merged commit e974e1b into main Nov 15, 2026
26 of 27 checks passed
@thymikee
thymikee deleted the feat/deadline-aware-read-recovery branch November 15, 2026 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant