Repository navigation
feat(recovery): bound remote health and runner-busy recovery by the command's deadline - #4541
Conversation
Size Report
Startup median (7 runs, lower is better):
|
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
All reported issues were addressed across 14 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
Apex Sentinel · ✅ ApproveSince the last review: 0 new · 0 still open · 0 no longer raised No open findings. 1 finding below 💬 Talking to SentinelIn a finding's thread, reply with:
On this conversation, comment Earlier reviewsReview of
|
| Where | Finding |
|---|---|
packages/daemon-client/src/daemon-client-remote-health-recovery.ts:15 |
A dead remote daemon now hangs every command up to its full request timeout |
2 findings below minor hidden (user_settings.min_severity).
Reviewed with this repository's own Sentinel configuration (d78be6c88715).
There was a problem hiding this comment.
Apex Sentinel: 1 new finding on 43fd323. The verdict and the full list are in the summary comment.
There was a problem hiding this comment.
All reported issues were addressed across 10 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 10 files (changes from recent commits).
Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
View guided diff | Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
|
Reviewed 6ef40ee. One change is needed before this can merge: the new deadline window for runner-busy resends also applies to read-only steps inside mutating actions. In packages/apple-runner/src/index.ts:178, resolveReadOnlyResendBudget keys the window on the runner command, but getRequestDeadlineMs returns the deadline of the whole request. A swipe or scroll calls gestureViewport first, and a selector press calls querySelector. Both are read-only runner commands sent under the action's request id. A RUNNER_BUSY drain on that step is now resent until 2s before the action's deadline (before: 8 attempts, about 5.4s), and the gesture is then dispatched with 2s or less left. That action can reach the device and then hit the deadline, so it reports dispatched unknown where it used to fail fast with RUNNER_BUSY dispatched no. This is the hazard the PR cites for not shortening the remote send, and it widens the "read-only commands" scope of #4522 to reads inside UI actions. The rule: a deadline-bounded busy window may spend only budget that no later dispatch in the same request needs. The smallest fix is to apply the window only when the request itself is a read-only command (pass that fact in AppleRunnerCommandOptions from the daemon dispatch, or reserve the mutation's budget) and keep the 8-attempt count for read-only steps inside mutating requests. Please add a test with a busy gestureViewport step under a 30s deadline. I read the code paths and did not run them. The dispatched-unknown outcome depends on gesture time against the remaining 2s, so the mechanism is confirmed but the end-to-end result is not reproduced. I also did not trace read-only runner commands beyond gestureViewport and querySelector. Smoke Tests (ios), Smoke Tests (macos) and one review check are still in progress, not failed. The diff is on their route, since every read-only runner command now goes through resolveReadOnlyResendBudget and the new appleRunnerHost wiring, so please wait for Smoke Tests (ios) to pass at 6ef40ee. A live forced RUNNER_BUSY is not needed. The busy branch is covered at its owner with the real retryWithPolicy. The smoke jobs use a local daemon, so they do not exercise the remote gate. There are no conflicts. The next step is to scope the Apple busy window to read-only requests, then get a green iOS smoke run at the head. Not blocking, take or leave: the loop in daemon-client-remote-health-recovery.ts:38 re-implements host-kit pollUntilSettled; retryWithPolicy at apple-runner/src/index.ts:180 already accepts a deadline, so the attempt arithmetic and the 2x delay reserve could go; at daemon-client/src/index.ts:111 the health gate and the send each get the full requestTimeoutMs, so wall time can reach about twice the envelope unless you share one deadline or state that bound in the docs; no test aborts the signal while the gate re-probes (lifecycle-remote-health test); commands.md:595 says it probes every second, but a timed-out probe takes up to 3s first, and healthRecoveryBudgetMs is undocumented; and the health cache in daemon-client-remote-health-cache.ts:44 has a bounded/unbounded split whose in-flight sharing serves only the gate without a request budget. Could this be simpler? The remote gate could be pollUntilSettled over the cached health probe, with a Deadline created before the first probe and settled = reachable or non-transient, caching only settled answers. The Apple side could pass the deadline to retryWithPolicy and apply it only for read-only requests, keeping the 8-attempt count inside actions. That would remove ReadOnlyResendBudget, the attempt arithmetic, the in-flight cache mode and the hand-rolled loop. What would have to change first is that AppleRunnerCommandOptions needs the request's read-only fact, or a reserved mutation budget. The inline thread on a refused connection no longer applies, because a refused connection is non-transient and refuses at once with attempts 1 (covered by the "is refused at once" test), while a blackholed host waits the budget by design as documented in commands.md. Please resolve it: #4541 (comment) |
|
Thanks. Addressed in 9ea971b. Blocking: the busy window inside actions. Fixed with the fact passed from daemon dispatch, as you suggested:
Could this be simpler? Partly yes. Taken:
Kept, with reasons:
Pending: whether the Local at 9ea971b:
|
|
@cubic-dev-ai review |
@thymikee I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 24 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 7 files (changes from recent commits).
You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
|
This PR is ready at ce1cead. The code review found no problems, and the findings from the earlier review at 6ef40ee (#4541 (comment)) are fixed. All 22 checks pass at this commit, including the iOS smoke run on the changed apple-runner resend route that the earlier review asked for. Nothing else must happen before merge. Not blocking, and you can take or leave these: no test admits a request through a transport and reads the stamped read-only fact, so changing None of the earlier inline threads still apply. These are fixed at this commit and can be resolved: the observability branch is gone (#4541 (comment)); the generation counter rejects earlier probes (#4541 (comment)); the busy-resend test comment now says 8 sends (#4541 (comment)); commands.md now says at most 8 sends (#4541 (comment)); the recovery-budget test loop is bounded and asserts at least 3 probes (#4541 (comment)); and the registry doc no longer claims a special case (#4541 (comment)). This one is benign and can be resolved: a refused connection fails at once, and only a 5xx or timed-out probe waits the budget, as commands.md:595 documents (#4541 (comment)). I did not run any test; I read the code at head and relied on the 22/0 check count, and I did not see the iOS smoke job by name. I did not trace every read-only request for a mutating runner command sent after a long busy read in the same request. I found no such route for snapshot, screenshot, alert get or find read actions, but that search was not exhaustive. I did not re-read |
|
This branch now conflicts with main at ce1cead. Please rebase onto main and push; the earlier clean review still applies to the code, and I will re-check the rebased head. |
…mmand's deadline A remote `/health` gate that finds a daemon there but not ready (a 5xx answer or a probe timeout) now probes again every second for the command's own request budget before it refuses, since nothing was sent while the gate is closed. A connection refused outright still refuses at once. The refusal reports `details.attempts`. A read-only Apple runner command refused with RUNNER_BUSY is now resent until shortly before the request's own deadline instead of a fixed eight attempts, and the final refusal reports `details.attempts`. Mutations are still sent once. The runner host port gains getRequestDeadlineMs to read that deadline. Refs #4522 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Transport resends keep their three attempts near a deadline, and the attempt bound stays finite in retry diagnostics. The docs state that the remote health wait leaves the dispatched command its full timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on a refused probe - The first /health probe runs under the command's budget and signal, and the recovery deadline starts with it. - The re-probe loop stops at once when a re-probe is not transient (a refused connection). - Concurrent callers share one in-flight probe, but each applies its own timeout and abort to its wait. The cache moves to daemon-client-remote-health-cache.ts. - remote_daemon_unavailable always carries attempts, and healthRecoveryBudgetMs when the budget ran out. - Tests: busy refusal after the window closed, refused re-probe, shared-probe abort, and stricter dispatched-press assertions. Docs: a closed busy window still answers runner_busy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lify the health cache - RemoteDaemonHealth and RemoteDaemonHealthLink are declared in daemon-client-transport.ts again, where the wire-compat manifest expects them. - The remote health cache module loads lazily on the remote path, so the CLI and client entries evaluate no extra module. - The cache shares one in-flight probe among unbounded callers, as on main. A bounded caller reads only a settled answer, or runs its own probe under its own timeout and signal. It never joins another caller's probe, so a stale probe cannot spend its budget, nothing is left unobserved, and an aborted or budgetless caller sends no request. - The budget test asserts 90 probes exactly one second apart on the faked clock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed local request no longer loads the remote health cache: invalidation returns at once when the daemon has no base URL. A transport test pins both sides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elf is read-only A read-only runner step inside an action (the viewport read before a swipe, the selector query before a press) runs under the action's request id, so the busy window could spend the budget the action needs and turn a fast RUNNER_BUSY into a dispatch with no time left. The daemon now stamps whether each admitted request is read-only (isReadOnlyRequest, from the descriptor's recording effect), host-kit answers getRequestReadOnlyDeadlineMs only for such requests, and the runner reads that through its host port. Reads inside actions keep the 8-attempt window. The remote health cache now holds only settled usable answers; each caller probes under its own bound. Adds an abort-while-re-probing test and documents the probe timing, the wall-time bound and healthRecoveryBudgetMs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rd stale health writes - isReadOnlyRequest no longer counts observability commands (logs start/stop/mark, logs clear --restart, audio probe start/stop) as reads, since they change state; isHumanControlMutation keeps its original classification. - A /health answer that lands after an invalidation is returned but not cached. - Wording: a busy read inside an action is sent at most 8 times in total; the budget test's wait loop is bounded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… effect Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rebased onto main at 7848a1a. New head: 0e2a0c6, pushed with --force-with-lease from ce1cead. Conflicts: only No other file conflicted. Re-run at the new head:
The |
ce1cead to
0e2a0c6
Compare
|
Thanks for the rebase. The new head 0e2a0c6 has the same code as the reviewed ce1cead. The only change is the resolved conflict on the iOS runner refusals line in |
Summary
Before, callers wrapped read-only commands in their own retry loops because recovery used fixed counts. This PR bounds two recovery paths by the command's own deadline. 15 files.
/healthgate (daemon-client): when the probe before the first send gets a 5xx or times out, it probes again every second, for up to the command's request timeout. Nothing was sent, so this is safe for UI actions too. A refused connection still fails at once, so a URL with nothing listening never waits. The wait does not shorten the dispatched command's budget, because a command sent with too little time left could time out after reaching the device.remote_daemon_unavailablenow reportsdetails.attempts.RUNNER_BUSY(apple-runner): read-only commands resend until 2 s before the request deadline (getRequestDeadlineMs, added to the runner host port), instead of 8 attempts (≈5.4 s). Transport resends keep 3 attempts. Mutations are sent once. The final refusal reportsdetails.attempts. A busy read now takes up to about 58 s to fail instead of about 5 s.Not in this PR (follow-ups for #4522):
device not foundobservation and deadline-bounded waits after boot or helper install. This stays after Drive Android boot polling with typed observations instead of error text #4426, which owns the typed adb boot observations (still unpublished).batch/replay(60 s execution deadline), a busy resend the runner executes in the last ~1 s can end as the request-deadline error withoutattempts. Plain reads are unaffected.Part of #4522 (not closing: Android remains). Related: #4426.
Validation
At head
43fd323c2a: focused vitest, all passing: apple-runner project (963), daemon-client remote-health/index/explicit-lifecycle/transport + CLI refusal (70), platform-apple core + test-kit (467).pnpm typecheck,pnpm format, and oxlint are clean. Mutation check: reverting each production change fails the new tests. CI ownscheck:affected. No device run: these are host-side retry decisions, tested at each owner without devices.🤖 Generated with Claude Code
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.