Repository navigation
20260924 - Recognise the download failures the retry missed, and power cuts - #10
Merged
Merged
Conversation
…r cuts Replayed over all 45 historical failure logs, the classifier missed two real download failures. It now also retries "Update Module Download process timed out" (nightcrawler2, three hours on Wi-Fi) and a lost "stream-next" pipe to the module (Jonathan 1): 4 retries instead of 2, still none wrong. It also recognises a power cut mid-deployment: the client daemon starting again after the node was dark for 10 minutes or more (ret9573ecda and retc47d6f72, four and eight hours on 2026-09-24). A broken OS image reboots within about a minute and fails ArtifactVerifyReboot, so the gap keeps it out, and a failure logged before the gap is still the cause. Retrying these is stage 3 of the agreed rollout and is off unless DEPLOY_RETRY_INTERRUPTED=1; until then the verdict is reported. A download that gave up is checked before the power cut, so a node that went dark after its download failed (nightcrawler1, 44 hours) is still retried as a download failure. The service still runs without --apply, so this changes verdicts only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item 6 of ClickUp OTA deployments miss part of the fleet on every release (123zgec4tmx). Changes verdicts only: the service still runs without
--apply.Download failures it missed (stage 2 scope)
Replayed over all 45 historical failure logs, #9's classifier missed two real download failures. Added to the allowlist:
Update Module Download process timed out(nightcrawler2, owl-os v0.16.1, 2026-09-04: three hours of resumed reads on Wi-Fi)Cannot open .../stream-next(Jonathan 1, owl-os v0.8.20, 2026-06-18: the client lost its pipe to the module)Power cut mid-deployment (stage 3, behind a flag)
Recognised when the client daemon starts again after the node was dark for
DEPLOY_RETRY_INTERRUPT_GAP_MINUTES(10) or more, with no update step failing before the gap. A broken OS image reboots in about a minute and failsArtifactVerifyReboot, so the gap keeps it out. Off unlessDEPLOY_RETRY_INTERRUPTED=1; until then the journal reportspower lost mid-deployment (node dark N h); retrying these is not enabled.A download that gave up is checked first, so a node that went dark after its download failed (nightcrawler1, 44 h) is still a download retry. The first draft of this change got that wrong; the replay caught it and a test now pins it.
Replay over all 45 historical logs
No wrong retries in any mode. A deliberate 51 s sysrq power cut on Josh Test Node 2, a simulated broken OS image, and a failure followed by a power cut all stay refused.
Tests
46 pass (12 new, fixtures trimmed from real device logs); ruff clean.
🤖 Generated with Claude Code