Auto-retry parked shards once at phase end; add scan-rate timeline tab - #66
Merged
Merged
Conversation
Two independent UX improvements requested together:
1. advance()'s parked-shard guard used to hold every phase transition
open indefinitely the instant any shard parked, requiring operator
intervention even when the park cause was transient at the moment
it happened (a mount blip, a brief NFS hiccup) rather than
deterministic. On a long-running pass this could mean a job sitting
for hours waiting on a condition that had long since cleared.
advance() now calls store.RetryParkedByJob (resets attempt to 0,
so the retry gets the same fresh 5-attempt budget as a new shard)
once per pass before falling back to the original block-and-alert
behavior. Bounded to exactly one automatic round via
Controller.parkedAutoRetried (keyed by pass id, in-memory, same
"safe direction to be wrong in" restart semantics as the existing
parkedAlerted map) — an unbounded retry on a genuinely stuck shard
would just be an infinite loop wearing a different name.
checkParkedShards' operator alerting composes with this unchanged.
2. The "Aggregate throughput" panel now has a small tab toggle
("throughput" / "scan rate") that switches the same timeline canvas
between bwHist and the already-tracked-but-never-graphed scanHist,
via a new scanRate() formatter alongside the existing rate(). No
new data plumbing needed — scanHist already fed the small KPI-strip
sparkline; this just makes it available on the big auto-scaling
graph too. Choice persists across reloads like unitsMode does.
(A third requested item — bulk retry-all/drop-all buttons for parked
shards — turned out to already be fully implemented, both the
/api/v1/jobs/{name}/parked/retry|drop endpoints and the console's
job-scoped bulk buttons; no change needed there.)
New tests: TestAdvanceAutoRetriesParkedShardsOnce,
TestAdvanceBlocksOnSecondParkAfterAutoRetry (the infinite-loop guard),
TestAdvanceProceedsAfterAutoRetrySucceeds, and a console.test.mjs case
driving the actual tab click end to end. All verified to fail against
the pre-fix code before asserting the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsdNZLfmAFrMX2VUtkLtmm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
advance()'s parked-shard guard used to block a pass indefinitely the instant any shard parked, requiring operator intervention even for transient causes (mount blip, brief NFS hiccup). It now gives the backlog one automatic retry round (store.RetryParkedByJob, resets the attempt counter) before falling back to today's block-and-alert behavior. Bounded to exactly one round — an unbounded retry on a genuinely stuck shard would just be an infinite loop wearing a different name.scanHistdata (previously only fed a small KPI sparkline) on the big auto-scaling timeline graph. Choice persists across reloads./api/v1/jobs/{name}/parked/retry|drop+ job-scoped console buttons). No change needed.Test plan
go build ./...,go vet ./...,gofmt -l .cleango test ./...full coordinator suite greengo test -race ./internal/passctrl/...cleanmake webui-test— 61/61 passing, including a new end-to-end test driving the tab clickTestAdvanceAutoRetriesParkedShardsOnce,TestAdvanceBlocksOnSecondParkAfterAutoRetry,TestAdvanceProceedsAfterAutoRetrySucceeds) and the new console test all verified to fail against the pre-fix code before asserting the fix🤖 Generated with Claude Code
https://claude.ai/code/session_01PsdNZLfmAFrMX2VUtkLtmm