Repository navigation
fix: claim the pending build marker atomically - #29
Merged
Merged
Conversation
Concurrent cache misses for one key each read _pending/<key>, found it absent, and wrote their own nonce into it. Only the last write survived, so every build but one was dispatched carrying a nonce the worker no longer recognised, and failed its push with a 403 after occupying a runner and a monthly build slot. The marker is now claimed with a conditional put, so exactly one caller dispatches. An expired marker is replaced by etag rather than deleted and rewritten, which would let one caller wipe a claim another had just made.
Nothing ran either suite automatically. The only workflow in the repo is the gem builder, which is workflow_dispatch only, so a regression could land on main without anything noticing. Ruby runs across the versions the README claims support for. Rubocop is deliberately left out for now: the tree has 25 outstanding offences and wiring it up would make CI red on arrival.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #30.
Three builds for the same gem were dispatched within three seconds of each other. Two of them compiled fine and then failed on the final upload with a 403, while the third succeeded. The two that failed had produced perfectly good binaries and threw them away, each having occupied a runner and consumed one of the month's build allowance.
When several people request the same uncached gem at once, the service is meant to start one build and let the rest wait for it. The check deciding whether a build was already running read a marker that the other requests had not yet written, so all three concluded nothing was running. Each then stamped its own one-time upload token over the previous one, and only the last token was still valid by the time the builds finished.
That decision is now made in a single step only one request can win, so simultaneous requests for the same gem start one build. Requests that lose the race stop there instead of dispatching a build that gets rejected at the end. A second commit adds the automation that runs the test suites, which until now nothing did.