Repository navigation
ci(release): make the npm publish re-runnable and the provenance poll long enough - #988
Merged
Merged
Conversation
… long enough The "Publish npm package" job failed on both 0.81.0 and 0.82.0, and both times the release itself was complete: npm had the version with SLSA provenance minutes later. Two separate bugs. Verify provenance landed gave up too early. It polled 10 x 15s and quit after 2m34s. The registry serves the version document and tarball at once but replicates the packument `npm view` reads separately: 0.81.0 published 03:30:01Z, visible 03:32:40Z (158s), gave up 03:32:35Z 0.82.0 published 05:23:52Z, visible 05:26:30Z (158s), gave up 05:26:25Z Four and five seconds short. Now ~10 minutes, which is what Localidraw, Hallway, Local Review, cli-contract and BragDoc all use and none of them has ever failed; Managers waits 30 because it once saw metadata take ~9 minutes. The wait backs off 5s to 30s so the usual case still reports promptly. `npm view`'s stderr is dropped: the first minutes are expected to be nothing but multi-line E404s, and leaving them in the log is what made registry lag look like a failed publish. Publish was not re-runnable. An npm version can never be published twice, so re-running the job to get past the timeout above died with "You cannot publish over the previously published versions: 0.81.0" (run 41960228079) -- turning a recoverable red release into a stuck one. It now skips a version already on npm, as the five sibling repos do, so "Re-run failed jobs" finishes the release. RELEASING.md gains a section saying a red release does not mean a failed publish, and that re-running the job is now the recovery path. CI-only, so no changeset and no version bump (CONTRIBUTING). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploying paddock with
|
| Latest commit: |
0d5e984
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://76019b83.paddock-7u2.pages.dev |
| Branch Preview URL: | https://release-verify-fix.paddock-7u2.pages.dev |
edspencer
added a commit
to edspencer/managers
that referenced
this pull request
Nov 5, 2026
An npm version can never be published twice, so re-running this job after a LATER step failed dies with "You cannot publish over the previously published versions". The Verify provenance landed comment worked around that by telling you to "never re-run this job" -- which leaves a release that published fine and then tripped over a later step with no recovery path at all. Paddock hit exactly this on 0.81.0: its provenance poll timed out, the re-run died on the republish, and the run is permanently red even though npm has the version with provenance (edspencer/paddock#988). Skip a version already on the registry, so "Re-run failed jobs" finishes the release. Localidraw, Hallway, Local Review, cli-contract, BragDoc and now Paddock all do this; Managers was the last one without it. Asks the registry directly with caching off, as Verify provenance landed does, rather than through `npm view`: a cached packument predating the publish would give a false "not on npm", which is the failure this guards against. Only a 200 skips -- a 404, or a registry having a bad day, publishes, exactly as this step did unconditionally before. Managers' 30-minute poll already covers the registry lag that broke Paddock's releases, so the verify step is unchanged apart from its comment no longer banning a re-run. CI-only, so no changeset and no version bump (CONTRIBUTING). Co-authored-by: HomeLab Agent <homelab-infra@valfenda.net> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Publish npm package job failed on both 0.81.0 and 0.82.0. Both times the release itself was complete — npm had the version with SLSA provenance minutes later, the images were pushed and the GitHub Release existed. Only the job was wrong. There are two separate bugs behind that, and this fixes both.
1.
Verify provenance landedgave up too earlyIt polled
10 × 15sand quit after 2m34s. The registry serves a new version's document and tarball immediately, but the aggregate packument thatnpm viewresolves against replicates separately — the same lag RELEASING.md's bootstrap notes already describe.I pulled the timings out of the two runs (
gh run view <id> --log --attempt N) and cross-checked them against the registry's owntimefield:npm publishreturnedFour and five seconds short, respectively. Every one of the 10 attempts in both runs logged a multi-line
npm error code E404— the version was not merely un-attested, it was not visible at all.The poll is now ~10 minutes, which is what Localidraw, Hallway, Local Review, cli-contract and BragDoc all use and none of them has ever failed. Managers waits 30 minutes because it once watched metadata take ~9 minutes and a tarball ~14, so 10 is the floor rather than a generous margin — noted in the comment.
The wait backs off 5s → 10s → 20s → 30s and then holds at 30s, so the common case still reports within seconds of the registry catching up while the tail costs few requests.
npm view's stderr is dropped (2>/dev/null, as every sibling repo does): the first minutes are expected to be nothing but E404, and leaving that in the log is precisely what made ordinary registry lag read as a failed publish. An E404 here means "not yet", not "the publish failed".2.
Publishwas not re-runnableAn npm version can never be published twice, so re-running the job to get past the timeout above died with:
(run 41960228079, attempt 2). A recoverable red release became a stuck one. Paddock was the only one of the six repos without the guard; Localidraw, Hallway, Local Review, cli-contract and BragDoc all skip a version that is already on npm. Copied from Localidraw's, verbatim apart from the package name:
So "Re-run failed jobs" now finishes a release that published fine and tripped over a later step.
How I verified it
No UI, so no screenshots. The risk here is shell logic in a step that runs once per release and is painful to get wrong, so I extracted both
run:blocks straight out of the committed workflow and drove them against a stubnpmthat reproduces the real behaviours (multi-lineE404on stderr, exit 1, nothing on stdout, flipping to success after N seconds).Publish:already on npm; not publishing it again, 0npm publishcalls+ @edspencer/paddock@…, 1npm publishcallVerify provenance landed:::error::One real risk needed ruling out: this step runs in
working-directory: dist-npm, which holds a stagedpackage.jsonfor this very package at this very version. Ifnpm viewever answered from that local manifest, the guard would skip every publish and no release would ever reach npm again. Checked against the real registry:@edspencer/paddock@99.99.99(unpublished)[]dist-npmwhose package.json declares exactly99.99.99[]@edspencer/paddock@0.82.0(published)[0.82.0]So an explicit spec goes to the registry and the local manifest is never consulted.
The 158s case is the control that matters: it is the exact lag that broke both releases, it fails on
mainand passes here.set -esafety was the specific trap I was watching for — a bare[ "$delay" -lt 30 ] && delay=$((delay*2))exits the script the moment the test is false, which would have ended the poll silently at the 30s cap. Written asif/fiinstead, and the 22-attempt run is what proves it.The whole file still parses, with the job's steps and
needs: [release, verify-ci, build-image]unchanged:What CI here does and does not prove
release.ymlonly runson: pushtomainandworkflow_dispatch, and Paddock's CI has no actionlint job, so this PR's CI never executes the changed workflow. Green here means nothing else broke. The changed steps are covered by the stub runs above plus a YAML parse; the first live exercise is the next real release, and the idempotent publish is what makes that recoverable if anything else is wrong.Notes
next) already poll 40 × 15s and already skip an existing version: nothing to do. Managers polls for 30 minutes but has the non-idempotent publish, and its comment tells you to "never re-run this job" — edspencer/managers#18 fixes that.G taskthe orchestrator filed in the CLI-contract Status log for 0.81.0.🤖 Generated with Claude Code