Skip to content

ci(release): make the npm publish re-runnable and the provenance poll long enough - #988

Merged
edspencer merged 1 commit into
mainfrom
release-verify-fix
Nov 5, 2026
Merged

edspencer merged 1 commit into
mainfrom
release-verify-fix

Conversation

@edspencer

@edspencer edspencer commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

The Publish npm package job failed on both 0.81.0 and 0.82.0. Both times the release itself was complete — npm had the version with SLSA provenance minutes later, the images were pushed and the GitHub Release existed. Only the job was wrong. There are two separate bugs behind that, and this fixes both.

1. Verify provenance landed gave up too early

It polled 10 × 15s and quit after 2m34s. The registry serves a new version's document and tarball immediately, but the aggregate packument that npm view resolves against replicates separately — the same lag RELEASING.md's bootstrap notes already describe.

I pulled the timings out of the two runs (gh run view <id> --log --attempt N) and cross-checked them against the registry's own time field:

Version npm publish returned Visible on the registry Lag Step gave up
0.81.0 03:30:01Z 03:32:40Z 158s 03:32:35Z
0.82.0 05:23:52Z 05:26:30Z 158s 05:26:25Z

Four and five seconds short, respectively. Every one of the 10 attempts in both runs logged a multi-line npm error code E404 — the version was not merely un-attested, it was not visible at all.

The poll is now ~10 minutes, which is what Localidraw, Hallway, Local Review, cli-contract and BragDoc all use and none of them has ever failed. Managers waits 30 minutes because it once watched metadata take ~9 minutes and a tarball ~14, so 10 is the floor rather than a generous margin — noted in the comment.

The wait backs off 5s → 10s → 20s → 30s and then holds at 30s, so the common case still reports within seconds of the registry catching up while the tail costs few requests. npm view's stderr is dropped (2>/dev/null, as every sibling repo does): the first minutes are expected to be nothing but E404, and leaving that in the log is precisely what made ordinary registry lag read as a failed publish. An E404 here means "not yet", not "the publish failed".

2. Publish was not re-runnable

An npm version can never be published twice, so re-running the job to get past the timeout above died with:

npm error You cannot publish over the previously published versions: 0.81.0.

(run 41960228079, attempt 2). A recoverable red release became a stuck one. Paddock was the only one of the six repos without the guard; Localidraw, Hallway, Local Review, cli-contract and BragDoc all skip a version that is already on npm. Copied from Localidraw's, verbatim apart from the package name:

if [ -n "$(npm view "@edspencer/paddock@${VERSION}" version 2>/dev/null || true)" ]; then
  echo "@edspencer/paddock@${VERSION} is already on npm; not publishing it again"
else
  npm publish
fi

So "Re-run failed jobs" now finishes a release that published fine and tripped over a later step.

How I verified it

No UI, so no screenshots. The risk here is shell logic in a step that runs once per release and is painful to get wrong, so I extracted both run: blocks straight out of the committed workflow and drove them against a stub npm that reproduces the real behaviours (multi-line E404 on stderr, exit 1, nothing on stdout, flipping to success after N seconds).

Publish:

Case Expected Result
version already on npm skip, exit 0 ✅ already on npm; not publishing it again, 0 npm publish calls
version not on npm publish, exit 0 ✅ + @edspencer/paddock@…, 1 npm publish call

Verify provenance landed:

Case Expected Result
appears after exactly 158s (replays the two failures) pass ✅ attested on attempt 9 at 185s — the old step gave up at 154s
never appears fail at ~600s with ::error:: ✅ exit 1 after 600s, 22 attempts
backoff shape 5, 10, 20, 30, 30… ✅ elapsed 0, 5, 15, 35, 65, 95, 125, 155, 185…

One real risk needed ruling out: this step runs in working-directory: dist-npm, which holds a staged package.json for this very package at this very version. If npm view ever answered from that local manifest, the guard would skip every publish and no release would ever reach npm again. Checked against the real registry:

cwd spec Expected Result
neutral dir @edspencer/paddock@99.99.99 (unpublished) empty ✅ []
a dist-npm whose package.json declares exactly 99.99.99 same still empty ✅ []
same dir @edspencer/paddock@0.82.0 (published) the version ✅ [0.82.0]

So an explicit spec goes to the registry and the local manifest is never consulted.

The 158s case is the control that matters: it is the exact lag that broke both releases, it fails on main and passes here.

set -e safety was the specific trap I was watching for — a bare [ "$delay" -lt 30 ] && delay=$((delay*2)) exits the script the moment the test is false, which would have ended the poll silently at the 30s cap. Written as if/fi instead, and the 22-attempt run is what proves it.

The whole file still parses, with the job's steps and needs: [release, verify-ci, build-image] unchanged:

 - actions/checkout@v7 / actions/setup-node@v7 / Upgrade npm for trusted publishing
 - npm ci / Build (server + web) / Stage the npm package / Publish / Verify provenance landed
needs: ["release","verify-ci","build-image"]

What CI here does and does not prove

release.yml only runs on: push to main and workflow_dispatch, and Paddock's CI has no actionlint job, so this PR's CI never executes the changed workflow. Green here means nothing else broke. The changed steps are covered by the stub runs above plus a YAML parse; the first live exercise is the next real release, and the idempotent publish is what makes that recoverable if anything else is wrong.

Notes

  • No changeset and no version bump. CI-only, per CONTRIBUTING: "No changeset is needed for pure-internal changes (tests, CI, …)". Nothing here ships in a package.
  • RELEASING.md gains "A red release does not mean a failed publish" — read the Publish step's log before reacting, and re-run the job rather than assuming the release needs redoing.
  • The other repos were checked. Hallway, Local Review, cli-contract and BragDoc (next) already poll 40 × 15s and already skip an existing version: nothing to do. Managers polls for 30 minutes but has the non-idempotent publish, and its comment tells you to "never re-run this job" — edspencer/managers#18 fixes that.
  • This is the G task the orchestrator filed in the CLI-contract Status log for 0.81.0.

🤖 Generated with Claude Code

… long enough

The "Publish npm package" job failed on both 0.81.0 and 0.82.0, and both
times the release itself was complete: npm had the version with SLSA
provenance minutes later. Two separate bugs.

Verify provenance landed gave up too early. It polled 10 x 15s and
quit after 2m34s. The registry serves the version document and tarball
at once but replicates the packument `npm view` reads separately:

  0.81.0  published 03:30:01Z, visible 03:32:40Z (158s), gave up 03:32:35Z
  0.82.0  published 05:23:52Z, visible 05:26:30Z (158s), gave up 05:26:25Z

Four and five seconds short. Now ~10 minutes, which is what Localidraw,
Hallway, Local Review, cli-contract and BragDoc all use and none of them
has ever failed; Managers waits 30 because it once saw metadata take
~9 minutes. The wait backs off 5s to 30s so the usual case still reports
promptly. `npm view`'s stderr is dropped: the first minutes are expected
to be nothing but multi-line E404s, and leaving them in the log is what
made registry lag look like a failed publish.

Publish was not re-runnable. An npm version can never be published
twice, so re-running the job to get past the timeout above died with
"You cannot publish over the previously published versions: 0.81.0"
(run 41960228079) -- turning a recoverable red release into a stuck one.
It now skips a version already on npm, as the five sibling repos do, so
"Re-run failed jobs" finishes the release.

RELEASING.md gains a section saying a red release does not mean a failed
publish, and that re-running the job is now the recovery path.

CI-only, so no changeset and no version bump (CONTRIBUTING).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying paddock with  Cloudflare Pages  Cloudflare Pages

Latest commit: 0d5e984
Status: ✅  Deploy successful!
Preview URL: https://76019b83.paddock-7u2.pages.dev
Branch Preview URL: https://release-verify-fix.paddock-7u2.pages.dev

View logs

@edspencer
edspencer merged commit 936f5b8 into main Nov 5, 2026
10 checks passed
@edspencer
edspencer deleted the release-verify-fix branch November 5, 2026 05:50
edspencer added a commit to edspencer/managers that referenced this pull request Nov 5, 2026
An npm version can never be published twice, so re-running this job after
a LATER step failed dies with "You cannot publish over the previously
published versions". The Verify provenance landed comment worked around
that by telling you to "never re-run this job" -- which leaves a release
that published fine and then tripped over a later step with no recovery
path at all.

Paddock hit exactly this on 0.81.0: its provenance poll timed out, the
re-run died on the republish, and the run is permanently red even though
npm has the version with provenance (edspencer/paddock#988).

Skip a version already on the registry, so "Re-run failed jobs" finishes
the release. Localidraw, Hallway, Local Review, cli-contract, BragDoc and
now Paddock all do this; Managers was the last one without it.

Asks the registry directly with caching off, as Verify provenance landed
does, rather than through `npm view`: a cached packument predating the
publish would give a false "not on npm", which is the failure this
guards against. Only a 200 skips -- a 404, or a registry having a bad
day, publishes, exactly as this step did unconditionally before.

Managers' 30-minute poll already covers the registry lag that broke
Paddock's releases, so the verify step is unchanged apart from its
comment no longer banning a re-run.

CI-only, so no changeset and no version bump (CONTRIBUTING).

Co-authored-by: HomeLab Agent <homelab-infra@valfenda.net>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant