Repository navigation
Commit 9ce6d72
authored
ci(release): fix OIDC trusted publishing, add Slack alert, allow manual re-runs (#2275)
## Summary
Four ci-only changes to `release.yml` that together unblock npm OIDC trusted publishing, route future failures to Slack, and give us a manual retry path:
1. **Upgrade npm to `latest` in `check_changesets`** so `npm publish` can perform the OIDC handshake. Runner ships Node 22 + npm 10.9.4 by default, but client-side OIDC trusted publishing landed in **npm 11.5.1** (July 2025). Without this upgrade `npm publish` has no OIDC code path, falls back to looking for a `NODE_AUTH_TOKEN` (deliberately not set, to avoid conflicting with OIDC), and exits `ENEEDAUTH`.
2. **Drop `continue-on-error: true`** from the `Create Release Pull Request or Publish to npm` step so a future publish failure fails the workflow run instead of leaving it green with no releases shipped.
3. **Add `slack-notify-failure` job** mirroring the pattern in `release-nightly.yml`: gated on `if: failure() && always()`, waits on every major job, lists the failed ones via `listJobsForWorkflowRun`, and posts a danger-colored Slack message to `SLACK_WEBHOOK_URL_ENG_NOTIFS` with the commit SHA + author. Combined with (2), the next release.yml failure surfaces on Slack instead of going silent.
4. **Add `workflow_dispatch:` trigger** so we can manually re-run the workflow against an arbitrary `main` SHA (`gh workflow run release.yml --ref main` or the UI "Run workflow" button). Needed for the recovery path below — `gh run rerun <run-id>` re-uses the original commit's workflow file, so it can't test fixes added in later commits.
## Why we didn't catch this on the April-23 OIDC switch
The April-23 release that introduced OIDC ([commit 3ae1b85](3ae1b85d8)) only **appeared** to work — OIDC never actually fired. Sequence:
1. [Run 24862921978](https://github.com/hyperdxio/hyperdx/actions/runs/24862921978) (2026-04-23, on the release-PR commit before the OIDC switch) tried to publish `@hyperdx/cli@0.4.0` and failed with `E404 Not Found - PUT /@HyperDX%2fcli` — same `continue-on-error: true` mask, workflow green, no GH Releases created.
2. Between 22:55 and 23:30 UTC, `@hyperdx/cli@0.4.0` was published out of band (manual `npm publish`).
3. [Run 24864061403](https://github.com/hyperdxio/hyperdx/actions/runs/24864061403), kicked off by the OIDC commit itself, re-ran `changeset publish` which saw the package was already on npm and emitted:
```
🦋 warn @hyperdx/cli is not being published because version 0.4.0 is already published on npm
🦋 warn No unpublished projects to publish
```
`changeset publish` then exited zero, `changesets/action` proceeded with its `createRelease` loop, and the five `@hyperdx/{api,app,cli,common-utils,otel-collector}` GH Releases got created. OIDC was never exercised — the publish was a no-op.
Today's 2.25.0 release ([run 25827737203](https://github.com/hyperdxio/hyperdx/actions/runs/25827737203)) had no out-of-band publish to fall back on, so the underlying `ENEEDAUTH` surfaced as a real failure for `@hyperdx/cli@0.4.1`. `changesets/action` aborts inside `runPublish` (the `execWithOutput` call sits above the `createGithubReleases` loop, and it re-throws on non-zero exit), so neither the cli nor the four private-package GH Releases (`@hyperdx/api@2.25.0`, `@hyperdx/app@2.25.0`, `@hyperdx/common-utils@0.19.0`, `@hyperdx/otel-collector@2.25.0`) ever got created. The lone `cli-v0.4.1` entry on the Releases page came from the separate `release-cli` job using `softprops/action-gh-release` — not from changesets.
The npm trusted-publisher config also had a `release.yaml` → `release.yml` filename typo (fixed in the npm UI). Real bug, but **independent** — when the npm client never sends an OIDC token in the first place, registry-side trusted-publisher matching is moot.
## Recovery for the stuck 2.25.0 / cli@0.4.1 cycle
After merging this PR:
```bash
gh workflow run release.yml --ref main
```
The run will:
- `check_changesets`: install npm `latest`, see no changesets, `yarn release` → genuine OIDC `npm publish` of `@hyperdx/cli@0.4.1` (with provenance) → push the five 2.25.0 / 0.19.0 git tags → `changesets/action` creates all five GH Releases.
- `check_version`: `manifest inspect` finds the existing 2.25.0 image tag → `should_release=false` → all Docker build / publish / downstream-notify jobs short-circuit.
- `release-cli`: `gh release view cli-v0.4.1` succeeds → `exists=true` → compile + create-release steps are skipped.
- `slack-notify-failure`: only runs if any of the above fail — silent on success.1 parent 46fe675 commit 9ce6d72
1 file changed
Lines changed: 87 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
29 | 35 | | |
30 | 36 | | |
31 | 37 | | |
| |||
39 | 45 | | |
40 | 46 | | |
41 | 47 | | |
42 | | - | |
43 | 48 | | |
44 | 49 | | |
45 | 50 | | |
| |||
687 | 692 | | |
688 | 693 | | |
689 | 694 | | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
690 | 775 | | |
691 | 776 | | |
692 | 777 | | |
| |||
702 | 787 | | |
703 | 788 | | |
704 | 789 | | |
| 790 | + | |
705 | 791 | | |
706 | 792 | | |
707 | 793 | | |
| |||
0 commit comments