Skip to content

ci: retry Node setup on the manifest-miss legs; fix trunk-red's close rule - #588

Open
colinhacks wants to merge 1 commit into
mainfrom
ci-node-matrix-flake
Open

ci: retry Node setup on the manifest-miss legs; fix trunk-red's close rule#588
colinhacks wants to merge 1 commit into
mainfrom
ci-node-matrix-flake

Conversation

@colinhacks

Copy link
Copy Markdown
Contributor

Smoke (node 25.2) went red on main inside actions/setup-node, before nub's binary ran. Transient — the same leg passed next run.

actions/node-versions ships no 23.x/25.x, so four smoke legs fetch nodejs.org/dist/index.json live. That request retries only on 502/503/504, so a network blip kills the leg with no diagnostic (actions/setup-node#1136).

  • Retry setup-node once in the smoke job.
  • trunk-red cleared on any green CI, though seven workflows file the issue — CI is failing on main #550 was filed by WPT Worker and Node matrix, closed by a CI green. Now needs every reported workflow green.
  • The gate named no failing leg. It now annotates each, and treats abandoned as red.

Refs #585

… rule

The `Smoke (node 25.2)` leg of Node matrix went red on main inside
actions/setup-node, before nub's own binary ran. actions/node-versions publishes
no 23.x or 25.x, so four smoke legs (23.6, 23.11, 25.0, 25.2) miss the manifest
and setup-node fetches nodejs.org/dist/index.json live. That request retries only
on 502/503/504, so a transient network failure kills the leg outright and prints
no diagnostic at all — actions/setup-node#1136. The identical leg passed on the
next run. Retry the step once.

Two defects found while tracing it:

- trunk-red cleared its tracking issue on any green `CI` run, although seven
  workflows can file one. #550 was filed by WPT Worker and Node matrix, then
  closed by a CI green with neither verified. It now re-checks every workflow the
  issue actually reports and closes only once they are all green on main.

- The Node matrix gate reported `smoke=failure` and named no leg, which is why
  attributing this red took a full investigation. It now emits an error
  annotation listing each failing job with its URL, and treats `abandoned` (a
  reclaimed runner, seen on #550) as red rather than an unenumerated value.

Refs #585
Copilot AI review requested due to automatic review settings July 28, 2026 18:40
@vercel

vercel Bot commented Jul 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nub Ready Ready Preview, Comment Jul 28, 2026 6:42pm

Request Review

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — two minor robustness edges in the trunk-red close rule, inline.

Reviewed changes — two CI-workflow-only changes (no product code) that harden the Node smoke matrix against a transient setup-node failure and fix the trunk-red tracking issue's over-eager auto-close. Refs #585.

  • Retry setup-node once on the manifest-miss smoke legs — the first actions/setup-node runs with continue-on-error: true and a second identical step retries only if: steps.setup-node.outcome == 'failure', so a lone nodejs.org/dist/index.json blip on the 23.x/25.x legs no longer reds the leg.
  • node-matrix-gate now names the failing legs — gains permissions: actions: read + GH_TOKEN and, on red, enumerates the non-green jobs via gh api .../jobs instead of printing a bare "smoke failed"; cancelled/abandoned are treated as red (fail-closed).
  • Rewrite trunk-red's close rule to require every implicated workflow green — parses the implicated workflow names from the issue body/comments and re-checks each one's latest main run, closing the tracking issue only when all are terminal-green (was: close on any CI success, the #550 false-clear).

The change is sound and unusually well-commented; the retry logic and the gate's fail-closed reading are correct. The two inline notes are robustness edges in the new close rule, both safe-direction (they bias toward leaving the issue open, not toward a false-clear) and non-blocking.

Pullfrog  | Fix all ➔Fix 👍s ➔View workflow run | Using Claude Opus𝕏

case "$latest" in
success|cancelled|skipped) ;;
*) still_red="$still_red $wf" ;;
esac

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ Treating cancelled as clearing a workflow is safe for the common merge-burst case — a superseding push means --limit 1 returns the newer run, not the cancelled one, so the cancelled run is never what's read here. The narrow gap is a cancel with no superseding run yet (a maintainer manually cancels a stuck run, or a runner-availability timeout): that cancelled run genuinely is the latest, nothing verified the fix, and this clears the workflow anyway.

The comment above ("cancelled is a superseded run, which the job-level if: above already treats as noise") conflates two paths — the job-level if: gates whether this trigger event fires, not what gh run list returns for an unrelated already-red workflow. Worth either dropping cancelled from the clearing set (fail-closed, consistent with the gate change) or tightening the comment to name the manual-cancel gap.

Technical details
# `cancelled` in the clearing set can false-clear on a non-superseded cancel

## Affected sites
- `.github/workflows/trunk-red.yml:96``success|cancelled|skipped) ;;` clears the workflow.
- `.github/workflows/trunk-red.yml:93-94` — comment justifies `cancelled` as "superseded run," but that reasoning only covers the merge-burst case.

## Required outcome
- A workflow whose latest `main` run is `cancelled` *without* a newer superseding run should not be read as green.

## Suggested approach (optional)
- Drop `cancelled` from line 96 so it falls to `*)` (still-red) — the fail-closed reading, matching the node-matrix-gate change in this same PR. Superseded-run convergence still holds because the newer run re-fires trunk-red when it settles.
- Or keep it and reword the comment to state the manual/timeout-cancel gap explicitly rather than implying it's fully covered.

# shellcheck disable=SC2016 # the backticks are literal: they match the report's markdown
reported=$(gh issue view "$existing" --repo "$REPO" --json body,comments \
--jq '[.body] + [.comments[].body] | .[]' \
| sed -n 's/^`\([^`]*\)` failed on main at .*/\1/p' | sort -u)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ This reported=$(...) pipeline is unguarded under set -euo pipefail. If gh issue view has a transient failure (rate limit, network blip), pipefail propagates its non-zero status even though sed/sort succeed on empty input, and set -e aborts the whole step. Contrast line 88's latest=$(gh run list ... || echo ""), which is explicitly guarded. The failure direction is safe (the step just fails and the issue is left untouched, self-healing on the next trigger), but the inconsistency is worth closing — e.g. append || true here too, or wrap the whole assignment. A legitimate empty match is already handled by the [[ -n "$reported" ]] fallback below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants