[julianknutsen]

Summary

The hourly Update Aggregate Registry workflow has failed on every run since 2026-10-24 18:28 UTC, and the refresh PR #617 is green but never merges.

Root cause. The refresh step force-pushes chore/aggregate-refresh and then, on the App-token path, immediately runs gh pr checks "$pr" --watch --fail-fast. At that moment GitHub has not yet registered any check on the new head, so gh prints no checks reported on the 'chore/aggregate-refresh' branch, exits non-zero, and the job fails about a second after the push. CI then runs and passes on the PR, but nothing is left to merge it. Example from run 40239867630:

22:22:33  + be0b20f...a41ccac chore/aggregate-refresh -> chore/aggregate-refresh (forced update)
22:22:34  reusing open refresh PR #617
22:22:35  no checks reported on the 'chore/aggregate-refresh' branch
22:22:35  ##[error]CI failed on the aggregate refresh; PR #617 left open for inspection

The first failure, run 40120817169 (2026-10-24 18:27 UTC), opened #617 and failed the same way. Later examples: 40235202615, 40230605523, 40225557675. There was also a second, latent problem: gh pr checks reads the PR's head when it runs, so right after a force-push it could see the previous head's green checks, and the unpinned gh pr merge could then merge a commit nobody had watched.

Fix

• New scripts/await-head-checks.ts, a small Bun helper with a pure evaluateChecks and an injectable polling loop. It polls GET /repos/{repo}/commits/{sha}/check-runs for the SHA the job just pushed, so the previous head's checks can never count. It then:
  1. waits up to --register-timeout (600s) for any check to register on that commit;
  2. waits up to --timeout (3600s) for every registered check to finish, and fails fast on any failed, cancelled, timed-out or action-required conclusion;
  3. passes only once every --require check (CI / required, the check main's branch protection requires) has succeeded. This also covers the gap where verify is green but the dependent CI / required job hasn't registered yet, a gap where "all reported checks green" would be a false positive.
  4. retries transient API errors (up to 3 consecutive failures).
• update-registry.yml: captures sha=$(git rev-parse HEAD) after the push and calls the helper instead of gh pr checks --watch. The built-in-token fallback's dispatched-run lookup now also filters on --commit "$sha", since previously it could pick up the previous push's run on the same branch. The merge is gh pr merge --squash --match-head-commit "$sha", so the job can only merge the commit it watched.
• Unchanged from fix(ci): scope write tokens to jobs and pin actions and base images #616: the SHA-pinned actions, the job-scoped permissions and the token selection.

Testing

• bun test scripts/await-head-checks.test.ts: 21 pass. The tests cover:

  • no checks yet, then a successful finish;
  • a bounded timeout when no checks ever register, and when checks never finish;
  • fail-fast;
  • a missing required check while every other check is green;
  • a skipped required check;
  • pagination, argument parsing, and transient vs persistent API errors.

  Two workflow assertions fail against the current main workflow and pass with this change.

• bun run test:unit: 504 pass, 0 fail. This includes scripts/supply-chain-pins.test.ts: every action is still SHA-pinned and there are no top-level write scopes.

• bun run typecheck, bun run generate:check and bun audit all pass.

• actionlint (with shellcheck): no findings for update-registry.yml apart from the existing custom runner-label notice for blacksmith-2vcpu-ubuntu-2404.

• Live read-only run of the helper against the real API:

  • chore: update aggregate registry #617's head a41ccac: exits 0 once CI / required, CodeQL, Analyze (actions), Analyze (javascript-typescript), verify, image have all passed.
  • A commit with failed checks: exits 1 and lists them.
  • A nonexistent SHA: exits 2 after the bounded retries.

• I did not run the workflow via workflow_dispatch on this branch. The job checks out ref: main, so on this branch it would run the new YAML against main's tree, where the helper doesn't exist yet. It would also force-push the live refresh branch and try to merge to main from unreviewed code. The first scheduled run after this merges (at :17 past the hour) is the end-to-end check. It should reuse chore: update aggregate registry #617, wait for the new head's checks and merge it.

Checklist

• Explained why no issue is needed: this is the follow-up flagged in the fix(ci): scope write tokens to jobs and pin actions and base images #616 review
• Added tests for the behavior change
• No user-facing docs or generated registry outputs change

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[gascity-eddie]

This PR has been added to the review queue.

Track progress: https://factory.gascity.com/city/maintainer-city/runs/gcg--9223372036836987040

[julianknutsen]

Max-effort review: approve and merge

Verdict: clean after two Low fixes, which I pushed as 41233e8 (fast-forward). The helper fixes the root cause:

• It keys every poll on the pushed SHA, so the previous head's checks never count.
• It rides out the gap before checks register.
• It won't pass until CI / required exists and has succeeded.

--match-head-commit means the job can only merge the commit it watched. All 6 checks are green on 41233e8.

Findings

Sev	Where	Finding	Status
Low	update-registry.yml (no concurrency)	Overlapping runs race on the shared branch. Cron fires late and unevenly (the 00:59 and 01:38 runs today were 39 min apart), and the helper could wait up to 60 min. The newer force-push supersedes the commit the older run is watching: image.yml (cancel-in-progress on refs/pull/617/merge) cancels that commit's build, or the older run's --match-head-commit merge is refused. The older run fails for nothing. It never merges an unchecked commit.	Fixed: workflow concurrency: {group: update-registry, cancel-in-progress: false}
Low	update-registry.yml --timeout 3600	The job runs on an App installation token that expires 1 h after it is minted. The helper starts seconds after minting and could poll for 3600 s. A slow CI would end in 401s (exit 2 after 3 failed polls), not a clean timeout, and no merge could succeed in that tail.	Fixed: --timeout 2700 plus job timeout-minutes: 60. The job cap also bounds how long a stuck run can hold the group, and how long a hung fetch can last, since fetch has no per-request timeout.
Info	App permissions	gascity-registry-aggregate has contents: write, pull_requests: write and metadata: read, but not checks: read. Its check-runs read works only because the repo is public: the endpoint is allowsPublicRead, and an unauthenticated request returns all 6 runs. If the repo is ever made private, the helper will get 403s. Grant checks: read then.	Not changed
Info	tsconfig.*.json	bun run typecheck doesn't include scripts/, so it never checked the helper. I typechecked the helper and its tests separately with --strict --noUncheckedIndexedAccess --exactOptionalPropertyTypes: clean. This is a pre-existing gap for all of scripts/*.ts.	Follow-up
Info	gh pr merge	When mergeStateStatus is BLOCKED, gh refuses to merge, and in non-TTY mode it prints nothing. I polled #617's head 6250911 every ~1.8 s. The state flipped to CLEAN in the same poll where CI / required showed completed, so a merge straight after the helper doesn't race a stale state.	No change

Verified correct

• Pagination: the helper follows Link rel="next". Against a commit with 234 check runs (gastownhall/gascity adae2e6), it found a check that appears only on page 3.
• Re-runs: the endpoint's default filter=latest returns only the newest attempt for each check name.
  • Checked on gascity/bd-enterprise 7f68226, where attempt 1 failed and attempt 2 passed. latest returned 33 runs, none of them the failed attempt; filter=all returned 62.
  • Same-name runs from different suites are handled conservatively: any failure fails and any incomplete run waits. It can't pass falsely.
• Other apps, neutral and skipped: the GHAS CodeQL run is included. Non-required checks pass when neutral or skipped. The required check must be exactly success, which is stricter than branch protection.
• Late registration: CI / required registers only after verify finishes. It was missing from 6250911's list while verify ran. The helper's "not registered yet" pending state covers that gap, bounded by --timeout.
• Which pushes trigger CI:
  • App pushes start chore: update aggregate registry #617's pull_request CI with no approval gate. Each head got the full check set within ~10 s of the push.
  • Runs from a PR created or updated with GITHUB_TOKEN start approval-gated (GitHub changelog, 2026-06-11). That's why the fallback path dispatches CI itself, and it now finds that run with --commit "$sha".
• No unchecked merge: protection requires CI / required (strict mode, any app). The helper watches only the pushed SHA. With expectedHeadOid set, GitHub refuses the merge if the head has moved.
• fix(ci): scope write tokens to jobs and pin actions and base images #616 unchanged: the pins and job-scoped permissions are untouched, and supply-chain-pins.test.ts passes.

Verification (Bun 1.4.4, as in CI)

• Helper tests: 23 pass. The 2 new workflow assertions fail on ccc0385 and pass on 41233e8.
• test:unit: 506 pass. typecheck, generate:check and bun audit are clean.
• actionlint with shellcheck: the same 9 pre-existing notices as main.
• Live, read-only helper runs:
  • green 9cf358b: exit 0
  • failed 61f3d8d: exit 1, listing the failed checks
  • nonexistent SHA: exit 2 after 3 polls
  • in-flight 41233e8: waited for checks to register, then for CI / required, then exit 0

End-to-end: the first scheduled run after the merge should reuse #617, wait for its new head's checks and merge it. I'll report that run here.

🤖 Generated with Claude Code

[julianknutsen]

End-to-end result: the hourly refresh works again

Run 40274509516 (schedule, 05:29Z) was the first scheduled run after the merge (f72fb35). It succeeded and merged #617:

05:30:03  + 6250911...a93e7c4 chore/aggregate-refresh -> chore/aggregate-refresh (forced update)
05:30:04  reusing open refresh PR #617
05:30:04  awaiting checks on gascity/registry@a93e7c4…; required: CI / required
05:30:05  no checks registered on a93e7c4… yet
05:30:25  waiting on: CI / required (not registered yet), Analyze (actions), image, Analyze (javascript-typescript), verify
05:31:47  waiting on: CI / required (not registered yet), verify
05:33:09  all checks passed on a93e7c4…: CI / required, CodeQL, Analyze (actions), image, Analyze (javascript-typescript), verify
05:33:13  merged PR #617

• The run hit the same "no checks yet" window that failed every run since 2026-10-24 18:28. This time it waited.
• It also waited through the gap where verify is running and CI / required hasn't registered yet. It passed only on the pushed SHA.
• chore: update aggregate registry #617 was squash-merged by gascity-registry-aggregate[bot] as 6d7327c. Its head was a93e7c4, the SHA the helper watched.
• CI, CodeQL, image, Promote to production and Scorecard are all green on 6d7327c.
• Side note: the GHAS CodeQL check on a93e7c4 reported neutral for about 30 s before it switched to success. That confirms treating neutral as non-blocking is the right call.

Run 40282364257 (schedule, 06:52Z, the next one) logged aggregate registry already current and exited 0. That's the steady state.

No code-scanning alerts were introduced. The only open one is the pre-existing Scorecard CodeReview process item (#7).

🤖 Generated with Claude Code