Skip to content

guard: fix stats-refresh job (root cause of fleet-wide staleness jam) + failure visibility - #716

Merged
matt82198 merged 3 commits into
mainfrom
guard/stats-refresh-fix
Aug 4, 2026
Merged

matt82198 merged 3 commits into
mainfrom
guard/stats-refresh-fix

Conversation

@matt82198

Copy link
Copy Markdown
Owner

The escape

The nightly stats-refresh job concluded FAILURE on all four of its last runs (2026-07-30 .. 08-02) with no signal anywhere. stats.json drifted 200+ commits stale, which red-lined the stats freshness gate on every open PR — a fleet-wide jam. Main was healed by a manual regen (2ddfcb14), but the job itself was still broken and would have re-tripped at the next drift threshold.

Three stacked defects, each hidden behind the one before it

1. A cosmetic label was load-bearing. The actual error from run 30730178877:

could not add label: 'chore' not found
##[error]Process completed with exit code 1.

gh resolves --label before it opens the PR, so a label that did not exist in the repo failed the whole step and no PR was ever created. Four orphan branches, zero PRs. (The chore label has since been created repo-side, which unmasked defect 2.)

2. Actions was not permitted to create pull requests. With the label present, run 30780718553 failed with:

pull request create failed: GraphQL: GitHub Actions is not permitted to
create or approve pull requests (createPullRequest)

can_approve_pull_request_reviews was false on the repo.

3. The date-suffixed branch scheme was not idempotent. With the setting flipped, run 30780802496 failed one step earlier:

error: Your local changes to the following files would be overwritten by checkout:
	stats.json

chore/stats-refresh-<date> plus a switch to an existing branch aborts on any re-run, because the regenerate step leaves stats.json dirty. The job only ever worked on the first attempt of a given date.

Fixes

  • Labels attached best-effort after the PR exists, never passed to gh pr create. A missing or renamed label can no longer fail the job.
  • merge-queue label added alongside chore, so routine stats bumps go to the merge-queue advancer daemon instead of needing a human on high-velocity days. This closes the deeper gap: even a working coalescing PR previously sat until someone noticed it.
  • One coalescing branch chore/stats-refresh, replaced in place with -B rather than a switch. Exactly one open stats PR at a time; re-runs are idempotent.
  • Explicit push lease. --force-with-lease with no remote-tracking ref is rejected as "stale info", so the lease value is computed from a fetched ref (--force-with-lease=<ref>:<sha>), with a plain create when the branch does not exist yet.

Failure visibility — the wall for the wall

Nothing watches a scheduled workflow's conclusion, which is exactly why four consecutive failures were silent. The job now:

  • opens a deduplicated tracking issue on failure (title-matched against open issues, so repeats comment rather than spawn new issues), and
  • closes that issue on success, so the signal stays truthful rather than rotting open.

An issue was chosen over the alternatives because it is the only option that is actually real here: it needs no push access, no branch-protection exception, and no new credential; it is visible on the repo's issue list without anyone tailing Actions; and it outlives log retention. (ci_job_execution_verifier from #698 will catch the complementary case — jobs defined but never executed — once wired.)

Repo settings changed in this lane

can_approve_pull_request_reviews set to true (one API call, trivially reversible). Branch protection is untouched: enforce_admins stands, all five required status checks (ci (0..3), windows) stand, and required_approving_review_count is 0 — so the "approve" half of that setting grants no bypass power whatsoever. No --admin, no protection exception.

Verification

Manual dispatch of this branch's workflow — run 30780963496 — concluded SUCCESS, every step green:

success  Regenerate stats
success  Check for changes
success  Commit onto the coalescing stats branch
success  Push branch
success  Create or update pull request
success  Close failure tracking issue on success
skipped  Surface failure as a tracking issue

It opened #715 on chore/stats-refresh carrying both the chore and merge-queue labels. A second dispatch confirmed the re-run path (defect 3) is now idempotent.

Gates run locally, all green: ci_workflow_lint.py, ci_gate_runability.py, workflow_model_linter.py, verify_gates_wired.py, watcher_linter.py --check, encoding_lint.py, secret_scan.py (CLEAN).

Janitor follow-up (not done here — deliberately)

Orphan branches hold regenerated stats that nobody ever saw. Left in place for a janitor pass, not deleted:

  • chore/stats-refresh-2026-07-30
  • chore/stats-refresh-2026-07-31
  • chore/stats-refresh-2026-08-01
  • chore/stats-refresh-2026-08-02
  • chore/stats-refresh-2026-08-03 (created by the diagnostic dispatches during this investigation)

🤖 Generated with Claude Code

matt82198 and others added 2 commits August 2, 2026 22:06
The nightly stats-refresh job had concluded FAILURE on every one of its
last four runs (2026-07-30..08-02) with no signal anywhere, letting
stats.json drift 200+ commits stale and red-line the stats freshness gate
on every open PR in the repo.

Three stacked defects, each hidden behind the one before it:

1. `gh pr create --label "chore"` - the repo had no `chore` label and gh
   resolves labels BEFORE opening the PR, so the command exited 1 and no
   PR was ever created. Four orphan branches, zero PRs.
2. Actions was not permitted to create pull requests
   (`can_approve_pull_request_reviews: false`), so even with the label
   present `gh pr create` failed with "GitHub Actions is not permitted to
   create or approve pull requests".
3. The date-suffixed branch scheme (`chore/stats-refresh-<date>`) aborted
   on any re-run: switching to an existing branch is refused while the
   regenerated stats.json sits dirty in the working tree.

Fixes:
- Labels are attached best-effort AFTER the PR exists, never passed to
  `gh pr create`, so a cosmetic label can no longer fail the job. The
  `merge-queue` label is added too, handing routine stats bumps to the
  merge-queue advancer instead of a human.
- One coalescing branch (`chore/stats-refresh`) replaced in place, so
  there is exactly one open stats PR at a time and re-runs are idempotent.
- The push takes an explicit lease (`--force-with-lease=<ref>:<sha>`)
  computed from a fetched remote-tracking ref, and falls back to a plain
  create when the branch does not exist yet.

Failure visibility (the wall for the wall): nothing watches a scheduled
workflow's conclusion, which is why four failures were silent. The job now
opens a deduplicated tracking issue on failure (comments on repeats rather
than spamming) and closes it again on success. An issue needs no push
access and no branch-protection exception, and it outlives log retention.

Repo settings changed in this lane (reversible, one API call):
`can_approve_pull_request_reviews` set to true. Branch protection is
untouched - `enforce_admins` and all five required status checks stand,
and `required_approving_review_count` is 0, so the "approve" half of that
setting grants no bypass power.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tests/test_traps.py::TestCIDriftTrap::test_ci_workflow_files_are_valid_yaml
lexes workflow files line by line and rejects any non-comment line holding
an odd number of double quotes. It has no notion of YAML block scalars,
shell line continuations or heredocs, so the previous revision tripped it
on ten lines that were nonetheless valid for Actions: multi-line
double-quoted PR/issue bodies, backslash-continued gh invocations, and jq
filters with escaped quotes nested inside double quotes.

Rewritten so every line balances its own double quotes, with no change in
behaviour:

- PR body and failure-issue body are built with unquoted heredocs into
  $RUNNER_TEMP and passed via --body-file, so the prose lines carry no
  quotes at all and interpolation still works.
- gh invocations are single lines; the PR title moved to single quotes.
- The issue-dedup jq filter reads the title through jq's env.* rather than
  nesting escaped quotes, which removes the last odd-quote line.
- A comment at the top of the file records the trap's rule so the next
  editor does not rediscover it the hard way.

Verified: the exact failing test passes, the full tests/test_traps.py suite
passes (14), ci_workflow_lint / workflow_model_linter / ci_gate_runability /
verify_gates_wired / encoding_lint all pass, and both heredocs were dry-run
through bash to confirm they render flush-left after YAML dedent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@matt82198

Copy link
Copy Markdown
Owner Author

Evicted from the merge queue: batch #736 (integrate/q-1785730980) red with every member individually green

@matt82198 matt82198 added the merge-queue Queued for the merge-queue advancer daemon label Aug 3, 2026
@matt82198 matt82198 removed the queue-rejected Evicted from the merge queue (red or culprit) label Aug 4, 2026
@matt82198 matt82198 added queue-rejected Evicted from the merge queue (red or culprit) and removed merge-queue Queued for the merge-queue advancer daemon labels Aug 4, 2026
@matt82198

Copy link
Copy Markdown
Owner Author

Evicted from the merge queue: batch #766 (integrate/q-1785803317) red with every member individually green

@matt82198 matt82198 added merge-queue Queued for the merge-queue advancer daemon and removed queue-rejected Evicted from the merge queue (red or culprit) labels Aug 4, 2026
@matt82198
matt82198 merged commit feabbad into main Aug 4, 2026
6 of 12 checks passed
@matt82198
matt82198 deleted the guard/stats-refresh-fix branch August 4, 2026 04:12
@matt82198 matt82198 mentioned this pull request Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-queue Queued for the merge-queue advancer daemon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant