guard: fix stats-refresh job (root cause of fleet-wide staleness jam) + failure visibility - #716
Merged
Merged
Conversation
The nightly stats-refresh job had concluded FAILURE on every one of its last four runs (2026-07-30..08-02) with no signal anywhere, letting stats.json drift 200+ commits stale and red-line the stats freshness gate on every open PR in the repo. Three stacked defects, each hidden behind the one before it: 1. `gh pr create --label "chore"` - the repo had no `chore` label and gh resolves labels BEFORE opening the PR, so the command exited 1 and no PR was ever created. Four orphan branches, zero PRs. 2. Actions was not permitted to create pull requests (`can_approve_pull_request_reviews: false`), so even with the label present `gh pr create` failed with "GitHub Actions is not permitted to create or approve pull requests". 3. The date-suffixed branch scheme (`chore/stats-refresh-<date>`) aborted on any re-run: switching to an existing branch is refused while the regenerated stats.json sits dirty in the working tree. Fixes: - Labels are attached best-effort AFTER the PR exists, never passed to `gh pr create`, so a cosmetic label can no longer fail the job. The `merge-queue` label is added too, handing routine stats bumps to the merge-queue advancer instead of a human. - One coalescing branch (`chore/stats-refresh`) replaced in place, so there is exactly one open stats PR at a time and re-runs are idempotent. - The push takes an explicit lease (`--force-with-lease=<ref>:<sha>`) computed from a fetched remote-tracking ref, and falls back to a plain create when the branch does not exist yet. Failure visibility (the wall for the wall): nothing watches a scheduled workflow's conclusion, which is why four failures were silent. The job now opens a deduplicated tracking issue on failure (comments on repeats rather than spamming) and closes it again on success. An issue needs no push access and no branch-protection exception, and it outlives log retention. Repo settings changed in this lane (reversible, one API call): `can_approve_pull_request_reviews` set to true. Branch protection is untouched - `enforce_admins` and all five required status checks stand, and `required_approving_review_count` is 0, so the "approve" half of that setting grants no bypass power. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tests/test_traps.py::TestCIDriftTrap::test_ci_workflow_files_are_valid_yaml lexes workflow files line by line and rejects any non-comment line holding an odd number of double quotes. It has no notion of YAML block scalars, shell line continuations or heredocs, so the previous revision tripped it on ten lines that were nonetheless valid for Actions: multi-line double-quoted PR/issue bodies, backslash-continued gh invocations, and jq filters with escaped quotes nested inside double quotes. Rewritten so every line balances its own double quotes, with no change in behaviour: - PR body and failure-issue body are built with unquoted heredocs into $RUNNER_TEMP and passed via --body-file, so the prose lines carry no quotes at all and interpolation still works. - gh invocations are single lines; the PR title moved to single quotes. - The issue-dedup jq filter reads the title through jq's env.* rather than nesting escaped quotes, which removes the last odd-quote line. - A comment at the top of the file records the trap's rule so the next editor does not rediscover it the hard way. Verified: the exact failing test passes, the full tests/test_traps.py suite passes (14), ci_workflow_lint / workflow_model_linter / ci_gate_runability / verify_gates_wired / encoding_lint all pass, and both heredocs were dry-run through bash to confirm they render flush-left after YAML dedent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Aug 3, 2026
Merged
Owner
Author
|
Evicted from the merge queue: batch #736 (integrate/q-1785730980) red with every member individually green |
This was referenced Aug 4, 2026
Owner
Author
|
Evicted from the merge queue: batch #766 (integrate/q-1785803317) red with every member individually green |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The escape
The nightly
stats-refreshjob concluded FAILURE on all four of its last runs (2026-07-30 .. 08-02) with no signal anywhere.stats.jsondrifted 200+ commits stale, which red-lined the stats freshness gate on every open PR — a fleet-wide jam. Main was healed by a manual regen (2ddfcb14), but the job itself was still broken and would have re-tripped at the next drift threshold.Three stacked defects, each hidden behind the one before it
1. A cosmetic label was load-bearing. The actual error from run
30730178877:ghresolves--labelbefore it opens the PR, so a label that did not exist in the repo failed the whole step and no PR was ever created. Four orphan branches, zero PRs. (Thechorelabel has since been created repo-side, which unmasked defect 2.)2. Actions was not permitted to create pull requests. With the label present, run
30780718553failed with:can_approve_pull_request_reviewswasfalseon the repo.3. The date-suffixed branch scheme was not idempotent. With the setting flipped, run
30780802496failed one step earlier:chore/stats-refresh-<date>plus a switch to an existing branch aborts on any re-run, because the regenerate step leavesstats.jsondirty. The job only ever worked on the first attempt of a given date.Fixes
gh pr create. A missing or renamed label can no longer fail the job.merge-queuelabel added alongsidechore, so routine stats bumps go to the merge-queue advancer daemon instead of needing a human on high-velocity days. This closes the deeper gap: even a working coalescing PR previously sat until someone noticed it.chore/stats-refresh, replaced in place with-Brather than a switch. Exactly one open stats PR at a time; re-runs are idempotent.--force-with-leasewith no remote-tracking ref is rejected as "stale info", so the lease value is computed from a fetched ref (--force-with-lease=<ref>:<sha>), with a plain create when the branch does not exist yet.Failure visibility — the wall for the wall
Nothing watches a scheduled workflow's conclusion, which is exactly why four consecutive failures were silent. The job now:
An issue was chosen over the alternatives because it is the only option that is actually real here: it needs no push access, no branch-protection exception, and no new credential; it is visible on the repo's issue list without anyone tailing Actions; and it outlives log retention. (
ci_job_execution_verifierfrom #698 will catch the complementary case — jobs defined but never executed — once wired.)Repo settings changed in this lane
can_approve_pull_request_reviewsset totrue(one API call, trivially reversible). Branch protection is untouched:enforce_adminsstands, all five required status checks (ci (0..3),windows) stand, andrequired_approving_review_countis0— so the "approve" half of that setting grants no bypass power whatsoever. No--admin, no protection exception.Verification
Manual dispatch of this branch's workflow — run
30780963496— concluded SUCCESS, every step green:It opened #715 on
chore/stats-refreshcarrying both thechoreandmerge-queuelabels. A second dispatch confirmed the re-run path (defect 3) is now idempotent.Gates run locally, all green:
ci_workflow_lint.py,ci_gate_runability.py,workflow_model_linter.py,verify_gates_wired.py,watcher_linter.py --check,encoding_lint.py,secret_scan.py(CLEAN).Janitor follow-up (not done here — deliberately)
Orphan branches hold regenerated stats that nobody ever saw. Left in place for a janitor pass, not deleted:
chore/stats-refresh-2026-07-30chore/stats-refresh-2026-07-31chore/stats-refresh-2026-08-01chore/stats-refresh-2026-08-02chore/stats-refresh-2026-08-03(created by the diagnostic dispatches during this investigation)🤖 Generated with Claude Code