ci: main health watch -- one alarm issue while main is red, and test heads CI never ran on - #3355
Merged
Merged
Conversation
…heads CI never ran on Two failure modes, one of which the 2026-09-25 incident turns out to have been. A red push run on main is read by nobody; and a merge made by github-actions (dependabot auto-merge) starts no push run at all, because GITHUB_TOKEN events trigger no workflows -- the five broken dependabot merges that day produced no red run, only silence, until #3300's push. scripts/main-health-watch.py, run after every Quality/Containers run on main and hourly: - red: one main-red-alarm issue with the failing jobs, first red / last green commit and the commits between; a new comment only when the failing head changes; closed automatically when both workflows are green again. Cancelled runs are ignored. - untested: main's head older than an hour with no run of a watched workflow gets that workflow dispatched on main (workflow_dispatch is the event GITHUB_TOKEN may start); dispatched runs are judged like push runs. - --before ISO8601 replays any past moment read-only. Replaying 2026-09-25 10:00Z flags 87df7e1 (the sha2 bump) as untested; 06:40Z is inside the grace hour; now is green. 8 tests against a fake gh cover open, dedupe, append, close, cancelled runs, dispatch, the grace period and dispatched runs counting. Closes #3324
Xore
added
enhancement
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files
|
Xore
added a commit
that referenced
this pull request
Sep 26, 2026
#3355 added main-health-watch.yml, which uses workflow_run. That is the first file in the tree to trip zizmor's dangerous-triggers rule, a finding class the ADVISORY set in the #3314 gate predates, so the new gate blocked every PR on it. The workflows concerned are already hardened in the ways that rule is pointing at: every action SHA-pinned, permissions: {} at the workflow default with job-scoped grants, persist-credentials: false, and no untrusted event input consumed. #3313 owns the remaining trigger hardening. Reporting without blocking matches how the other four rules this gate already defers are handled, and keeps a documented ADVISORY entry from silently becoming a hard gate on unrelated work. The rule is still counted and printed, so it stays visible; any rule outside the set still fails the build.
Xore
added a commit
that referenced
this pull request
Sep 26, 2026
…tion and eight truncated names (#3358) * ci: run actionlint and zizmor on every workflow; fix a template injection and eight truncated names .github/actionlint.yaml existed but nothing ran actionlint, and nothing audited the workflows for injection or credential issues. - "Workflow lint (actionlint, #3314)" row, blocking: pinned 1.7.7 with upstream's checksum, SHELLCHECK_OPTS=-S warning to match the repo's high-severity ShellCheck policy (all 8 current findings are info). Also fails on an unquoted `name:` containing " #": YAML reads the rest as a comment, and eight names were shipping truncated ("Sandbox shell tests (#2268,") -- all eight now quoted. - "Workflow security audit (zizmor, #3314)" row: pinned 1.30.1 (no upstream checksum file; pinned on first download). Fails on every medium+ finding except the four rules #3313's hardening owns (unpinned-uses, excessive-permissions, artipacked, self-repository), which are reported until then. - Fixed now: template injection in disk-usage-watch.yml (dispatch input interpolated into run: -> env). Justified suppression for the dependabot bot-condition (github.actor is ANDed with the unspoofable PR author). Measured before: zizmor 93 medium+ findings, 77 of them #3313's scope; after: 0 blocking. Both rows executed exactly as CI runs them: exit 0; a probe workflow with an unquoted "(x, #1)" name fails the lint row. Closes #3314 * ci: treat zizmor dangerous-triggers as advisory until #3313 lands #3355 added main-health-watch.yml, which uses workflow_run. That is the first file in the tree to trip zizmor's dangerous-triggers rule, a finding class the ADVISORY set in the #3314 gate predates, so the new gate blocked every PR on it. The workflows concerned are already hardened in the ways that rule is pointing at: every action SHA-pinned, permissions: {} at the workflow default with job-scoped grants, persist-credentials: false, and no untrusted event input consumed. #3313 owns the remaining trigger hardening. Reporting without blocking matches how the other four rules this gate already defers are handled, and keeps a documented ADVISORY entry from silently becoming a hard gate on unrelated work. The rule is still counted and printed, so it stays visible; any rule outside the set still fails the build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #3324, and closes a gap the investigation turned up. The 2026-09-25 incident wasn't "main red for 7 hours". No CI ran on those five Dependabot merges at all. Merges made by
github-actions(auto-merge withGITHUB_TOKEN) start no push workflows, somainwas broken and silent until #3300's push. Every future Dependabot auto-merge leavesmainuntested the same way.scripts/main-health-watch.py+main-health-watch.yml(after every Quality/Containers run onmain, and hourly):main-red-alarmissue with the failing jobs, the first-red and last-green commits, and the suspect commits between them. It comments again only when the failing head changes, and closes itself when both workflows are green. Cancelled runs are ignored.main's head is over an hour old and a watched workflow never ran on it, the watch dispatches that workflow onmain.workflow_dispatchis the exceptionGITHUB_TOKENmay start. The next sweep judges the result like any push run.--before ISO8601shows any past moment, read-only.The workflow has empty default permissions; the job gets
issues: write+actions: write(to dispatch) +contents: read. It runs on a GitHub-hosted runner with a SHA-pinned checkout and no persisted credentials.Validation
untested head 87df7e145d (chore(deps): bump sha2 (#3289)), would dispatch both workflows.scripts/tests/test_main_health_watch.py: 8 tests against a fakeghcovering open, dedupe on the same head, append on a new head, close on green, ignored cancelled runs, dispatch, the grace period, and dispatched runs counting.check-doc-paths-exist.pypasses.Security impact
.envfiles were added.