Repository navigation
Refuse a merge-gate run that examined nothing it could verify - #23
Merged
Merged
Conversation
On 2026-11-03 the gate ran with a token that carried no checks: read, because bot/policy.json had not been widened yet, and six of the ten org jobs still reported success: those orgs had no open Dependabot pull requests, so they never called the API at all. Only the four orgs that happened to have pull requests went red, which read as four isolated org problems rather than one blind control plane. Had the branch sweep left nothing open anywhere, the run would have been ten green jobs over a gate that could not read a single check. A quiet run now proves it could have read a check. With nothing examined the gate exercises the same check-runs endpoint against a real commit and exits non-zero if that read fails, while distinguishing the two cases that are not failures: an account with no repository to probe, and an empty repository answering 409. Where pull requests were examined and every one of them errored, the summary names it as a single systemic failure. This is the "assert non-vacuity first" rule the estate already applies to its other discovery-driven checks, and the same shape as verify-release-publish.py, whose clean answer is also an empty result.
imanimanyara
added a commit
that referenced
this pull request
Nov 8, 2026
) checks: read is enforced on private repositories only, so the positive control added in #23 can report "readable" against a public repo while the gate is blind to every repository it actually works on. The estate's Dependabot pull requests sit overwhelmingly in private repos, so that is not a corner case. The 2026-11-03 run is the proof, and it was misread when #23 was written. The simtabi job examined one pull request and it passed -- on the public simtabi/cli-kit-js, with the same token that answered 403 for every private repository across four other orgs. So "every pull request the gate examined errored" was wrong: the one in a public repo succeeded, and the discriminator was visibility, not the org. Measured against the control as merged: for simtabi and ichava it picks a public repo, so for those two it would have passed while private repos were unreadable. The other six pass only by luck of which repository was pushed most recently, which changes constantly. Sort private-first over a wide enough page, name the visibility that was verified, and where only a public repository was available say that the read does not exercise checks: read. An account with no private repository has no private blindness to detect, so a public probe is complete for it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The merge gate can report success having proven nothing. A run with no open Dependabot pull
requests never calls the checks API, so its exit code says only that the search came back empty —
never that the gate could have read a check had there been one.
This makes a quiet run earn its green: with nothing examined, the gate exercises the same
commits/{sha}/check-runsendpoint against a real commit and exits non-zero if that read fails.Where pull requests were examined and every one errored, the summary names it as one systemic
failure rather than N unrelated ones.
The run this is written against
dependabot-merge.ymlrun 41713055813, 2026-11-03, scheduled. It failed withResource not accessible by integration (HTTP 403)on every pull request it examined, because atthat commit (
a21ad90)bot/policy.jsongranted the job only{contents: write, pull-requests: write}— so the minted token had nochecks: read. That was fixed next morning in#21, and is not what this PR addresses.
What it addresses is how the failure presented:
brandgulp,ichava,laranail,mukoracms,simsangesimtabisimbi-prjs-clttayariaitusentesimbi-prjs-intEvery pull request the gate examined errored, but it read as four isolated org problems. Had the
2026-11-02 branch sweep left nothing open anywhere, the run would have been 10/10 green over a gate
that could not read a single check.
This is the "assert non-vacuity first" rule the estate already applies elsewhere, and the same
shape as
verify-release-publish.py, whose clean answer is likewise an empty result.Not treated as failures
Two empty results genuinely are not permission problems, and
checks_readable()returnsNoneforthem rather than failing: an account with no repository to probe, and an empty repository answering
409.
Verification
Suite: 21 → 27 tests, OK.
The guard was proven to have teeth. Neutering the one line that raises the error count makes
test_quiet_run_FAILS_when_checks_are_unreadablefail —FAILED (failures=1)— and restoring itreturns 27 OK. The negative half is covered too:
test_control_reports_absence_not_just_presenceasserts the control can report
False, so it is not a check that can only ever say "readable".Live dry runs, read-only, no
--apply:All four of those orgs were 100% error rows on 2026-11-03, so this also independently confirms
bot: widen the ceiling to the reads the merge gate needs #21 fixed the permission.
Note
DEPENDABOT_MERGE_APPLYis nowtrue(set 2026-11-04), so the next scheduled run is the firstarmed one. Nothing here changes what the gate merges — only whether it may report success without
having looked.