Skip to content

Refuse a merge-gate run that examined nothing it could verify - #23

Merged
imanimanyara merged 1 commit into
mainfrom
fix/merge-gate-non-vacuity
Nov 8, 2026
Merged

imanimanyara merged 1 commit into
mainfrom
fix/merge-gate-non-vacuity

Conversation

@imanimanyara

Copy link
Copy Markdown
Member

Summary

The merge gate can report success having proven nothing. A run with no open Dependabot pull
requests never calls the checks API, so its exit code says only that the search came back empty —
never that the gate could have read a check had there been one.

This makes a quiet run earn its green: with nothing examined, the gate exercises the same
commits/{sha}/check-runs endpoint against a real commit and exits non-zero if that read fails.
Where pull requests were examined and every one errored, the summary names it as one systemic
failure rather than N unrelated ones.

The run this is written against

dependabot-merge.yml run 41713055813, 2026-11-03, scheduled. It failed with
Resource not accessible by integration (HTTP 403) on every pull request it examined, because at
that commit (a21ad90) bot/policy.json granted the job only {contents: write, pull-requests: write} — so the minted token had no checks: read. That was fixed next morning in
#21, and is not what this PR addresses.

What it addresses is how the failure presented:

org open Dependabot PRs job
brandgulp, ichava, laranail, mukoracms, simsange 0 green — never called the API
simtabi 1 green
simbi-prjs-clt 24 failed
tayariai 24 failed
tusente 19 failed
simbi-prjs-int 6 failed

Every pull request the gate examined errored, but it read as four isolated org problems. Had the
2026-11-02 branch sweep left nothing open anywhere, the run would have been 10/10 green over a gate
that could not read a single check.

This is the "assert non-vacuity first" rule the estate already applies elsewhere, and the same
shape as verify-release-publish.py, whose clean answer is likewise an empty result.

Not treated as failures

Two empty results genuinely are not permission problems, and checks_readable() returns None for
them rather than failing: an account with no repository to probe, and an empty repository answering
409.

Verification

  • Suite: 21 → 27 tests, OK.

  • The guard was proven to have teeth. Neutering the one line that raises the error count makes
    test_quiet_run_FAILS_when_checks_are_unreadable fail — FAILED (failures=1) — and restoring it
    returns 27 OK. The negative half is covered too: test_control_reports_absence_not_just_presence
    asserts the control can report False, so it is not a check that can only ever say "readable".

  • Live dry runs, read-only, no --apply:

    laranail        → 0 open PRs; "checks API confirmed readable (verified on laranail/tayari-ui@main)"; exit 0
    simbi-prjs-clt  → 18 PRs, 0 error rows, exit 0
    tusente         → 19 PRs, 0 error rows, exit 0
    tayariai        → 17 PRs, 0 error rows, exit 0
    simbi-prjs-int  →  6 PRs, 0 error rows, exit 0
    

    All four of those orgs were 100% error rows on 2026-11-03, so this also independently confirms
    bot: widen the ceiling to the reads the merge gate needs #21 fixed the permission.

Note

DEPENDABOT_MERGE_APPLY is now true (set 2026-11-04), so the next scheduled run is the first
armed one. Nothing here changes what the gate merges — only whether it may report success without
having looked.

On 2026-11-03 the gate ran with a token that carried no checks: read,
because bot/policy.json had not been widened yet, and six of the ten org
jobs still reported success: those orgs had no open Dependabot pull
requests, so they never called the API at all. Only the four orgs that
happened to have pull requests went red, which read as four isolated org
problems rather than one blind control plane. Had the branch sweep left
nothing open anywhere, the run would have been ten green jobs over a gate
that could not read a single check.

A quiet run now proves it could have read a check. With nothing examined
the gate exercises the same check-runs endpoint against a real commit and
exits non-zero if that read fails, while distinguishing the two cases that
are not failures: an account with no repository to probe, and an empty
repository answering 409. Where pull requests were examined and every one
of them errored, the summary names it as a single systemic failure.

This is the "assert non-vacuity first" rule the estate already applies to
its other discovery-driven checks, and the same shape as
verify-release-publish.py, whose clean answer is also an empty result.
Copilot AI balanced review requested due to automatic review settings November 8, 2026 12:54

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@imanimanyara
imanimanyara merged commit 35b110a into main Nov 8, 2026
6 checks passed
imanimanyara added a commit that referenced this pull request Nov 8, 2026
)

checks: read is enforced on private repositories only, so the positive
control added in #23 can report "readable" against a public repo while the
gate is blind to every repository it actually works on. The estate's
Dependabot pull requests sit overwhelmingly in private repos, so that is
not a corner case.

The 2026-11-03 run is the proof, and it was misread when #23 was written.
The simtabi job examined one pull request and it passed -- on the public
simtabi/cli-kit-js, with the same token that answered 403 for every
private repository across four other orgs. So "every pull request the gate
examined errored" was wrong: the one in a public repo succeeded, and the
discriminator was visibility, not the org.

Measured against the control as merged: for simtabi and ichava it picks a
public repo, so for those two it would have passed while private repos
were unreadable. The other six pass only by luck of which repository was
pushed most recently, which changes constantly.

Sort private-first over a wide enough page, name the visibility that was
verified, and where only a public repository was available say that the
read does not exercise checks: read. An account with no private repository
has no private blindness to detect, so a public probe is complete for it.
@imanimanyara
imanimanyara deleted the fix/merge-gate-non-vacuity branch November 14, 2026 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants