Skip to content

Fail the control only on evidence, not on any error - #25

Merged
imanimanyara merged 1 commit into
mainfrom
fix/control-fails-only-on-evidence
Nov 8, 2026
Merged

imanimanyara merged 1 commit into
mainfrom
fix/control-fails-only-on-evidence

Conversation

@imanimanyara

Copy link
Copy Markdown
Member

Summary

#24's control enumerated the errors that are not permission failures — 409 and the word
"empty" — and the enumeration was incomplete. An empty repository whose default_branch is
nominally main answers 422 "No commit found for SHA" on commits/{ref}/check-runs, while
/commits answers 409. So the shape never matched.

Inverted: fail only where GitHub says the permission is missing (403, or Resource not accessible by integration), treat anything else as inconclusive, keep looking, and return None
if nothing conclusive is found. One shape to recognise, instead of an open-ended list of
exemptions.

Found by running it, immediately after #24 merged

simsange  ok=False  cannot read check runs on simsange/documentation@main:
                    gh api --paginate --slurp: gh: No commit found for SHA: main (HTTP 422)

simsange/documentation is empty — size=0, 0 branches, /commits → 409 — and it is that org's
most recently pushed private repository, so #24's private-first sort selected it. A false
permission failure, and the gate would have gone red for simsange every quiet week.

After this change, all ten owned orgs verify and none reports a false failure:

simtabi   simtabi/somesha-ios@main (private)        tusente         tusente/web-app@main (private)
laranail  laranail/tayari-ui@main (private)         ichava          ichava/ichava.com@main (private)
brandgulp brandgulp.com@main (private)              simsange        simsange/sange@main (public)
tayariai  tayariai/tayariai@main (private)          simbi-prjs-clt  swamari.com@main (private)
mukoracms git-commit-checker@main (private)         simbi-prjs-int  creamer.simtabi.com@main (private)

simsange falls past the empty repository to a public one and is labelled (public), so the
summary states that the read does not exercise checks: read rather than implying it does.

Two existing tests were asserting against the wrong thing

test_quiet_run_FAILS_when_checks_are_unreadable and
test_control_reports_absence_not_just_presence raised FakeGh's generic no route for ...
error, which is not a 403 — so they passed for a reason unrelated to what they claimed to
check, and they are why the 422 case was never caught. Both now raise the real message, and the
fixtures for the real shapes are named constants (DENIED, NO_COMMIT).

Verification

  • Suite 30 → 32, OK.
  • Guard proven: making denied() return True for any error — which is what Probe a private repository, or the control verifies the wrong thing #24 effectively
    did — fails three tests (FAILED (failures=3)), including the new 422 case; restoring returns 32
    OK.
  • Both directions covered: test_an_unlistable_account_is_inconclusive_unless_denied asserts
    None on a 404 and False on a 403, so the control can still report a genuine denial.

Note

This is the third pass on this control (#23 added it, #24 made it probe the right repository, this
makes it fail for the right reason). Each correction came from running it against the real estate
rather than from re-reading it — and each previous version would have reported health while
something was wrong.

#24's control enumerated the errors that are NOT permission failures --
409 and the word "empty" -- and the enumeration was incomplete. An empty
repository whose default_branch is nominally main answers 422 "No commit
found for SHA" on commits/{ref}/check-runs while /commits answers 409, so
the shape never matched.

Measured live after #24 merged: simsange reported
  ok=False cannot read check runs on simsange/documentation@main:
          gh: No commit found for SHA: main (HTTP 422)
because its newest private repository is the empty simsange/documentation.
That is a false permission failure, and the gate would have gone red for
simsange every quiet week.

Inverted: fail only where GitHub says the permission is missing -- 403, or
"Resource not accessible by integration" -- and treat anything else as
inconclusive, keep looking, and return None if nothing conclusive is
found. One shape to recognise instead of an open-ended list of exemptions.

Two existing tests were asserting against the wrong thing and are fixed
here: they raised FakeGh's generic "no route for ..." error, which is not a
403, so they passed for a reason unrelated to what they claimed to check.
They now raise the real message.
Copilot AI balanced review requested due to automatic review settings November 8, 2026 14:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@imanimanyara
imanimanyara merged commit 1e31bbf into main Nov 8, 2026
6 checks passed
@imanimanyara
imanimanyara deleted the fix/control-fails-only-on-evidence branch November 14, 2026 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants