Repository navigation
feat(audits): audits.extraPrLabels for every PR the audits open #124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
f6b220f
f1f0fc9
8512b3f
084eb73
2a260ee
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # `audits.extraPrLabels`: extra labels on audit PRs | ||
|
|
||
| `config.audits.extraPrLabels` (default `[]`) lists labels every audit applies to every **PR** it | ||
| opens, alongside its category label and `config.labels.automated`. Issues never get them. The | ||
| usual entry is a label that hands the PR to downstream automation, such as `foreman:watch`. | ||
|
|
||
| Every audit that opens PRs follows the same three rules. | ||
|
|
||
| 1. **Check they exist, once per run, before the first PR.** When the list is absent or empty, | ||
| skip this check and every report line about it. Otherwise run | ||
| `gh api "repos/<config.repo>/labels" --paginate --jq '.[].name'` and drop any | ||
| `extraPrLabels` entry it does not list. Name each dropped label once in the run report. Never | ||
| create a label. If the read itself fails (non-zero exit, an error body, or no output at all), | ||
| that is not an empty list: fail closed per rule 1 of | ||
| [`gh-rest-fallbacks.md`](gh-rest-fallbacks.md). Apply none of the extra labels, open PRs with | ||
| the category labels only, and report "extra labels not applied: label check failed", never | ||
| that the labels are missing. This matters for three reasons: `gh pr create --label` fails outright on an | ||
| unknown name after the branch is already pushed; a single `gh pr edit --add-label …` fails as a | ||
| whole, so one missing name would also cost the category and `automated` labels; and the REST | ||
|
allenhutchison marked this conversation as resolved.
Resolved
|
||
| `POST /issues/{n}/labels` call **creates** a missing label as a side effect. | ||
| 2. **Apply the survivors wherever the category label is applied**: the `gh pr create` call, the | ||
| follow-up `gh pr edit`, or the REST labels call in | ||
| [`gh-rest-fallbacks.md`](gh-rest-fallbacks.md). When the skill delegates to `create-pr`, tell | ||
| it to apply them on its own create call, exactly as it applies the category labels. | ||
| 3. **Ordering on the REST fallback.** These are routing labels, not review-gating ones: `foreman` | ||
| finds watched PRs by listing open PRs that carry the label on every tick, not from the | ||
| `opened` webhook, so a label that lands a moment after `opened` is still picked up. Create the | ||
| PR non-draft and label it immediately after. If a repo puts a label here that a review bot | ||
| reads at open time, it belongs in the draft-then-stop branch of that reference instead, and | ||
| this key is the wrong place for it. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2 · On the REST path the watch label is still added just after the PR opens. That is safe only if foreman finds watched PRs by polling, which the author asserts but this repo cannot show. Reply Details for agents: evidence, failure scenario, verification, proving testThe round-f6b220f finding said that on the non-draft REST path, a foreman:watch label posted after the opened webhook can be missed by the response loop. The fix does not add a draft or ordering step. It rewrites rule 3 to assert, as fact, that foreman 'finds watched PRs by listing open PRs that carry the label on every tick, not from the opened webhook'. Nothing in this repo supports that: the only mention of foreman is config examples and the build-record marker, and foreman lives in another repository (Vycari/foreman). The reference then directs the agent to create the PR non-draft and label it immediately after, and says a label a review bot reads at open time 'belongs in the draft-then-stop branch... this key is the wrong place for it'. The config-schema example for this very key is foreman:watch 'so a response loop answers its review'. So the doc uses an unverified polling claim to classify the example label as routing-only, and the prior race stays open. Evidence: extra-pr-labels.md:20-25 (the rule 3 polling claim and the 'wrong place' sentence). audit-architecture/SKILL.md:239-248 implements it: non-draft, label immediately after. gh-rest-fallbacks.md:325-337 says the review-gating vs routing distinction is the caller's call and offers the draft branch for labels that must be in place before review starts. I searched the checkout for any foreman polling implementation or design doc and found none, so the claim is unverified here. The config examples (plugins/audits/references/config-schema.md:114) use foreman:watch as the example for this key. Failure scenario Audit-architecture runs in a repo where GraphQL is blocked and extraPrLabels is ["foreman:watch"]. It POSTs /pulls with draft:false, the opened webhook fires, and a review or response loop reads labels when it handles opened. The agent then POSTs the foreman:watch label. If foreman handles only the opened event for this PR, the PR is never watched. Nothing re-triggers on the later label event, the report says the label was applied, and the PR sits unanswered. How this was verified (read-only-reasoning) Read rule 3 and the audit-architecture ordering text in the diff. The foreman source is not in this checkout, so the claim cannot be checked. Reachable from: audit-architecture REST fallback. Proving test lens spec-adversary
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Disputed: foreman does not pick up watched PRs from the Generated by Claude Code There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Held @ f1f0fc9 after the dispute: Restated by a finding at this head. Left for Greptile to arbitrate. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seen again @ f1f0fc9: linked to this finding. Also raised at plugins/audits/references/extra-pr-labels.md lines 20–25.
vycari-review[bot] marked this conversation as resolved.
Resolved
vycari-review[bot] marked this conversation as resolved.
Resolved
|
Uh oh!
There was an error while loading. Please reload this page.