Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions plugins/audits/references/config-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,14 @@ Read with whatever is convenient — the `Read` tool, or `jq` for a single value
"depsPrCap": 3, // audit-deps: max PRs per run (a batched routine-bump PR counts as one)
"depsIssueCap": 5, // audit-deps: max issues per run

// OPTIONAL extra label names applied to every PR (not issue) any audit opens, alongside the
// audit's own category label and labels.automated, wherever those are applied (create, the
// follow-up `gh pr edit`, the REST fallback). Use it for a label that hands the PR to
// downstream automation, e.g. ["foreman:watch"] so a response loop answers its review.
// Each must already exist; the audits check once per run, drop a missing one and report it,
// and never create labels (audits plugin: references/extra-pr-labels.md). Default [] if absent.
"extraPrLabels": [],

// audit-architecture: oversized-file line-count threshold, shared across every language
// block (Python, TypeScript, and the language-generic fallback). Raise it on a repo with
// many large-but-legitimate files rather than let the category flag dozens of files and
Expand Down
30 changes: 30 additions & 0 deletions plugins/audits/references/extra-pr-labels.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# `audits.extraPrLabels`: extra labels on audit PRs

`config.audits.extraPrLabels` (default `[]`) lists labels every audit applies to every **PR** it
opens, alongside its category label and `config.labels.automated`. Issues never get them. The
usual entry is a label that hands the PR to downstream automation, such as `foreman:watch`.

Every audit that opens PRs follows the same three rules.

1. **Check they exist, once per run, before the first PR.** When the list is absent or empty,
skip this check and every report line about it. Otherwise run
`gh api "repos/<config.repo>/labels" --paginate --jq '.[].name'` and drop any
`extraPrLabels` entry it does not list. Name each dropped label once in the run report. Never
create a label. If the read itself fails (non-zero exit, an error body, or no output at all),
that is not an empty list: fail closed per rule 1 of
[`gh-rest-fallbacks.md`](gh-rest-fallbacks.md). Apply none of the extra labels, open PRs with
the category labels only, and report "extra labels not applied: label check failed", never
that the labels are missing. This matters for three reasons: `gh pr create --label` fails outright on an
unknown name after the branch is already pushed; a single `gh pr edit --add-label …` fails as a
Comment thread
allenhutchison marked this conversation as resolved.
Resolved
whole, so one missing name would also cost the category and `automated` labels; and the REST
Comment thread
allenhutchison marked this conversation as resolved.
Resolved
`POST /issues/{n}/labels` call **creates** a missing label as a side effect.
2. **Apply the survivors wherever the category label is applied**: the `gh pr create` call, the
follow-up `gh pr edit`, or the REST labels call in
[`gh-rest-fallbacks.md`](gh-rest-fallbacks.md). When the skill delegates to `create-pr`, tell
it to apply them on its own create call, exactly as it applies the category labels.
3. **Ordering on the REST fallback.** These are routing labels, not review-gating ones: `foreman`
finds watched PRs by listing open PRs that carry the label on every tick, not from the
`opened` webhook, so a label that lands a moment after `opened` is still picked up. Create the
PR non-draft and label it immediately after. If a repo puts a label here that a review bot
reads at open time, it belongs in the draft-then-stop branch of that reference instead, and
this key is the wrong place for it.
14 changes: 11 additions & 3 deletions plugins/audits/skills/audit-architecture/SKILL.md

@vycari-review vycari-review Bot Nov 8, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · VR-e850eb06 · The fix for the watch-label race rests on an unverified claim about foreman, so the late-label path still loses the PR

On the REST path the watch label is still added just after the PR opens. That is safe only if foreman finds watched PRs by polling, which the author asserts but this repo cannot show.

Reply Fixed in <sha> or Disputed: with a reason.

Details for agents: evidence, failure scenario, verification, proving test

The round-f6b220f finding said that on the non-draft REST path, a foreman:watch label posted after the opened webhook can be missed by the response loop. The fix does not add a draft or ordering step. It rewrites rule 3 to assert, as fact, that foreman 'finds watched PRs by listing open PRs that carry the label on every tick, not from the opened webhook'. Nothing in this repo supports that: the only mention of foreman is config examples and the build-record marker, and foreman lives in another repository (Vycari/foreman). The reference then directs the agent to create the PR non-draft and label it immediately after, and says a label a review bot reads at open time 'belongs in the draft-then-stop branch... this key is the wrong place for it'. The config-schema example for this very key is foreman:watch 'so a response loop answers its review'. So the doc uses an unverified polling claim to classify the example label as routing-only, and the prior race stays open.

Evidence: extra-pr-labels.md:20-25 (the rule 3 polling claim and the 'wrong place' sentence). audit-architecture/SKILL.md:239-248 implements it: non-draft, label immediately after. gh-rest-fallbacks.md:325-337 says the review-gating vs routing distinction is the caller's call and offers the draft branch for labels that must be in place before review starts. I searched the checkout for any foreman polling implementation or design doc and found none, so the claim is unverified here. The config examples (plugins/audits/references/config-schema.md:114) use foreman:watch as the example for this key.

Failure scenario

Audit-architecture runs in a repo where GraphQL is blocked and extraPrLabels is ["foreman:watch"]. It POSTs /pulls with draft:false, the opened webhook fires, and a review or response loop reads labels when it handles opened. The agent then POSTs the foreman:watch label. If foreman handles only the opened event for this PR, the PR is never watched. Nothing re-triggers on the later label event, the report says the label was applied, and the PR sits unanswered.

How this was verified (read-only-reasoning)

Read rule 3 and the audit-architecture ordering text in the diff. The foreman source is not in this checkout, so the claim cannot be checked. Reachable from: audit-architecture REST fallback.

Proving test

Doc-level: a reviewer or foreman maintainer must show that foreman re-lists open PRs by label on each tick (the claim in extra-pr-labels.md:20-22). Behavioural: for a blocked-GraphQL run with extraPrLabels ["foreman:watch"], assert the PR is either opened as draft with the label then stopped for a human, or that the watch label is present at open time. On head the non-draft late-label path is the one the reference prescribes.

lens spec-adversary

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disputed: foreman does not pick up watched PRs from the opened webhook. reconcile() in foreman/src/foreman/reconcile.py calls gh.list_watched(repo, label) on every tick, listing open PRs that carry the label, so a label added a moment after open is seen on the next tick. foreman:watch is a routing label, not a review gate. In f1f0fc9 the new references/extra-pr-labels.md says so, and says a label a bot reads at open time belongs in the draft-then-stop branch instead.


Generated by Claude Code

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Held @ f1f0fc9 after the dispute: Restated by a finding at this head. Left for Greptile to arbitrate.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seen again @ f1f0fc9: linked to this finding. Also raised at plugins/audits/references/extra-pr-labels.md lines 20–25.

Comment thread
vycari-review[bot] marked this conversation as resolved.
Resolved
Comment thread
vycari-review[bot] marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Before doing anything else, load the repo config (see [`../../references/config-
> This repo has no `.claude/maintainerd.json`. Run `/bootstrap` to generate it, then re-run me.

Do not guess values or hardcode another repo's settings.
3. Read the keys this skill uses: `config.repo`, `config.defaultBranch`, `config.language`, `config.paths.source`, `config.paths.tests`, `config.paths.skillsDir`, `config.commands.format`/`lint`/`build`/`test`, `config.guidelines.invariants`, `config.guidelines.coding`, `config.labels.architecture`, `config.labels.automated`, `config.audits.prCap`, `config.audits.issueCap`, `config.audits.oversizedLines`, `config.audits.promoteThreshold`, `config.audits.promoteLookbackDays`. Treat any `null` command as *"this repo has no such step — skip it"*. For any key that is **absent**, fall back to the documented default (caps default to 3 PRs / 5 issues; the oversized-files line-count threshold defaults to 500; pattern-promotion defaults to a threshold of 3 within 90 days) and mention the fallback in your run report.
3. Read the keys this skill uses: `config.repo`, `config.defaultBranch`, `config.language`, `config.paths.source`, `config.paths.tests`, `config.paths.skillsDir`, `config.commands.format`/`lint`/`build`/`test`, `config.guidelines.invariants`, `config.guidelines.coding`, `config.labels.architecture`, `config.labels.automated`, `config.audits.extraPrLabels` (default `[]`), `config.audits.prCap`, `config.audits.issueCap`, `config.audits.oversizedLines`, `config.audits.promoteThreshold`, `config.audits.promoteLookbackDays`. Treat any `null` command as *"this repo has no such step — skip it"*. For any key that is **absent**, fall back to the documented default (caps default to 3 PRs / 5 issues; the oversized-files line-count threshold defaults to 500; pattern-promotion defaults to a threshold of 3 within 90 days) and mention the fallback in your run report.

Throughout this skill, `config.<path>` refers to a value from that file. In the detection commands below, `config.paths.source` stands for your configured source root (e.g. `src/pepper/` or `src/`) and `config.paths.tests` your test root — substitute the literal value when you run them.

Expand Down Expand Up @@ -202,6 +202,7 @@ git push -u origin "$SLUG"
gh pr create --repo <config.repo> \
--title "<prefix>: <one-line description>" \
--label <config.labels.architecture> --label <config.labels.automated> \
--label <each config.audits.extraPrLabels entry that passed the existence check> \
--body "$(cat <<'EOF'
## Summary

Expand Down Expand Up @@ -229,12 +230,19 @@ EOF

Match the conventional prefix to the work (`refactor:`, `chore:`, `fix:`).

Before the first PR of the run, run the extra-label existence check (rule 1 of the reference below).
Repeat `--label` once per `config.audits.extraPrLabels` entry that passed the once-per-run
existence check (omit the line when none did). They go on PRs only, never on the issues in step 6,
and they go on the PR whether you open it here or delegate to `create-pr`. The check, the
delegation and the REST ordering are in
[`../../references/extra-pr-labels.md`](../../references/extra-pr-labels.md).

**When GraphQL is blocked**, `gh pr create` is 403. Push the branch the same way, then open the PR
with `POST /repos/{owner}/{repo}/pulls` — **Create a PR** in
[`../../references/gh-rest-fallbacks.md`](../../references/gh-rest-fallbacks.md). Labels are a
separate `POST /issues/{n}/labels` call there, so `config.labels.architecture` and
`config.labels.automated` land a moment after the `opened` webhook rather than with it. For these
two labels that is fine — they are routing labels, not review-gating ones — so create the PR
`config.labels.automated` (and the checked `config.audits.extraPrLabels` entries) land a moment
after the `opened` webhook rather than with it. For these labels that is fine — they are routing labels, not review-gating ones — so create the PR
non-draft and apply them immediately after, and note the ordering in the report. If this repo's
labels *do* gate a review bot, take the second branch that reference describes: leave the PR a
draft, label it, and stop with the PR URL and the one action a human must take. `gh pr ready` is
Expand Down
6 changes: 5 additions & 1 deletion plugins/audits/skills/audit-deps/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ Keys this skill uses:
- `config.labels.dependencies` — label applied to every PR/issue this skill opens (default
`dependencies`).
- `config.labels.automated` — also applied (default `automated`).
- `config.audits.extraPrLabels` — extra labels applied to every PR (not issue) this skill opens,
including through `create-pr` (default `[]`). Run the existence check once, before this run's
first PR, and drop missing ones; never create one. See [`../../references/extra-pr-labels.md`](../../references/extra-pr-labels.md).
- `config.audits.depsPrCap` / `config.audits.depsIssueCap` — per-run caps (default **3 / 5**).
- `config.guidelines.*` — if the repo documents a **dependency or license policy** (an allowed-license
allowlist, a "pin exactly / never auto-bump X" rule, a "stay on major N of Y" constraint), honor
Expand Down Expand Up @@ -155,7 +158,8 @@ Everything else is its **own** unit of work:

For every PR: branch `deps-<category>-<descriptor>` off `config.defaultBranch`, **delegate to
`create-pr` if installed** (else run `config.commands.*` as pre-flight; no `--no-verify`), label
`config.labels.dependencies` + `config.labels.automated`, never auto-merge. Caps default **3 PRs /
`config.labels.dependencies` + `config.labels.automated` plus every checked `config.audits.extraPrLabels`
entry (pass them to `create-pr` when delegating), never auto-merge. Caps default **3 PRs /
5 issues**; defer overflow to the next run and note it in the report.

### 6. Report
Expand Down
5 changes: 4 additions & 1 deletion plugins/audits/skills/audit-security/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ Before anything else, load the repo config (see
- `config.commands.{format,lint,build,test}` — pre-flight gate before any PR (skip `null` ones).
- `config.labels.security` — label applied to every PR/issue this skill opens (default `security`).
- `config.labels.automated` — also applied (default `automated`).
- `config.audits.extraPrLabels` — extra labels applied to every PR (not issue) this skill opens, including through `create-pr` (default `[]`). Run the existence check once, before this run's first PR, and drop missing ones; never create one. See [`../../references/extra-pr-labels.md`](../../references/extra-pr-labels.md).
- `config.audits.securityPrCap` / `config.audits.securityIssueCap` — per-run caps (default **3 / 5**).
- `config.audits.promoteThreshold` / `config.audits.promoteLookbackDays` — pattern-promotion knobs (default **3** within **90** days). See step 6.
- `config.guidelines.invariants` / `config.guidelines.coding` — **the repo's own security rules.**
Expand Down Expand Up @@ -174,7 +175,9 @@ Process findings **highest-severity first**. For each:
- Branch `sec-<category>-<descriptor>` off `config.defaultBranch`.
- **Delegate to `create-pr` if installed**; else run `config.commands.{format,lint,build,test}`
(skip `null`) as pre-flight before pushing. No `--no-verify`. Never auto-merge.
- Label `config.labels.security` + `config.labels.automated`. For a dependency bump, the PR body
- Label `config.labels.security` + `config.labels.automated`, plus every
`config.audits.extraPrLabels` entry that passed the existence check (pass them to `create-pr`
when delegating). For a dependency bump, the PR body
must name the CVE/GHSA, the version delta, and that tests pass.
- **Issue** (counts against `config.audits.securityIssueCap`) otherwise — describe the vuln, its
impact, evidence (`file:line`), and a proposed fix; don't write the code.
Expand Down
4 changes: 3 additions & 1 deletion plugins/audits/skills/audit-tests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ Before anything else, load the repo config (see
- `config.commands.coverage` — coverage command. **If `null`, skip the coverage-gap category entirely** (don't invent one).
- `config.commands.format` / `config.commands.lint` — pre-flight gates (skip any that are `null`).
- `config.labels.testQuality` / `config.labels.automated` — labels applied to every PR/issue this skill opens.
- `config.audits.extraPrLabels` — extra labels applied to every PR (not issue) this skill opens, including through `create-pr` (default `[]`). Run the existence check once, before this run's first PR, and drop missing ones; never create one. See [`../../references/extra-pr-labels.md`](../../references/extra-pr-labels.md).
- `config.audits.testPrCap` / `config.audits.testIssueCap` — per-run caps (default **2 / 2** if absent; note the fallback in your report).
- `config.audits.promoteThreshold` / `config.audits.promoteLookbackDays` — pattern-promotion knobs (default **3** within **90** days if absent). See step 7.
- `config.guidelines.testing` — **the repo's own test conventions.** Read this file in full; it holds the repo-specific rules (the DB/fixture pattern, what may and may not be mocked, isolation rules, assertion expectations, redundant framework boilerplate, naming) that this skill checks the suite against. If it's missing or still full of `TODO` markers, say so in your report and proceed with the language-generic checks only.
Expand Down Expand Up @@ -207,7 +208,8 @@ git push -u origin "$SLUG"
gh pr create --repo <config.repo> --base <config.defaultBranch> \
--title "test: <one-line description>" \
--body "<template below>"
gh pr edit <NUM> --add-label <config.labels.testQuality> --add-label <config.labels.automated>
gh pr edit <NUM> --add-label <config.labels.testQuality> --add-label <config.labels.automated> \
--add-label <each config.audits.extraPrLabels entry that passed the existence check>
```

PR body:
Expand Down
8 changes: 8 additions & 0 deletions plugins/auto-dev/references/config-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,14 @@ Read with whatever is convenient — the `Read` tool, or `jq` for a single value
"depsPrCap": 3, // audit-deps: max PRs per run (a batched routine-bump PR counts as one)
"depsIssueCap": 5, // audit-deps: max issues per run

// OPTIONAL extra label names applied to every PR (not issue) any audit opens, alongside the
// audit's own category label and labels.automated, wherever those are applied (create, the
// follow-up `gh pr edit`, the REST fallback). Use it for a label that hands the PR to
// downstream automation, e.g. ["foreman:watch"] so a response loop answers its review.
// Each must already exist; the audits check once per run, drop a missing one and report it,
// and never create labels (audits plugin: references/extra-pr-labels.md). Default [] if absent.
"extraPrLabels": [],

// audit-architecture: oversized-file line-count threshold, shared across every language
// block (Python, TypeScript, and the language-generic fallback). Raise it on a repo with
// many large-but-legitimate files rather than let the category flag dozens of files and
Expand Down
8 changes: 8 additions & 0 deletions plugins/core/references/config-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,14 @@ Read with whatever is convenient — the `Read` tool, or `jq` for a single value
"depsPrCap": 3, // audit-deps: max PRs per run (a batched routine-bump PR counts as one)
"depsIssueCap": 5, // audit-deps: max issues per run

// OPTIONAL extra label names applied to every PR (not issue) any audit opens, alongside the
// audit's own category label and labels.automated, wherever those are applied (create, the
// follow-up `gh pr edit`, the REST fallback). Use it for a label that hands the PR to
// downstream automation, e.g. ["foreman:watch"] so a response loop answers its review.
// Each must already exist; the audits check once per run, drop a missing one and report it,
// and never create labels (audits plugin: references/extra-pr-labels.md). Default [] if absent.
"extraPrLabels": [],

// audit-architecture: oversized-file line-count threshold, shared across every language
// block (Python, TypeScript, and the language-generic fallback). Raise it on a repo with
// many large-but-legitimate files rather than let the category flag dozens of files and
Expand Down
8 changes: 8 additions & 0 deletions plugins/deps-flow/references/config-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,14 @@ Read with whatever is convenient — the `Read` tool, or `jq` for a single value
"depsPrCap": 3, // audit-deps: max PRs per run (a batched routine-bump PR counts as one)
"depsIssueCap": 5, // audit-deps: max issues per run

// OPTIONAL extra label names applied to every PR (not issue) any audit opens, alongside the
// audit's own category label and labels.automated, wherever those are applied (create, the
// follow-up `gh pr edit`, the REST fallback). Use it for a label that hands the PR to
// downstream automation, e.g. ["foreman:watch"] so a response loop answers its review.
// Each must already exist; the audits check once per run, drop a missing one and report it,
// and never create labels (audits plugin: references/extra-pr-labels.md). Default [] if absent.
"extraPrLabels": [],

// audit-architecture: oversized-file line-count threshold, shared across every language
// block (Python, TypeScript, and the language-generic fallback). Raise it on a repo with
// many large-but-legitimate files rather than let the category flag dozens of files and
Expand Down
8 changes: 8 additions & 0 deletions plugins/journal/references/config-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,14 @@ Read with whatever is convenient — the `Read` tool, or `jq` for a single value
"depsPrCap": 3, // audit-deps: max PRs per run (a batched routine-bump PR counts as one)
"depsIssueCap": 5, // audit-deps: max issues per run

// OPTIONAL extra label names applied to every PR (not issue) any audit opens, alongside the
// audit's own category label and labels.automated, wherever those are applied (create, the
// follow-up `gh pr edit`, the REST fallback). Use it for a label that hands the PR to
// downstream automation, e.g. ["foreman:watch"] so a response loop answers its review.
// Each must already exist; the audits check once per run, drop a missing one and report it,
// and never create labels (audits plugin: references/extra-pr-labels.md). Default [] if absent.
"extraPrLabels": [],

// audit-architecture: oversized-file line-count threshold, shared across every language
// block (Python, TypeScript, and the language-generic fallback). Raise it on a repo with
// many large-but-legitimate files rather than let the category flag dozens of files and
Expand Down
8 changes: 8 additions & 0 deletions plugins/repo-ops/references/config-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,14 @@ Read with whatever is convenient — the `Read` tool, or `jq` for a single value
"depsPrCap": 3, // audit-deps: max PRs per run (a batched routine-bump PR counts as one)
"depsIssueCap": 5, // audit-deps: max issues per run

// OPTIONAL extra label names applied to every PR (not issue) any audit opens, alongside the
// audit's own category label and labels.automated, wherever those are applied (create, the
// follow-up `gh pr edit`, the REST fallback). Use it for a label that hands the PR to
// downstream automation, e.g. ["foreman:watch"] so a response loop answers its review.
// Each must already exist; the audits check once per run, drop a missing one and report it,
// and never create labels (audits plugin: references/extra-pr-labels.md). Default [] if absent.
"extraPrLabels": [],

// audit-architecture: oversized-file line-count threshold, shared across every language
// block (Python, TypeScript, and the language-generic fallback). Raise it on a repo with
// many large-but-legitimate files rather than let the category flag dozens of files and
Expand Down
8 changes: 8 additions & 0 deletions plugins/research/references/config-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,14 @@ Read with whatever is convenient — the `Read` tool, or `jq` for a single value
"depsPrCap": 3, // audit-deps: max PRs per run (a batched routine-bump PR counts as one)
"depsIssueCap": 5, // audit-deps: max issues per run

// OPTIONAL extra label names applied to every PR (not issue) any audit opens, alongside the
// audit's own category label and labels.automated, wherever those are applied (create, the
// follow-up `gh pr edit`, the REST fallback). Use it for a label that hands the PR to
// downstream automation, e.g. ["foreman:watch"] so a response loop answers its review.
// Each must already exist; the audits check once per run, drop a missing one and report it,
// and never create labels (audits plugin: references/extra-pr-labels.md). Default [] if absent.
"extraPrLabels": [],

// audit-architecture: oversized-file line-count threshold, shared across every language
// block (Python, TypeScript, and the language-generic fallback). Raise it on a repo with
// many large-but-legitimate files rather than let the category flag dozens of files and
Expand Down
Loading