-
Notifications
You must be signed in to change notification settings - Fork 14
docs: add adversarial code review guide #292
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jayaramkr
wants to merge
2
commits into
AgentToolkit:main
Choose a base branch
from
jayaramkr:docs/adversarial-review-guide
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| description: Adversarial, reproduce-by-execution review of a GitHub PR (fan-out sub-agents, verify, draft review) | ||
| argument-hint: <PR#> [repo] [--post] | ||
| allowed-tools: Bash, Read, Grep, Glob, Agent, Write | ||
| --- | ||
|
|
||
| Run an adversarial code review of pull request **#$1**, following `docs/adversarial-review.md`. | ||
|
|
||
| Method — evidence over opinion; every finding carries a reproduction or it doesn't ship. Do NOT report style nits. | ||
|
|
||
| ## Steps | ||
|
|
||
| 1. **Fetch & size the PR.** | ||
| - `gh pr view $1 --repo <REPO> --json title,author,state,mergeable,mergeStateStatus,body,additions,deletions,changedFiles,headRefOid,commits` (default REPO to this repo's upstream/origin; a second arg to this command overrides it). | ||
| - `gh pr view $1 --repo <REPO> --json files --jq '.files[]|"\(.additions)+ \(.deletions)- \(.path)"' | sort -rn` to see the shape. | ||
| - If the PR body claims specific bugs/fixes, note them — they become verification targets. If this is a **re-review**, fetch your prior review (`gh api repos/<REPO>/pulls/$1/reviews`) so each old finding gets a fixed / still-open / regressed verdict. | ||
|
|
||
| 2. **Isolate in a worktree.** Fetch the head into a temp branch and add a worktree under the scratchpad dir; never touch the user's working tree. Clean it up at the end (`git worktree remove --force`, delete the temp branch). | ||
|
|
||
| 3. **Baseline BEFORE judging.** Install extras if the PR needs them, then run the project's lint / type / test commands and record results, so PR-caused breakage is distinguishable from environmental noise. Note the base SHA (`upstream/main`) for `git diff main...HEAD`. | ||
|
|
||
| 4. **Fan out independent skeptics.** Launch 2–3 sub-agents **in parallel** (one message, multiple Agent calls), each scoped to ONE risk surface (e.g. core algorithm / integration & breaking changes / plugins-config-tests-packaging). Give each the sub-agent prompt template from `docs/adversarial-review.md`, filled in for its surface. They must reproduce findings by execution and not duplicate each other. | ||
|
|
||
| 5. **Verify headline claims yourself.** Re-run each agent's most severe finding with your own repro. Only keep what survives. Watch for claims that are wrong *in the author's favor* too (mis-stated breaking changes, "deleted" tests that were merely moved). | ||
|
|
||
| 6. **Synthesize & show the user.** Rank blocker → high → medium → low; separate verified from hypothesized; **credit what's genuinely correct** (a status table is ideal on re-reviews). Present the results and STOP — do not post unless `--post` was passed or the user asks. | ||
|
|
||
| 7. **Post (only when asked).** Build a review with a summary body + inline comments anchored to `file:line` at the head SHA (inline comments must land on diff lines; otherwise put them in the body). Pick the verdict deliberately (`REQUEST_CHANGES` for a real correctness drop or several mediums; else `COMMENT`/`APPROVE`). Submit as the user's own GitHub account with **no AI attribution**. Verify every inline comment anchored, then report the review URL. | ||
|
|
||
| Extra arguments: `$ARGUMENTS` (e.g. a repo override, or `--post` to post without a second confirmation). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Adversarial Code Review | ||
|
|
||
| A method for reviewing pull requests where **every claim carries a reproduction or it doesn't ship**. Optimized for finding real correctness bugs (not style nits) in high-risk changes: security/compliance seams, algorithms, vendored code, async/concurrency, and anything that fails silently. | ||
|
|
||
| ## The method | ||
|
|
||
| Five moves, in order. | ||
|
|
||
| ### 1. Isolate & baseline | ||
| Review in a throwaway git worktree checked out at the PR head, so the working tree is never touched: | ||
|
|
||
| ```bash | ||
| git fetch <remote> pull/<PR#>/head:pr-<PR#> | ||
| git worktree add /tmp/review-<PR#> pr-<PR#> | ||
| ``` | ||
|
|
||
| *Before* judging anything, run the project's lint / type / test commands and record the result. That's your baseline — it lets you separate PR-caused breakage from pre-existing environmental noise (a missing optional dep, deselected markers, a flaky unrelated test). | ||
|
|
||
| ### 2. Fan out independent skeptics | ||
| Spawn several sub-agents, each scoped to **one** risk surface (core algorithm / integration / tests-and-packaging). They must not see each other's conclusions — independent agreement is signal, not echo. Tell each to **hunt for bugs and reproduce them**, not to summarize the code. Two or three agents is usually right; more just produces overlap. | ||
|
|
||
| ### 3. Reproduce by execution, not eyeballing | ||
| A finding isn't real until it's been run. "Crashes on `tool_calls: None`" is worth nothing until you've watched it throw. Standalone repros (`uv run python -c "..."`, a scratch test, a one-off script) are the currency of the review. | ||
|
|
||
| ### 4. Verify before you report | ||
| Re-run the agents' **headline** claims yourself. Agents are confidently wrong sometimes — catching a claim that's false *in the author's favor* matters as much as catching a bug. Only surface what survives your own repro. | ||
|
|
||
| ### 5. Rank, separate, and credit | ||
| Blockers first, then high / medium / low. Distinguish verified from hypothesized. Say plainly which claimed fixes are genuinely correct so the author doesn't churn on the parts they nailed. A review that only lists faults is a worse review. | ||
|
|
||
| **Through-line: evidence over opinion.** | ||
|
|
||
| ## The reusable sub-agent prompt | ||
|
|
||
| Fill in the bracketed parts, one instance per risk surface. | ||
|
|
||
| ``` | ||
| Adversarial code review. Code is checked out at: <WORKTREE_PATH> | ||
| Base branch is `main` (or `upstream/main`); `git diff main...HEAD -- <path>` | ||
| shows only this PR's changes. | ||
|
|
||
| Focus ONLY on: <SPECIFIC FILES / ONE RISK SURFACE>. | ||
| (Another reviewer owns <the other areas> — do not duplicate.) | ||
|
|
||
| Context: <1–3 sentences on what the code does and any claims the author makes>. | ||
|
|
||
| If this is a RE-review, here are the previously-reported issues that were | ||
| supposedly fixed — verify each is ACTUALLY fixed AND that the fix introduced | ||
| no new bug: | ||
| <list prior findings, or delete this block for a first review> | ||
|
|
||
| Your job: find REAL bugs by reasoning AND by executing. Be adversarial. | ||
| Specifically probe: | ||
| 1. Empty / degenerate / malformed inputs — write and RUN standalone repros | ||
| (`cd <worktree> && uv run python -c "..."`). Try: empty collections, | ||
| single elements, None/""/negative/huge values, missing keys, wrong types. | ||
| 2. Correctness of the core logic — off-by-one, shape/index bugs, wrong math, | ||
| division-by-zero, NaN propagation, silently-swallowed exceptions, | ||
| mutable defaults, incorrect normalization. | ||
| 3. Fail-open vs fail-closed — if this guards something (auth, PII, money, | ||
| deletes), does an unexpected error let bad data THROUGH? State it plainly. | ||
| 4. Integration seams — every call site that reaches the risky path; anything | ||
| that bypasses the intended choke point; breaking changes to public APIs. | ||
| 5. Test quality — are tests real assertions or over-mocked to pass trivially? | ||
| Do they use the REAL dependency or a stub that hides the bug? What is the | ||
| single highest-value MISSING test? | ||
|
|
||
| Rules: | ||
| - Report ONLY findings you verified by reading or executing. No speculation. | ||
| - For each finding: exact file:line, what's wrong, a concrete triggering | ||
| input, and observed-vs-expected. | ||
| - Rank by severity. Distinguish real bugs from cosmetic nits. | ||
| - If a claimed fix is genuinely correct, say so plainly. | ||
| - If the code is solid, say that — do NOT manufacture findings. | ||
| - Be concise. | ||
| ``` | ||
|
|
||
| Two knobs to turn per PR: | ||
| - **Number of agents** = number of genuinely independent risk surfaces (usually 2–3). Overlapping scopes produce echo, not coverage. | ||
| - **"Be adversarial" + "don't manufacture findings"** always appear together. The first pushes them to dig; the second stops them inventing severity to look useful. | ||
|
|
||
| ## Posting the review | ||
|
|
||
| - Draft first; show the human before posting. | ||
| - Structure: a summary body (a **status table crediting what's fixed** is great on re-reviews) plus inline comments anchored to `file:line` at the PR head SHA. | ||
| - Inline comments only attach to lines present in the diff. Files added wholesale are fully anchorable; for a change to an *unchanged* line, anchor to the nearest related diff line or put it in the body. | ||
| - Choose the verdict deliberately: `REQUEST_CHANGES` for a real correctness drop or several mediums; `COMMENT`/`APPROVE` when only nits remain. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add a language identifier to the fenced prompt block.
The unlabeled fence triggers markdownlint MD040. Use
text(or another suitable language) to keep documentation lint-clean.Proposed fix
📝 Committable suggestion
🧰 Tools
🪛 markdownlint-cli2 (0.23.0)
[warning] 37-37: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Source: Linters/SAST tools