refactor(#5974): log matched PR numbers in dispatch existing-PR guard skip notice

[ggallen]

Summary

• Include matched PR numbers in the dispatch existing-PR guard skip notice (e.g., Skipping code dispatch for issue #123: found open PR(s) #456, #789) so silent dispatch skips are debuggable.
• Update guard comments to clarify __typename + login filter intent: only the code-agent bot is excluded so it can re-dispatch; other bots (e.g. Renovate) still block.
• Add TestDispatchPRCheckBotFilter alignment test to prevent regression on both reusable-dispatch.yml and the scaffold template.

Note: Issue #5974 lists three items. Items 1 (structured reference pattern via closedByPullRequestsReferences) and 3 (bot-author filter via __typename + login) were already resolved on main prior to this PR — item 1 by commit 9f9a9ca (#5575, 2026-07-24) and item 3 by the same commit's GraphQL migration. The ggallen comment on #5974 reporting a broken bot filter ("Open PR(s) mentioning issue #6246 found") used the old "mentioning" wording removed in that commit, indicating the evidence came from a stale, unsynced per-org dispatch.yml copy (the drift tracked by #6013), not the current GraphQL-based code on main. This PR addresses item 2 (log matched PR numbers) and adds a regression test covering item 3.

Closes #5974

Test plan

• TestDispatchPRCheckBotFilter passes for both reusable-dispatch.yml and scaffold/dispatch.yml
• All internal/scaffold/ tests pass
• Codecov: all modified and coverable lines are covered
• Verify in a live dispatch run that the skip notice includes matched PR numbers — requires per-org resync (Per-org dispatch.yml doesn't propagate reusable-workflow fixes (sync-scaffold gap; per-repo mode unaffected) #6013) for orgs still on the standalone dispatch.yml copy; per-repo mode picks up the change immediately via reusable-dispatch.yml

🤖 Generated with Claude Code

[qodo-code-review]

PR Summary by Qodo

Fix existing-PR guard bot detection via __typename and log matched PRs

🐞 Bug fix 🧪 Tests ⚙️ Configuration changes 🕐 20-40 Minutes

AI Description

• Exclude bot-authored PRs in the existing-PR guard using GraphQL author.__typename.
• Include matched open PR numbers in the skip notice to aid dispatch debugging.
• Add an alignment test to prevent workflow/template regressions.

Diagram

graph TD
A["Dispatch workflows"] --> B["GitHub GraphQL query"] --> C["PR nodes (OPEN)"] --> D["Filter non-bot (__typename)"] --> E{"Open PRs found?"}
E -- "Yes" --> F["Notice includes PR numbers"] --> G["Set skipped=true"]
E -- "No" --> H["Proceed with dispatch"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Maintain login-based bot allow/deny list

• ➕ Can target specific bot accounts while allowing other bots/humans
• ➖ Fragile across REST vs GraphQL login formatting changes (e.g., app/ prefix vs [bot] suffix)
• ➖ Requires ongoing maintenance and can silently break when identities change

2. Include `login` for diagnostics but filter on `__typename`

• ➕ Keeps robust bot detection while preserving extra context for troubleshooting
• ➖ Slightly more data fetched/handled; must ensure no new logic regresses into login matching

Recommendation: Filtering on author.__typename == "Bot" is the most robust approach because it avoids brittle assumptions about bot login string formats. The added notice with matched PR numbers is a low-risk, high-value observability improvement; consider optionally retaining login purely for debugging if future investigations need it, but keep filtering keyed on __typename.

Files changed (3) +54 / -18

Bug fix (2) +18 / -18

reusable-dispatch.ymlSwitch existing-PR guard bot detection to __typename; log matched PR numbers +9/-9

Switch existing-PR guard bot detection to __typename; log matched PR numbers

• Updates the GraphQL query to only request 'author.__typename' and simplifies the jq filter to exclude bot-authored PRs via '__typename'. Formats matched PR numbers and includes them in the skip notice when dispatch is prevented.

.github/workflows/reusable-dispatch.yml

dispatch.ymlMirror __typename-based bot filter and improved skip notice in scaffold workflow +9/-9

Mirror __typename-based bot filter and improved skip notice in scaffold workflow

• Aligns the scaffolded dispatch workflow with the reusable workflow by filtering bot-authored PRs using 'author.__typename' only. Improves the skip notice to print the matched open PR numbers for easier debugging.

internal/scaffold/fullsend-repo/.github/workflows/dispatch.yml

Tests (1) +36 / -0

workflow_call_alignment_test.goAdd alignment test enforcing __typename bot filter and PR-number skip notice +36/-0

Add alignment test enforcing __typename bot filter and PR-number skip notice

• Adds 'TestDispatchPRCheckBotFilter' to assert both workflows request 'author { __typename }', filter bots via 'select(.author.__typename != "Bot")', and include '${LINKED_PRS}' in the skip notice.

internal/scaffold/workflow_call_alignment_test.go

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 2:10 PM UTC · Ended 2:20 PM UTC

Commit: 162f171 · View workflow run →

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Overbroad bot exclusion ✓ Resolved 🐞 Bug ≡ Correctness

Description

The existing-PR guard now filters linked PRs with select(.author.__typename != "Bot"), which
ignores all bot-authored PRs and may allow code dispatch even when a non-fullsend bot already has an
open PR linked to close the issue. This diverges from the documented guidance to combine
__typename with the specific bot login, and can cause duplicate/competing agent PRs and
unnecessary runs.

Code

.github/workflows/reusable-dispatch.yml[357]

+                   | select(.author.__typename != "Bot")]

Relevance

●●● Strong

Close precedent: existing-PR guard previously used __typename+specific bot login to exclude only
agent PRs (#5578).

PR-#5578

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Both dispatch workflows now drop login from the GraphQL query and filter out all bots solely by
__typename, which can ignore non-agent bot PRs. The bot identities doc explicitly calls for
combining __typename with the un-suffixed login to avoid fragile suffix assumptions while still
identifying the intended bot, and the repo already recognizes non-agent bots like Renovate.

.github/workflows/reusable-dispatch.yml[345-358]
internal/scaffold/fullsend-repo/.github/workflows/dispatch.yml[288-301]
docs/contributing/bot-identities.md[15-17]
scripts/check-e2e-authorization.sh[34-36]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The existing-PR guard currently excludes *all* bot-authored PRs (`__typename == "Bot"`) from the linked-PR check. This should exclude only the fullsend code-agent bot PRs (so re-dispatch works) while still blocking dispatch when other bots (e.g., Renovate) have an open PR linked to close the issue.

## Issue Context
`docs/contributing/bot-identities.md` recommends using GraphQL `__typename == "Bot"` **plus** the un-suffixed bot `login` to identify specific GitHub App identities.

## Fix Focus Areas
- .github/workflows/reusable-dispatch.yml[345-358]
- internal/scaffold/fullsend-repo/.github/workflows/dispatch.yml[288-301]
- internal/scaffold/workflow_call_alignment_test.go[693-725]

## Suggested change
1. Add `login` back to the GraphQL selection:
  - `author { login __typename }`
2. Update the jq predicate to exclude only the fullsend coder bot:
  - `select(.author.__typename != "Bot" or .author.login != "fullsend-ai-coder")`
  This keeps PRs authored by other bots in the guard.
3. Update `TestDispatchPRCheckBotFilter` accordingly (it currently enforces `author { __typename }` only, which would prevent the safer bot-specific filter).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context

✅ Compliance rules (platform): 61 rules

✅ Skills: writing-user-docs, writing-adrs

Tip of the day

💡 Did you know, you can turn on the rule miner and Qodo learns your standards from review history

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 2:21 PM UTC · Completed 2:38 PM UTC

Commit: 7c05bd2 · View workflow run →

[fullsend-ai-review]

Review

Findings

Medium

• [protected-path] .github/workflows/reusable-dispatch.yml — This PR modifies a file under the protected .github/ path. The PR links to issue Code dispatch existing-PR guard uses bare substring search, causing false-positive blocks #5974 and explains the rationale (improving logging in the dispatch existing-PR guard, addressing issue item 2). Human approval is always required for protected-path changes, regardless of context.

Previous run

Review

Findings

Medium

• [protected-path] .github/workflows/reusable-dispatch.yml — This PR modifies a file under the protected .github/ path. The PR links to issue Code dispatch existing-PR guard uses bare substring search, causing false-positive blocks #5974 and explains the rationale for the change. Human approval is always required for protected-path changes, regardless of context.

• [type-mismatch] PR metadata — PR title uses fix(#5974): but the change is not a user-visible bug fix. The PR improves diagnostic logging by adding PR numbers to skip notices, updates comments for clarity, and adds a regression test. The PR body acknowledges that the core bug (items 1 and 3) was already resolved on main; this PR addresses item 2 (skip notice lacking PR numbers) plus a regression test. Per COMMITS.md, fix is reserved for user-visible bug fixes; improving diagnostic logging is refactor or chore.
  Remediation: Change PR title to refactor(#5974): log matched PR numbers in dispatch existing-PR guard skip notice or chore(#5974): ....

Low

• [message-format-consistency] .github/workflows/reusable-dispatch.yml:364 — Notice message format inconsistent with codebase patterns. The changed format Skipping code dispatch for issue #N: found open PR(s) ... uses a colon separator, while other notices in the same file use an em-dash separator and subject-first ordering (e.g., Stage X skipped — role Y not in configured roles).
  Remediation: Consider using the established em-dash separator: Skipping code dispatch — issue #${ISSUE_NUMBER} has open PR(s) ${LINKED_PRS}.

Previous run (2)

Review

Findings

Medium

• [protected-path] .github/workflows/reusable-dispatch.yml — This PR modifies a file under the protected .github/ path. The PR links to issue Code dispatch existing-PR guard uses bare substring search, causing false-positive blocks #5974 and explains the rationale for the change. Human approval is always required for protected-path changes, regardless of context.

Previous run (3)

Review

Findings

Medium

• [protected-path] .github/workflows/reusable-dispatch.yml — This PR modifies a file under the protected .github/ path. The PR links to issue Code dispatch existing-PR guard uses bare substring search, causing false-positive blocks #5974 and explains the rationale for the change. Human approval is always required for protected-path changes, regardless of context.

Previous run (4)

Review

Findings

Medium

• [protected-path] .github/workflows/reusable-dispatch.yml — This PR modifies a file under the protected .github/ path. The PR links to issue Code dispatch existing-PR guard uses bare substring search, causing false-positive blocks #5974 and explains the rationale for the change. Human approval is always required for protected-path changes, regardless of context.

• [misleading-title] — The PR title "use __typename for bot detection" implies this PR introduces __typename-based bot detection. However, the select(.author.__typename != "Bot" or .author.login != "fullsend-ai-coder") filter is unchanged by this diff — it already existed on main. The actual changes are: comment text updates explaining the filter's intent, LINKED_PRS output reformatted to include # prefixes and comma-join, and an improved ::notice:: message with matched PR numbers.
  Remediation: Consider rewording the title to reflect the actual changes, e.g.: fix(#5974): log matched PR numbers in dispatch existing-PR guard skip notice.

Low

• [incomplete-scope] — Issue Code dispatch existing-PR guard uses bare substring search, causing false-positive blocks #5974 lists three items: (1) structured reference pattern, (2) log matched PR numbers, (3) fix bot-author filter. This PR addresses item 2 and adds a regression test for item 3. Items 1 and 3 were already resolved on main prior to this PR (item 1 via closedByPullRequestsReferences GraphQL, item 3 via the existing __typename filter). The Closes #5974 claim is reasonable, but the PR body could note that items 1 and 3 were pre-resolved.

---

Labels: PR modifies the dispatch existing-PR guard in both reusable-dispatch.yml and the scaffold template

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 2:41 PM UTC · Completed 2:55 PM UTC

Commit: 90fc907 · View workflow run →

[waynesun09]

MEDIUM: Unchecked "live dispatch run" test-plan item may not be satisfiable by merging alone (internal/scaffold/fullsend-repo/.github/workflows/dispatch.yml)

The PR description's test plan has one unchecked item: "Verify in a live dispatch run that the skip notice includes matched PR numbers." Issue #6013 (open) documents that fullsend-ai's own org repos are still on the deprecated per-org installation mode, where the executing .fullsend/dispatch.yml is a standalone copy of this scaffold template that does not auto-sync — it requires an explicit resync per #6013's described propagation gap. Merging this PR updates the template at internal/scaffold/fullsend-repo/.github/workflows/dispatch.yml, but the actually-running per-org dispatch.yml (including fullsend-ai's own dogfooding org) will keep emitting the old notice text until that resync happens. The PR doesn't mention scheduling or performing a resync, so the unchecked checkbox could be mistaken for "will be true once merged" when it isn't.

Suggestion: Caveat or check off the live-verification item to note it requires a per-org resync first (per #6013), or explicitly trigger/track that resync alongside merging this PR so the fix reaches production dispatch runs.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 3:02 PM UTC · Completed 3:15 PM UTC

Commit: d8b5471 · View workflow run →

[waynesun09]

MEDIUM: PR asserts item 3 (bot-author filter) "already resolved" without reconciling ggallen's same-day contradicting evidence on #5974

The PR body states as fact that issue #5974 items 1 and 3 (bot-author filter via __typename) "were already resolved on main prior to this PR." However, issue #5974 has a MEMBER comment from ggallen (2026-08-15T13:22:45Z, ~40 min before this PR's commits) titled "Confirmed: bot-author filter is broken," citing a live dispatch run log reading "Open PR(s) mentioning issue #6246 found — skipping code dispatch" as evidence the bot filter never fires, and recommending .author.is_bot. The triage-agent's follow-up comment explicitly lists .github/workflows/reusable-dispatch.yml (the file this PR edits) as an "Affected location" needing the fix upgraded from "verify" to "fix."

I independently verified via git log on reusable-dispatch.yml that the exact phrase "Open PR(s) mentioning issue #N found" (using gh pr list/MENTIONING_PRS) was removed from main on 2026-07-24 by commit 9f9a9ca ("fix(#5575): match linked-PR closing keywords instead of substring search"), which replaced the substring-search/gh pr list approach with the current closedByPullRequestsReferences GraphQL query and "linked to close issue" wording. Since ggallen's Aug-15 log quote uses the old "mentioning issue" wording (not present on main for 3+ weeks), the evidence almost certainly came from a stale, unsynced per-org dispatch.yml copy (the drift tracked by #6013), not from the current GraphQL-based code this PR touches. I also confirmed the current diff's select(.author.__typename != "Bot" or .author.login != "fullsend-ai-coder") is unchanged context (already on main, matching docs/contributing/bot-identities.md), so the underlying code claim is technically accurate.

But the PR body neither surfaces this reconciliation nor rebuts ggallen's report — it just asserts resolution and closes #5974, while the issue thread still contains a fresh, MEMBER-confirmed report saying the bot filter doesn't work in production. This gap is separate from the four existing review comments/replies on the PR (overbroad-filter regression, notice-format test gap, and the live-dispatch-run/#6013 resync caveat), all already fixed/acknowledged.

Suggestion: Before merging, add a note to the PR body or a comment on #5974 explicitly reconciling ggallen's "Confirmed: bot-author filter is broken" report — state which dispatch.yml revision actually produced the "mentioning issue #6246" log (it predates the 2026-07-24 GraphQL migration wording), and clarify that the observed production failure is explained by the known per-org sync gap (#6013) rather than by a live defect in the __typename+login logic this PR touches. Avoid letting the PR/issue closure imply the bot-filter bug is resolved everywhere when the stale per-org copies (#6013) demonstrably still run old logic in production.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 5:08 PM UTC · Completed 5:22 PM UTC

Commit: 56c2d2f · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 5:29 PM UTC · Completed 5:42 PM UTC

Commit: 886edc2 · View workflow run →

[waynesun09]

Reviewed via fullsend-pr-sweep across two rounds. Prior findings on the earlier head were resolved: the PR body now explicitly reconciles the #5974 bot-filter evidence (clarifying it came from a stale, unsynced per-org dispatch.yml copy, not a live defect in the code this PR touches). Rechecked at current head 886edc27 — no outstanding findings.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 7:51 PM UTC · Completed 8:07 PM UTC

Commit: 886edc2 · View workflow run →

[fullsend-ai-retro]

Retro: PR #6251 — Log matched PR numbers in dispatch existing-PR guard skip notice

Workflow overview

PR #6251 was a human-authored PR by ggallen addressing item 2 of issue #5974: improving the dispatch workflow's existing-PR guard to log matched PR numbers in a readable format (#123, #456), clarify bot-filter comments, and add a regression test. Three files changed (+53/−14 lines).

Timeline

Time (UTC)	Event
14:08	PR created
14:12	qodo-code-review[bot] finds overbroad __typename != "Bot" filter
14:19	ggallen force-pushes fix for bot filter
14:20	Review agent run 1 (31889616311) — finds misleading-title (medium), incomplete-scope (low); challenger correctly removes a false-positive "stub-test" finding
14:52	Human reviewer (waynesun09) — finds (1) test doesn't assert on the formatting pipeline, (2) per-org sync caveat for live-verification test-plan item
15:00	ggallen fixes test assertion
16:08	waynesun09 round 2 — PR body claims bot-filter "already resolved" but contradicts author's own evidence on #5974
17:07	Review agent run 2 (31897429806) — finds message-format-consistency (low): colon vs em-dash
17:28	ggallen fixes em-dash; force-push triggers run 3 (31898423749)
19:14	waynesun09 approves
19:49	Merged

Agent performance

What the agent did well:

• Message-format-consistency finding (em-dash vs colon) was actionable and fixed — human reviewer did not catch this.
• Challenger sub-agent correctly removed a false-positive "stub-test" finding, avoiding noise.
• Protected-path notice for .github/workflows/ correctly flagged for human approval.
• 3 review runs for 5 force-pushes is efficient; no excessive dispatch.

What humans caught that the agent missed:

1. Test assertion gap (waynesun09, medium) — TestDispatchPRCheckBotFilter checked for ${LINKED_PRS} variable presence but didn't assert on the map("#\(.number)") | join(", ") jq pipeline that actually implements the fix. Reverting to the old .[].number expression would still pass the test.
2. Per-org sync caveat (waynesun09, medium) — the "verify in a live dispatch run" test-plan item is not immediately satisfiable because per-org dispatch.yml copies don't auto-sync (Per-org dispatch.yml doesn't propagate reusable-workflow fixes (sync-scaffold gap; per-repo mode unaffected) #6013).
3. PR body contradiction (waynesun09, medium) — PR claimed bot-filter was "already resolved on main" while the author's own Code dispatch existing-PR guard uses bare substring search, causing false-positive blocks #5974 comment showed evidence of a broken filter (from a stale unsynced per-org copy).

Autonomy readiness

Delta: 3 human gaps, 1 agent-only novel finding. The agent found one actionable issue the human missed (message-format-consistency) but missed the three most substantive findings. The human's findings required (1) evaluating whether test assertions actually lock the behavioral fix, (2) deployment topology knowledge (per-org sync), and (3) cross-referencing linked-issue comment history against PR body claims. These are categories where the agent consistently underperforms. This PR does not support increasing agent autonomy.

Evidence for existing issues (not filing new proposals)

• agents#681 ("Correctness sub-agent should verify test assertion effectiveness"): PR refactor(#5974): log matched PR numbers in dispatch existing-PR guard skip notice #6251 provides a clear new example — the agent's correctness sub-agent reported "no findings" on a test that checked for variable presence without asserting on the actual formatting fix.
• agents#739 ("Review agent should cross-validate PR test-plan claims against code behavior"): The per-org sync caveat (test-plan item not satisfiable due to Per-org dispatch.yml doesn't propagate reusable-workflow fixes (sync-scaffold gap; per-repo mode unaffected) #6013 deployment drift) is a new instance of this gap.
• agents#265 ("Correctness sub-agent should cross-reference verified facts to detect misleading documentation narratives"): The PR body contradiction (claiming "already resolved" despite contradicting evidence on Code dispatch existing-PR guard uses bare substring search, causing false-positive blocks #5974) is a new example.

All three gaps are well-covered by existing open issues. No new proposals warranted.