Skip to content

ci: scan issue and comment bodies — this repo has never scanned one - #78

Open
yakimoto wants to merge 8 commits into
mainfrom
ci/1747-public-repo-guard-body-scan
Open

ci: scan issue and comment bodies — this repo has never scanned one#78
yakimoto wants to merge 8 commits into
mainfrom
ci/1747-public-repo-guard-body-scan

Conversation

@yakimoto

@yakimoto yakimoto commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

This repo's public-repo-guard has never scanned a single issue or comment body.

Measured across all 28 public wave-av repos (wave-av/claude-workstation#1747, #1794): two coverage shapes satisfy the one required check name Secrets + content policy.

repos triggers jobs
27 pull_request, push, workflow_dispatch guard
1 + issues, issue_comment + body-guard

This repo is in the 27. All 28 report the same green check — because a required check asserts that something named X passed, never what X examined.

The outlier is wave-moq-edge, and its own comment says why it matters:

edited matters as much as opened: a body can be made to leak long after the PR is first raised, and until this workflow covered it, nothing ever re-scanned.

That gap was not theoretical there: a PR was blocked for naming a private repo in wrangler.toml while the very same name, with more operational detail attached, sat unchallenged in its body.

What lands

Three files — the bundle the workflow's own header names, minus what this repo already has (.gitleaks.toml and content-policy.sh are already vendored, and are checked as prerequisites; a repo missing either is refused rather than half-installed):

.github/workflows/public-repo-guard.yml               replaced (72 -> 163 lines)
scripts/public-repo-guard/body-policy.sh              new, mode 100755
scripts/public-repo-guard/tests/body-policy.test.sh   new, mode 100755

The workflow's header names four files as the install unit but executes a fifthtests/body-policy.test.sh, in its own self-test step. Omitting it installs a workflow that fails on a step nobody read, so the manifest ships it. Modes are preserved via the git trees API; the contents API creates 100644 regardless, which would silently break running these scripts as executables.

Planned by governance/lib/vendor-bundle.mjs (claude-workstation#1850) against a checked-in manifest, not by ad-hoc shell.

One deliberate divergence from the reference, stated rather than silent

The shipped workflow is wave-moq-edge's with actions/checkout bumped from v5.0.1 to v7.0.1 (3d3c42e5aac5ba805825da76410c181273ba90b1), the pin already used by claude-workstation's own gate.

Copying verbatim was checked first and rejected on evidence: of the 18 target repos, 17 carry a byte-identical guard, and wave-realtime-edge already runs v7.0.0 — so a verbatim copy would have downgraded it, and shipped a stale pin to the other 17. A separate PR brings the reference itself up to the same pin.

Honest about what this can and cannot do

On a PR this PREVENTS the merge. On an issue or comment the text is already public the moment it posts, so this is DETECTION: it says go redact, fast. Only a client-side pre-write hook stops that class before publication.

Also inherited from the reference: concurrency moves from workflow-level to per job, because the two jobs want opposite behaviour. A workflow-level group forced one policy on both, and rapid body edits cancelled the tree job repeatedly — every cancelled check-run stays attached to the commit, so the PR reported UNSTABLE while the live runs were green.

The body gate ships with its own fixtures and runs them in CI. Its negative cases are the load-bearing half: a leak gate that blocks legitimate cross-repo references gets switched off, and then it protects nothing.

Refs wave-av/claude-workstation#1747.


Note

Medium Risk
Changes security CI behavior and merge blocking on PR bodies; issue/comment coverage is detection-only after publication. Low runtime risk—mostly vendored guard scripts and workflow logic, not application code.

Overview
Extends public-repo-guard so world-readable text—not just the published tree—is checked. The workflow now runs on issues, issue_comment, and pull_request edited, and adds a body-guard job that materializes title/body from the event payload into a file (no shell interpolation) and runs body-policy.sh.

body-policy.sh mirrors many tree rules (credentials, infra IDs, internal markers) but uses a proximity rule for private repos: bare cross-repo mentions stay allowed, while a private repo name within ~140 characters of operational detail (credential names, secret bindings, secret counts) blocks. An about-the-control allowlist and guard:allow keep security discussion and documented examples from tripping the gate; violations redact matched text in CI output.

The existing guard job is narrowed so it does not run on issue/comment events or PR body-only edited events; per-job concurrency replaces workflow-level grouping so rapid body edits do not cancel tree scans and leave stale failed checks. actions/checkout is bumped to v7.0.1, and body-policy.test.sh fixtures run in CI on the tree job.

Reviewed by Cursor Bugbot for commit c80ed0f. Configure here.


View with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled.

Note

Scan PR, issue, and comment bodies for credential and infrastructure leaks in CI

  • Adds a new body-guard job to public-repo-guard.yml that triggers on PR, issue, and issue comment events; materializes the title/body from the GitHub event payload and scans it with the new body-policy.sh scanner.
  • body-policy.sh uses rg -P (PCRE2 required) to block credential formats (Stripe, Anthropic, GitHub PATs, AWS AKIDs, embedded private keys), internal-only markers, operator home paths, and private-repo names near operational detail patterns driven by GUARD_PRIVATE_REPOS.
  • The job checks out scripts from the PR but resolves the scanner from a trusted base-sha copy, refusing to execute untrusted PR code; fails closed if no trusted copy exists.
  • Fixture tests in body-policy.test.sh are run as part of the guard job to validate scanner behavior before any real scan.
  • Risk: GUARD_PRIVATE_REPOS must be set in CI; an empty value exits 2 and fails the job.

Macroscope summarized c82fc3d.

Review in cubic

Review-driven hardening (post-review commits)

  • (?i) is now scoped to the private-repo name alternation ((?i:...)); a top-level flag leaked into OPS_DETAIL and made the SCREAMING_CASE credential rule match lowercase prose like cache_key. Two fixtures pin both halves (names stay case-insensitive, prose stays clean).

  • body-guard now executes the scanner from the trusted base ref (second sparse checkout at github.event.pull_request.base.sha), so a PR can no longer edit body-policy.sh in the same push that leaks in its body and go green. When the base ref predates the guard (a long-lived branch, or a retargeted PR), the fallback is the default branch's copy, never the PR's own checkout; if no trusted copy exists anywhere the job fails closed rather than execute untrusted code.

  • Both jobs probe for a PCRE2-capable ripgrep up front and fail with an error naming the problem instead of an opaque exit 2.

  • The about-the-control allowlist is scoped to prose rules only; a line naming the gate no longer exempts a credential-format hit on the same line.

…ment body

Measured across all 28 public wave-av repos (claude-workstation#1747, #1794):
TWO coverage shapes satisfy the one required check name `Secrets + content policy`.

  27 repos  triggers: pull_request, push, workflow_dispatch      jobs: guard
   1 repo   triggers: + issues, issue_comment                    jobs: + body-guard

This repo is in the 27. All 28 report the same green check.

The outlier is wave-moq-edge, and its own comment says why it matters:

  "`edited` matters as much as `opened`: a body can be made to leak long after the
   PR is first raised, and until this workflow covered it, nothing ever re-scanned."

A PR/issue/comment BODY is exactly as world-readable as the tree, and until now it
was scanned by nothing server-side. That gap was not theoretical on wave-moq-edge: a
PR was blocked for naming a private repo in wrangler.toml while the very same name,
with more operational detail attached, sat unchallenged in its body.

WHAT LANDS HERE — the bundle the workflow's own header names, minus what this repo
already has (.gitleaks.toml and content-policy.sh are already vendored):

  .github/workflows/public-repo-guard.yml          replaced (73 -> 163 lines)
  scripts/public-repo-guard/body-policy.sh         new, mode 100755
  scripts/public-repo-guard/tests/body-policy.test.sh  new, mode 100755

Copied from wave-moq-edge, which has run this shape in production. Modes preserved
via the git trees API — the contents API would have created both scripts 100644.

HONEST ABOUT WHAT IT CAN DO. On a PR this PREVENTS the merge. On an issue or comment
the text is already public the moment it posts, so this is DETECTION: it says go
redact, fast. Only a client-side pre-write hook stops that class before publication.

Also inherited from the reference: concurrency moves from workflow-level to PER JOB,
because the two jobs want opposite behaviour. A workflow-level group forced one
policy on both, and rapid body edits cancelled the tree job repeatedly — every
cancelled check-run stays attached to the commit, so the PR reported UNSTABLE while
the live runs were green.

The body gate ships with its own fixtures and runs them in CI. Its NEGATIVE cases are
the load-bearing half: a leak gate that blocks legitimate cross-repo references gets
switched off, and then it protects nothing.

Refs wave-av/claude-workstation#1747.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@yakimoto yakimoto added the rr:skip-coderabbit RF.P1 reviewer routing (#1039) label Aug 6, 2026
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Excluded labels (none allowed) (1)
  • rr:skip-coderabbit

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a928a462-ea84-47be-83c5-376437aa9e11

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@cursor

cursor Bot commented Aug 6, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_9af9e0c1-18c9-4915-a979-c562aab15a80)

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Aug 6, 2026
@macroscopeapp

macroscopeapp Bot commented Aug 6, 2026

Copy link
Copy Markdown

Approvability

Verdict: Needs human review

Unable to check for correctness in c82fc3d. This PR introduces a significant new security scanning feature for PR/issue/comment bodies with complex allowlist logic. Unresolved review comments raise potential bypass concerns in the allowlist design that warrant human evaluation.

You can customize Macroscope's approvability policy. Learn more.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Running ultrareview automatically — This adds a new merge-gating body-scanning job with regex-based allowlists and per-job concurrency; a subtle false-positive or fail-open bug would block or silently unguard PRs across every repo this guard is vendored to.. I'll post findings when complete.

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

CI: scan PR/issue/comment bodies in public-repo-guard

✨ Enhancement ⚙️ Configuration changes 🧪 Tests 🕐 40+ Minutes

Grey Divider

AI Description

• Add a new body-scanning job for PR/issue/comment title+body on open/edit events.
• Split concurrency per job to avoid body edits cancelling tree scans.
• Add body policy rules + fixture tests, and bump pinned checkout action.
Diagram

graph TD
E{{"GitHub events"}} --> WF["public-repo-guard.yml"] --> G1["Job: guard (tree)"] --> GL["gitleaks + content-policy"]
WF --> G2["Job: body-guard"] --> MAT["jq -> body.txt"] --> BP["body-policy.sh"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Use a marketplace body-scanning action
  • ➕ Less custom shell/regex maintenance
  • ➕ Potentially richer detectors and reporting
  • ➖ Supply-chain risk vs vendored, pinned scripts
  • ➖ Often requires broader permissions/tokens
  • ➖ Harder to guarantee fail-closed behavior on schema drift
2. Fetch bodies via GitHub API at runtime
  • ➕ Can scan the latest body even if payload schema changes
  • ➕ Could scan additional fields beyond the webhook payload
  • ➖ Requires auth/token handling; higher security risk on forks
  • ➖ More rate-limit and availability failure modes
  • ➖ More complex than reading $GITHUB_EVENT_PATH
3. Run gitleaks against the body text instead of custom rg rules
  • ➕ Reuses a single scanning engine and existing rule ecosystem
  • ➕ Potentially catches more secret formats automatically
  • ➖ Higher false-positive risk for prose (may become undeployable)
  • ➖ Slower/more expensive per comment edit
  • ➖ Still needs careful redaction of matched output

Recommendation: The chosen approach (materialize webhook payload to a file, scan with a tight rg-based policy, and run fail-closed on unknown payload shapes) is the best fit for public repos: minimal permissions, low runtime cost, and avoids command-injection surfaces. Keep the policy ruleset small and test-driven (as done here) to reduce false positives that would otherwise cause the gate to be disabled.

Files changed (3) +342 / -5

Enhancement (1) +139 / -0
body-policy.shNew body policy scanner for PR/issue/comment text +139/-0

New body policy scanner for PR/issue/comment text

• Introduces a ripgrep-based policy script that scans materialized PR/issue/comment title+body text for credential-like patterns, internal-only markers, and private-repo mentions near operational detail. Implements explicit allowlisting (guard:allow and an about-the-control allowlist), redacts matched content from annotations, and fails closed on scanner errors.

scripts/public-repo-guard/body-policy.sh

Tests (1) +108 / -0
body-policy.test.shFixture tests for body-policy rules and redaction behavior +108/-0

Fixture tests for body-policy rules and redaction behavior

• Adds hermetic fixture tests that verify both blocking detections and precision/negative cases (to keep the gate deployable). Includes checks that annotations never echo matched text and that the script fails closed on missing/invalid input.

scripts/public-repo-guard/tests/body-policy.test.sh

Other (1) +95 / -5
public-repo-guard.ymlAdd issue/comment body scanning job and per-job concurrency +95/-5

Add issue/comment body scanning job and per-job concurrency

• Expands workflow triggers to include issues and issue_comment, and includes PR edited events. Splits behavior into a tree-scanning job (guard) and a body-scanning job (body-guard) with separate concurrency policies, adds a self-test step for body-policy fixtures, and bumps the pinned checkout action to v7.0.1.

.github/workflows/public-repo-guard.yml

devin-ai-integration[bot]

This comment was marked as resolved.

@qodo-code-review

qodo-code-review Bot commented Aug 6, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. PR can bypass body scan ✓ Resolved 🐞 Bug ⛨ Security
Description
On pull_request events, body-guard checks out the PR revision and executes
scripts/public-repo-guard/body-policy.sh from that checkout, so a PR can modify the scanner to
always pass and thereby evade body scanning. This creates a false-green “Body content policy” result
for PR bodies (the new coverage this PR adds).
Code

.github/workflows/public-repo-guard.yml[R128-133]

+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          # Only the gate's own scripts are needed — no reason to pay for the whole
+          # tree on every comment.
+          sparse-checkout: scripts/public-repo-guard
+          sparse-checkout-cone-mode: false
Relevance

●●● Strong

Team has accepted CI hardening to prevent mutable/untrusted execution; likely will fix PR checkout
script bypass.

PR-#18
PR-#67

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The workflow explicitly runs on pull_request events and then executes an in-repo script after
checking out repository contents. Because the script path is part of the repository contents, any
changes to that script in a PR affect what code is executed during the PR’s body scan.

.github/workflows/public-repo-guard.yml[28-36]
.github/workflows/public-repo-guard.yml[113-134]
.github/workflows/public-repo-guard.yml[160-163]
scripts/public-repo-guard/body-policy.sh[1-27]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`body-guard` runs `bash scripts/public-repo-guard/body-policy.sh ...` from the repository checkout. For `pull_request` runs, that checkout includes PR changes, so the PR author can modify `body-policy.sh` (or its tests) to weaken/disable detection and still get a green body scan.

## Issue Context
This job is intended to scan *untrusted PR body text*, but it also currently runs *untrusted PR-controlled code* (the scanner itself). The scanner code should come from a trusted ref (e.g., the PR base SHA / default branch), while the scanned body content can still come from `$GITHUB_EVENT_PATH`.

## Fix Focus Areas
- .github/workflows/public-repo-guard.yml[113-163]

### Suggested fix approach
- For `pull_request` events, check out the scanner scripts from the trusted base ref (e.g. `${{ github.event.pull_request.base.sha }}`) into a separate directory (e.g. `path: trusted-guard`) and run `trusted-guard/scripts/public-repo-guard/body-policy.sh ...`.
- Keep the current sparse checkout.
- For non-PR events (`issues`, `issue_comment`), use a safe fallback ref such as `${{ github.event.repository.default_branch }}`.
- Optional bootstrap: if this repo is installing the scanner for the first time and the base ref won’t yet contain the scripts, add a one-time fallback to use the current ref only when `trusted-guard/scripts/public-repo-guard/body-policy.sh` is missing, so future PRs are protected by trusted scanner code.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context used
Review mode: ⚖️ Balanced: Downgraded extended -> standard: change is below the extended eligibility bar (hunks 5/18, lines 347/200; both must reach the floor). Router rationale: This is a security-sensitive CI/workflow change with substantial new shell logic across multiple independent paths, including event routing, untrusted payload handling, regex policy, fail-closed behavior, and fixture coverage; redundant review is warranted.

To customize comments, go to the Qodo configuration screen, or learn more in the docs.

Qodo Logo

qodo-code-review[bot]

This comment was marked as resolved.

@qodo-code-review

Copy link
Copy Markdown

Qodo Fixer

No findings are within the configured fix scope. To change which findings are fixed, adjust the setting on your Qodo configuration page.

…ue exit 2

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
@yakimoto
yakimoto force-pushed the ci/1747-public-repo-guard-body-scan branch from 9239ff5 to 178fd84 Compare August 6, 2026 17:30
…d base ref

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
cubic-dev-ai[bot]

This comment was marked as resolved.

…event a base change fires

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

…ckout credentials

Three review-driven fixes:
- check()'s allowlist filters were '|| true'd: a filter dying with exit >= 2
  emptied the match list and reported a DETECTED hit as clean. Each filter now
  captures its status and exits 2, same as the main scan.
- The test fixtures hardcoded real private repo names and credential names;
  the file is gate-exempt by path, so it published exactly what the gate
  blocks. All fixture names are now obviously synthetic (acme-*).
- All three checkout steps now set persist-credentials: false, matching every
  other workflow here: these jobs execute scripts from the PR's own tree, and
  that code must not run beside a token in .git/config. Also documents why a
  body-guard failure on an issue/comment lands on main's head commit and why
  the job must stay non-required.

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
cubic-dev-ai[bot]

This comment was marked as resolved.

…five

The self-test step runs tests/body-policy.test.sh, so an install that copies
only the four listed files fails on a step nobody read. List the fixture as
part of the install unit.

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

… allowlist to prose rules

Two review findings on the body gate:

- The bootstrap fallback ran the PR's own body-policy.sh whenever the base
  ref lacked a copy, which is not only the install PR: any PR opened against
  (or retargeted onto) a branch predating the guard would have its own code
  judge its own body. The fallback is now the default branch's copy (which a
  PR cannot control), and if no trusted copy exists anywhere the job fails
  closed instead of executing untrusted code.

- The about-the-control allowlist was applied to every rule, so a line
  mentioning the gate by name was exempt even when it also carried a live
  credential. It now applies only to prose rules (internal-marker,
  private-repo-ops); credential- and infrastructure-format rules stay strict.
  Fixtures pin both halves.

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

…logy rule strict on gate-naming lines

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Open in Devin Review

Comment on lines +121 to +122
- name: body policy self-test (fixtures)
run: bash scripts/public-repo-guard/tests/body-policy.test.sh

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟨 Tree-scan job executes the pull request's own guard scripts as part of a required check

The required guard job checks out the PR's tree and runs scripts/public-repo-guard/tests/body-policy.test.sh (and, pre-existing, content-policy.sh) straight from that untrusted checkout, so a PR can rewrite the fixtures or the policy script it is being judged by and make the self-test pass unconditionally. The new body-guard job explicitly avoids exactly this (.github/workflows/public-repo-guard.yml:164-192 resolves the scanner from the trusted base/default ref), so the two halves of the same gate apply opposite trust models.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rr:skip-coderabbit RF.P1 reviewer routing (#1039)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant