feat(bench): BugBench — detection-quality benchmark + CI gate (/bench)

.github/workflows/tests.yml

@@ -38,3 +38,28 @@ jobs:
 
       - name: Run test suite
         run: pytest -q
+
+  bugbench:
+    name: bugbench (detection-quality gate)
+    runs-on: ubuntu-latest
+    # NON-BLOCKING while the corpus is small: this reports detection quality on
+    # every PR but must not red-gate unrelated PRs if one detector regresses or a
+    # newly-added hard case fails. Flip to blocking (remove continue-on-error and
+    # make it a required check) once the corpus is broad enough to trust as a gate.
+    # NOTE: until then, adding a currently-failing case will show this job red but
+    # will NOT block merges.
+    continue-on-error: true
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      # Runs the toolkit's detectors against a ground-truth corpus and reports
+      # precision / recall / false-positive rate. Threshold is strict so quality
+      # regressions are visible; the job is continue-on-error so it informs
+      # without blocking while the corpus matures.
+      - name: Run BugBench detection-quality gate
+        run: python bughunter/tools/bench.py run --min-precision 1.0 --max-fp-rate 0.0

CLAUDE.md

@@ -70,6 +70,7 @@ This repo is a Claude Code plugin for professional bug bounty hunting across Hac
 | `/poc` | `/poc capture <url> [-H ...] \| from-request req.txt [--response resp.txt]` — reproducible PoC evidence bundle (request/response, curl, HAR, screenshot, report-ready evidence.md); secrets redacted by default |
 | `/dashboard` | `/dashboard [--port 8777] \| --export dashboard.html` — local web Hunt Dashboard: leads, recon surface, findings, reports, galleries, memory flywheel, in one live view |
 | `/verify` | `/verify check <bundle-dir> [--marker "signal"] \| sweep [root]` — report gate: independently re-derives a finding from its PoC bundle; PROVEN → reportable, REFUTED/UNPROVEN/INCONCLUSIVE → suppressed; writes verification.json; `--require-proof` gates CI |
+| `/bench` | `/bench run [--min-precision 1.0 --max-fp-rate 0.0] \| list` — BugBench: measure detector precision/recall/false-positive rate vs a ground-truth corpus (known-vuln + known-safe); `--min-precision`/`--max-fp-rate` gate CI against quality regressions |
 
 ### Agents (9 specialized agents)
 
@@ -124,6 +125,7 @@ This repo is a Claude Code plugin for professional bug bounty hunting across Hac
 - `tools/poc_bundler.py` — PoC evidence bundler (`capture` live via SSRF-guarded `safe_http`, or `from-request` offline from a saved/Burp request). Emits `request.http`, `response.http`, `repro.sh` (curl), `evidence.har` (HAR 1.2), optional `screenshot.png`, `evidence.md` (report-ready), and `bundle.json` (manifest + response SHA-256) under `findings/<target>-<class>/evidence/<id>/`. Secrets redacted by default (repro.sh uses `$ENV` placeholders); refuses PUT/DELETE/PATCH without `--confirm-unsafe`. See `/poc`.
 - `tools/hunt_dashboard.py` — local web Hunt Dashboard (`serve`/`export`); reads `memory/leads/`, `recon/`, `findings/`, `reports/`, galleries, and `hunt-memory/` into one live view. Pure stdlib, binds to 127.0.0.1, HTML-escapes all recon-derived data, path-traversal-guarded `/file` endpoint. Distinct from `tools/dashboard.py` (the in-terminal ANSI progress TUI). See `/dashboard`.
 - `tools/verifier.py` — the report gate (prove-or-suppress). Independently re-derives a finding from its `poc_bundler` bundle (re-sends via SSRF-guarded `safe_http` with an initial-host guard, checks a confirmation marker in the fresh response headers+body) and decides reportability: PROVEN only if the marker was in the original response AND is still present now; else REFUTED/UNPROVEN/INCONCLUSIVE → suppressed. Records the re-fetched response's SHA + status. `decide()` is a pure, total policy function; only PROVEN ships. Writes `verification.json`; `check`/`sweep`; `--require-proof` exits non-zero for gating; unsafe methods never re-fired without `--confirm-unsafe`; secrets read from env only. Reuses only merged code (poc_bundler). See `/verify`.
+- `tools/bench.py` — BugBench detection-quality harness. Runs the toolkit's detectors against a ground-truth corpus (`bughunter/bench/cases/*.json`, known-vuln + known-safe, recorded inputs = deterministic) and reports precision / recall / false-positive rate per detector. Detectors register a one-function adapter via `@detector`; cases are contributable JSON. `run` / `list`; `--min-precision`/`--max-fp-rate` exit non-zero to gate CI against quality regressions. Pure-stdlib. See `/bench`.
 - Full catalogue: **`tools/README.md`** (~50 tools). `hunt.py` auto-ingests leads after recon (`--graphql` / `--cve-hunt` / `--skip-leads` flags).
 
 ### External tool references

bughunter/bench/cases/cors-fixed-allowlist-safe.json

@@ -0,0 +1,15 @@
+{
+  "id": "cors-fixed-allowlist-safe",
+  "detector": "cors",
+  "note": "Server returns a fixed, trusted allow-list origin and ignores the attacker origin — correct CORS. A detector that flags this is a false positive.",
+  "input": {
+    "url": "https://api.example.com/me",
+    "sent_origin": "https://evil.example",
+    "acao": "https://app.example.com",
+    "acac": "true"
+  },
+  "expected": {
+    "vulnerable": false,
+    "class": "cors"
+  }
+}

bughunter/bench/cases/cors-null-origin-credentialed.json

@@ -0,0 +1,16 @@
+{
+  "id": "cors-null-origin-credentialed",
+  "detector": "cors",
+  "note": "Server reflects the `null` origin AND allows credentials — reachable from a sandboxed iframe / data: URI, cookies sent cross-origin. A real HIGH bug.",
+  "input": {
+    "url": "https://api.example.com/me",
+    "sent_origin": "null",
+    "acao": "null",
+    "acac": "true"
+  },
+  "expected": {
+    "vulnerable": true,
+    "class": "cors",
+    "min_severity": "high"
+  }
+}

bughunter/bench/cases/cors-reflect-credentialed.json

@@ -0,0 +1,16 @@
+{
+  "id": "cors-reflect-credentialed",
+  "detector": "cors",
+  "note": "Server reflects an arbitrary attacker origin AND allows credentials — a real, critical CORS bug (authenticated cross-origin read).",
+  "input": {
+    "url": "https://api.example.com/me",
+    "sent_origin": "https://evil.example",
+    "acao": "https://evil.example",
+    "acac": "true"
+  },
+  "expected": {
+    "vulnerable": true,
+    "class": "cors",
+    "min_severity": "high"
+  }
+}

bughunter/bench/cases/cors-wildcard-with-credentials.json

@@ -0,0 +1,16 @@
+{
+  "id": "cors-wildcard-with-credentials",
+  "detector": "cors",
+  "note": "ACAO:* together with ACAC:true is invalid per spec (browsers reject it) but signals a broken CORS layer — flagged MEDIUM, not treated as a clean public wildcard.",
+  "input": {
+    "url": "https://api.example.com/data",
+    "sent_origin": "https://evil.example",
+    "acao": "*",
+    "acac": "true"
+  },
+  "expected": {
+    "vulnerable": true,
+    "class": "cors",
+    "min_severity": "medium"
+  }
+}