Skip to content

Disclosure documents give no way to distinguish verified from unverified findings #210

Description

@NahumKorda

Summary

The report phase generates one disclosure document per finding for every finding, including those whose Stage 2 verification never completed. The documents contain no marker distinguishing an attacker-simulation-confirmed finding from an unadjudicated Stage 1 candidate, and no structured indication of which file they concern.

Evidence

Across all 413 generated documents in report/disclosures/:

Check Count
Documents generated 413
Containing **Affected:** [NOT PROVIDED] 413
Mentioning verification status in any wording 8
Containing [REQUIRES DYNAMIC TESTING] 408

In this run only 82 findings were upheld by Stage 2; 331 were never adjudicated. All 413 received a document titled # Security Disclosure: …, written with uniform confidence. The Affected: header is unpopulated in every one, even though the structured location (file and function) is present for each finding in pipeline_output.json.

Why it matters

The product premise is "Stage 1 detects. Stage 2 attacks. What survives is real." The disclosure output discards that distinction precisely where it reaches a human reader. In practice, handing this directory to a repository owner presents several hundred unadjudicated candidates in a format indistinguishable from confirmed vulnerabilities. Given the project is engaged in coordinated disclosure, emitting confident disclosure prose for unverified findings is a reputational risk as much as a usability defect.

Suggested fix

In descending order of value:

  1. Populate Affected: from the finding's structured location — the data is already available.
  2. Stamp the Stage 2 verdict in both the document header and the filename (e.g. DISCLOSURE_053_UNVERIFIED.md).
  3. Default to generating disclosures only for findings upheld by Stage 2, behind a --disclosures-for {confirmed,all} flag.

Observed on one full-pipeline run at production scale: a private TypeScript monorepo (Angular front end, NestJS back end), 1,351 analysis units from 866 source files, run with --verify and no --limit, ~6.5 hours wall clock. The engine was invoked directly (python -m openant scan …, Python 3.13); the Go CLI was not built, so none of this involves the Go↔Python envelope. Stage 1 detection and Stage 2 verification ran on Claude Opus 5; application context, enhancement and reporting on Claude Sonnet 5 — all seven phases routed through OpenRouter (anthropic/claude-opus-5, anthropic/claude-sonnet-5) via a locally-added openrouter provider adapter, on a build based on upstream 2ed78f6. Cost figures are expressed as proportions of the run total; absolute amounts are omitted deliberately, as is the identity of the scanned repository.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions