Feature proposal: versioned security scan benchmark corpus

[sylvesterkaczmarek]

Problem

Codex Security has good unit/integration coverage for scanner mechanics, but there is no first-party, versioned benchmark corpus that measures whether a release still finds the same classes of vulnerabilities with comparable precision and useful evidence.

That makes scanner-quality regressions harder to distinguish from ordinary implementation regressions. It also makes changes to prompts, finding validation, attack-path construction, deduplication, or model/provider defaults difficult to compare across releases in a reproducible way.

I searched current issues and source for an equivalent benchmark/replay corpus and did not find one.

Proposal

Add a small public-safe benchmark suite made only from synthetic or intentionally vulnerable toy repositories, with versioned expected finding contracts.

A benchmark case could define:

• repository fixture and language/runtime;
• vulnerability category and expected source/sink or affected file region;
• whether a finding is required, optional, or explicitly absent;
• bounded evidence expectations such as relevant files, attack-path stages, and severity range;
• allowed duplicates and expected dedupe grouping;
• scan mode and deterministic scanner settings used for evaluation.

The scorer should avoid requiring exact natural-language finding text. Instead it could match stable semantic properties such as category, file/region overlap, evidence references, reachability/precondition structure, and whether the finding survives validation.

Example layout:

benchmarks/
  path_traversal_basic/
    repo/
    expected.json
  authz_missing_check/
    repo/
    expected.json
  unsafe_deserialization/
    repo/
    expected.json

and a command such as:

codex-security benchmark benchmarks/ --mode deep

could report per-case recall, unexpected findings, duplicate rate, validation survival, and aggregate regressions relative to a stored baseline.

Scope

I would keep the first version deliberately small:

1. define a versioned expected-finding schema;
2. add a benchmark runner/scorer;
3. include a handful of synthetic cases across a few vulnerability categories;
4. ensure fixtures contain no real customer code, private findings, credentials, or exploit material;
5. make the suite useful in CI as a quality-regression signal without requiring exact model wording.

A later extension could support replaying previously captured public-safe scan artifacts to isolate scanner pipeline changes from model variability.

If this direction is useful, I would be happy to prototype the schema, runner, scoring logic, several synthetic fixtures, tests, and documentation.