Skip to content

[Analyzer] Expand security coverage + enforce global strict test runner + stabilize expectation matching #53

@SizzleUnrlsd

Description

@SizzleUnrlsd

Context

This repository has received a large set of changes:

  • new security detections in the analyzer,
  • stronger integration in pipeline/CLI/diagnostics,
  • hardened test runner (run_test.py) to prevent false-green runs when expectations are incomplete,
  • expanded/reorganized test corpus (nested, inter-TU, tricky scenarios).

Problems addressed

  1. Several CWE classes were missing or under-covered.
  2. The runner could pass tests even when expected comments were fewer than actual diagnostics.
  3. Test expectations were too brittle to line drift (format/refactor), or in some cases too permissive.
  4. Security fixtures were not organized by vulnerability category.

Goal

Make validation robust and maintainable:

  • broader analysis coverage,
  • strict consistency between expected and actual diagnostics,
  • more maintainable test corpus,
  • fewer silent regressions.

Scope (high-level)

  • Analyzer:
    • add/integrate security analyses (null deref, integer overflow, command injection, TOCTOU, OOB read, type confusion, etc.),
    • update pipeline/emission/models.
  • Runner:
    • global strict diagnostic count,
    • improved diagnostic-block matching,
    • fix location-tolerance regressions.
  • Tests:
    • add advanced cases (nested if/loop, inter-TU),
    • backfill missing expectations,
    • reorganize security corpus into test/security/<category>.

Acceptance criteria

  • python3 run_test.py --jobs=8 passes end-to-end.
  • Tests fail when expected warning/error comments are missing.
  • Clearly wrong line anchors (stale expectations) do not pass due to over-tolerance.
  • Multi-diagnostics in the same location block remain visible/testable.
  • tmp/ and local artifacts are excluded from PRs.

Notes

This is a broad initiative; it can be split into thematic PRs (analyzer core / runner / tests) if needed.

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions