Skip to content

fix(tests): e2e fixtures fail when new CVEs appear that match clean-project packages #726

@sonukapoor

Description

@sonukapoor

Several e2e tests create a "clean npm project" fixture and expect cve-lite . --offline to exit 0 (no findings). These tests are fragile because they depend on the advisory DB state at the time the test runs. When new CVEs are published that match the fixture's packages, the "clean" scan exits 1 instead of 0.

This caused 28 test failures in the v1.25.0 release CI run:

  • tests/e2e/commands-and-exit-codes.test.ts - default offline scan of a clean npm project exits 0
  • tests/e2e/commands-and-exit-codes.test.ts - --ratchet --check-overrides keeps override hygiene out of the baseline
  • tests/audit-log/emission-end-to-end.test.ts - captures scan.started, scan.finished on a clean scan
  • tests/cli/scan-fix-verify-exitcode.test.ts - exit code assertions
  • tests/e2e/fix-and-auditlog.test.ts
  • tests/e2e/validation-and-outputs.test.ts

Fix options:

  1. Replace "clean" fixture packages with packages that have no known CVEs and are unlikely to acquire them (e.g. packages with no dependencies, or pinned to versions with no advisories and overrides suppressing future ones in the fixture's lockfile).
  2. Where the test only needs a "no critical findings" guarantee, add --fail-on critical so the test is resilient to new low/medium advisories appearing.
  3. Maintain a small locked advisory DB snapshot checked into the repo that e2e tests use via --offline-db, so the DB never changes between runs.

Option 3 is the most robust but adds maintenance overhead. Option 2 is the least invasive for tests that don't care about severity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggood first issueGood for newcomers

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions