Several e2e tests create a "clean npm project" fixture and expect cve-lite . --offline to exit 0 (no findings). These tests are fragile because they depend on the advisory DB state at the time the test runs. When new CVEs are published that match the fixture's packages, the "clean" scan exits 1 instead of 0.
This caused 28 test failures in the v1.25.0 release CI run:
tests/e2e/commands-and-exit-codes.test.ts - default offline scan of a clean npm project exits 0
tests/e2e/commands-and-exit-codes.test.ts - --ratchet --check-overrides keeps override hygiene out of the baseline
tests/audit-log/emission-end-to-end.test.ts - captures scan.started, scan.finished on a clean scan
tests/cli/scan-fix-verify-exitcode.test.ts - exit code assertions
tests/e2e/fix-and-auditlog.test.ts
tests/e2e/validation-and-outputs.test.ts
Fix options:
- Replace "clean" fixture packages with packages that have no known CVEs and are unlikely to acquire them (e.g. packages with no dependencies, or pinned to versions with no advisories and
overrides suppressing future ones in the fixture's lockfile).
- Where the test only needs a "no critical findings" guarantee, add
--fail-on critical so the test is resilient to new low/medium advisories appearing.
- Maintain a small locked advisory DB snapshot checked into the repo that e2e tests use via
--offline-db, so the DB never changes between runs.
Option 3 is the most robust but adds maintenance overhead. Option 2 is the least invasive for tests that don't care about severity.
Several e2e tests create a "clean npm project" fixture and expect
cve-lite . --offlineto exit 0 (no findings). These tests are fragile because they depend on the advisory DB state at the time the test runs. When new CVEs are published that match the fixture's packages, the "clean" scan exits 1 instead of 0.This caused 28 test failures in the v1.25.0 release CI run:
tests/e2e/commands-and-exit-codes.test.ts-default offline scan of a clean npm project exits 0tests/e2e/commands-and-exit-codes.test.ts---ratchet --check-overrides keeps override hygiene out of the baselinetests/audit-log/emission-end-to-end.test.ts-captures scan.started, scan.finished on a clean scantests/cli/scan-fix-verify-exitcode.test.ts- exit code assertionstests/e2e/fix-and-auditlog.test.tstests/e2e/validation-and-outputs.test.tsFix options:
overridessuppressing future ones in the fixture's lockfile).--fail-on criticalso the test is resilient to new low/medium advisories appearing.--offline-db, so the DB never changes between runs.Option 3 is the most robust but adds maintenance overhead. Option 2 is the least invasive for tests that don't care about severity.