[BUG] Fail registry-lockfile benchmarks when warmup does not create a lockfile

[lukekarrys]

Problem

The registry-lockfile benchmark relies on its warmup to resolve the graph and write package-lock.json. If the warmup hits the 300-second command timeout, it can be killed before the lockfile is written. hyperfine --ignore-failure then continues silently, and timed run 0 pays for a full resolution against a partially warmed cache.

This explains the observed [110, 4, 4] and [294, 4, 4] shapes: the row intended to measure tarball serving instead includes a metadata-resolution run. The original vlt trigger was fixed, but slow third-party registries can still hit this failure mode.

Proposed direction

Verify that the expected lockfile exists after warmup and fail loudly when it does not. Separately decide whether warmups need a larger timeout than timed runs.

Acceptance criteria

• A lockfile benchmark cannot enter timed runs without a valid lockfile produced for that registry.
• Warmup timeout/failure is visible in the job result and summary.
• Timeout behavior is tested for both successful and failed warmups.
• The interaction with slow 300–380s third-party registry runs is documented.
• Successful timed lockfile runs continue to fetch no packuments.

Source

Performance investigation: item 6 — lockfile warmup timeout poisons run 0