Skip to content

Wait for exact-base quality baseline artifacts before comparison #516

Description

Problem

Exact-base quality workflows can start before the corresponding baseline workflow for a freshly merged base commit has uploaded artifacts. The download action fails immediately instead of waiting, causing a false-negative PR check.

Observed on PR #515:

  • base commit: ae216cad97a11eb389585edf3613a22dde4223b7
  • performance run: 31542690250
  • Linux benchmark itself completed successfully with healthy metrics
  • comparison failed closed with Snapshot file does not exist: baseline-perf/metrics.json
  • the exact-base Performance Baseline workflow completed successfully afterward
  • rerunning only the failed Linux job then passed

Coverage has the same exact-base artifact dependency and can race similarly.

Scope

  • wait for the exact-base performance/coverage baseline workflow and required artifact before download;
  • use a bounded timeout and explicit polling interval;
  • fail immediately with a clear error if the exact-base workflow completes unsuccessfully;
  • fail clearly on timeout or missing required artifact;
  • never fall back to a moving/latest-main baseline;
  • share and unit-test the waiting logic across performance and coverage workflows.

Acceptance criteria

  • a pending exact-base workflow is polled until its required artifact is available;
  • successful, failed, timed-out, and missing-artifact paths are tested;
  • performance and coverage workflows use the helper;
  • exact-base SHA remains visible in logs/reports;
  • full CI, comparator tests, coverage, performance, and CodeQL pass.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIssue identified by VS Code Team member as probable bugimportantIssue identified as high-priority

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions