Skip to content

Inventory missing merged PR benchmarks - #1148

Merged
helly25 merged 15 commits into
mainfrom
docs/benchmark-merge-coverage
Nov 8, 2026
Merged

helly25 merged 15 commits into
mainfrom
docs/benchmark-merge-coverage

Conversation

@helly25

@helly25 helly25 commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Record missing merged PR benchmarks, make existing measurements identifiable by PR, and provide a targeted recovery command that preserves usable results from partially failed campaigns.

AG;DR

  • Inventory merged PRs from September 19, 2026 absent from the merged CI viewer catalog, including observed workflow status and known limitations. The checked-in inventory is a snapshot.

  • Render associated merged PR numbers in version labels, retaining commit, source, links, and backfill provenance.

  • Add //tools:benchmark_recovery, which reads current merged PR metadata, the published catalog, active benchmark runs, and retained normal and backfill artifacts. It reports missing platforms, measurements still running, artifacts awaiting publication, and PRs needing backfill.

  • --dispatch-next dispatches a bounded batch of up to four missing PRs by default (configurable from one to eight) to the targeted main-branch backfill workflow added by PR Run benchmark measurements in parallel after building #1147. It refuses while a backfill campaign is active or queued. --pr and --since allow focused recovery. Operators must avoid concurrent dispatches; the check and dispatch are not atomic.

  • Reconcile completed failed or canceled backfill campaigns and retain each independently validated platform result, rather than losing every valid result when one job fails. Continue to reject incomplete or invalid records and retain immutable source and campaign provenance. The existing strict retention interface remains atomic by default.

  • This is an operator-driven recovery step, not an automatic durable retry scheduler. Recovery selects the union of missing platforms for its batch; a single-PR retry can preserve an already retained platform. Existing artifact presence does not guarantee validation or publication. The documented design identifies the remaining scheduler and status visibility work.

  • Uses the targeted revision input deployed by merged PR Run benchmark measurements in parallel after building #1147. This PR now targets main.

  • Validation: bazel test //tools:benchmark_recovery_test //tools:benchmark_campaign_test //tools:benchmark_history_test //tools:benchmark_landscape_test passed on macOS, including targeted plan selection and partial failed-campaign retention. Read-only live inventories correctly identified PR Distinguish benchmark layout versions in explorer #986 as requiring both platforms and PR Reduce basic traversal overhead and defer native-case metadata #1143 as already covered. Trunk check passed. The requested PR Explore benchmark history by measured version and platform #914/Distinguish benchmark layout versions in explorer #986 backfill was dispatched on main in run https://github.com/mboworks/xff/actions/runs/42498912253. Its end-to-end measurement and publication results remain pending.

  • Document the continuation after the active Explore benchmark history by measured version and platform #914/Distinguish benchmark layout versions in explorer #986 and queued Reduce basic traversal overhead and defer native-case metadata #1143 backfills: verify viewer publication, refresh and recover the missing-PR inventory, resume equivalent find/rg optimization, and qualify representative combined workloads before a separately identified historical backfill.

  • Pin historical Linux measurement jobs to Ubuntu 22.04 and explicitly install libicu70/libxml2. The initial Distinguish benchmark layout versions in explorer #986/Explore benchmark history by measured version and platform #914 Linux jobs failed before measurement because their historical LLD executables needed libicui18n.so.70 and libxml2.so.2 on an Ubuntu 26.04 runner. Preserve historical compiler/LTO settings and accept earlier ubuntu-latest plans during validation. The Ubuntu 22.04 retirement date is documented.

  • Add a platforms input to the backfill workflow and bind requested platforms into campaign identity. This allows Linux-only retries without repeating the active macOS work; omitted selection keeps both platforms. Focused campaign, history and recovery tests passed, as did workflow lint. The new runner has not yet completed an end-to-end historical build.

@helly25
helly25 changed the base branch from main to perf/parallel-benchmark-builds November 8, 2026 15:48
Base automatically changed from perf/parallel-benchmark-builds to main November 8, 2026 16:13
@helly25
helly25 merged commit c80d631 into main Nov 8, 2026
34 checks passed
@helly25
helly25 deleted the docs/benchmark-merge-coverage branch November 8, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant