diff --git a/.github/workflows/benchmarks.yml b/.github/workflows/benchmarks.yml index 05a830fe2f..bd341e6471 100644 --- a/.github/workflows/benchmarks.yml +++ b/.github/workflows/benchmarks.yml @@ -12,6 +12,8 @@ permissions: concurrency: group: benchmarks-${{ github.ref }} + # Keep each merged revision; the default single pending run replaces its predecessor. + queue: max cancel-in-progress: false jobs: diff --git a/TODO.md b/TODO.md index cba6bad8d1..02634717d3 100644 --- a/TODO.md +++ b/TODO.md @@ -797,6 +797,10 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms ## Benchmark publication follow-up +- [x] Preserve up to 100 pending main benchmark runs instead of replacing the previous pending + merge's measurement; keep active runs and exact revision identities. +- [ ] Reconcile missing main measurements after queue cancellation or overflow against retained + reports and source artifacts; backfill only missing revisions, without replacing existing data. - [ ] Retain separate macOS and Linux HTML reports plus raw JSON, with commit, build, hardware/runner and measurement-contract identities. Local temporary previews are not release assets. - [ ] Publish post-merge reports as official results and pre-merge reports as labeled PR previews. diff --git a/docs/benchmark-comparisons.md b/docs/benchmark-comparisons.md index 5c11fc15dc..7ffae20a0d 100644 --- a/docs/benchmark-comparisons.md +++ b/docs/benchmark-comparisons.md @@ -42,6 +42,20 @@ local history is needed. Use the [historical backfill workflow](benchmark-backfill.md) to build and measure a frozen revision list in one host session, with a common driver and reference-tool set. +## Retaining measurements for successive merges + +Each merge into `main` requests its own post-merge measurement. The `Benchmarks` workflow +serializes runs for the same ref with `queue: max` and `cancel-in-progress: false`. This keeps +up to 100 pending runs alongside the active run. GitHub's default `queue: single` replaces +an older pending run when a newer merge arrives, even when cancellation of the active run +is disabled. See [GitHub's concurrency queue rules](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency). + +The queue is bounded: additional runs are canceled once 100 are pending. This change does not +recover earlier canceled runs or guarantee measurement completion after runner or workflow +failures. Use the [backfill workflow](benchmark-backfill.md) for missing revisions. Queue order +also does not establish revision order; retained reports keep their exact commit and run identity. +PR measurements and publication use their own concurrency policies. + ## Relative performance and advisory alarm Each shape also has a CPU-grouped ratio table for participants doing the same task. It shows diff --git a/docs/performance-analysis.md b/docs/performance-analysis.md index f149ce8f7d..241c82e569 100644 --- a/docs/performance-analysis.md +++ b/docs/performance-analysis.md @@ -4970,3 +4970,18 @@ manual metadata reconciliation continues normally. API failures still fail rathe than masquerading as obsolete data. Fresh previews pay one bounded source/PR API check; no fixed elapsed-time improvement is claimed. Old queued workflow definitions retain their original steps, and rejected producer events still need independent recovery. + +### Post-merge measurement queue admission + +Main benchmark run `42354657340` for merge `1278c8943a6abd236a2f20e3937cc8e23a0f95a7` +was automatically canceled while pending when the next main merge requested a run. The workflow +already disabled active-run cancellation, but omitted `queue`, whose default permits only one +pending run. This loses a requested revision before measurement and is separate from CPU-count +agreement, runner performance, and publication backlogs. No agent cancellation caused this event. + +Use `queue: max` for the existing per-ref main measurement concurrency group, keeping +`cancel-in-progress: false`. A committed workflow regression checks all three properties. +GitHub permits up to 100 pending runs; overflow still cancels additional requests. Existing +canceled revisions need explicit reconciliation/backfill, not a claim that changing the workflow +recovered them. Measurement commands, sampling, CPU selection, artifact identity, PR checks and +publication policies are unchanged. There is no runtime or elapsed-time improvement claim. diff --git a/tools/benchmark_history_test.py b/tools/benchmark_history_test.py index c5c58905e4..d2a2cf3375 100644 --- a/tools/benchmark_history_test.py +++ b/tools/benchmark_history_test.py @@ -185,6 +185,13 @@ def test_comparisons_are_rendered_and_bound_to_head_binary(self): with self.assertRaisesRegex(ValueError, "head binary"): history.retain(Path(directory), value, source(2)) + def test_main_measurement_queue_preserves_pending_merged_revisions(self): + workflow = repository_file('.github/workflows/benchmarks.yml').read_text() + concurrency = workflow.split('\nconcurrency:\n', 1)[1].split('\njobs:', 1)[0] + self.assertIn('group: benchmarks-${{ github.ref }}', concurrency) + self.assertRegex(concurrency, r'(?m)^ queue: max$') + self.assertRegex(concurrency, r'(?m)^ cancel-in-progress: false$') + def test_workflow_keeps_measurement_unprivileged_and_publication_serialized(self): measure = repository_file(".github/workflows/benchmarks.yml").read_text() shard_action = repository_file(".github/actions/benchmark-measure/action.yml").read_text()