ci: run tests in parallel on hosted runners (the required lane was forced serial everywhere) - #4548
Merged
Merged
Conversation
…erial everywhere) `tests (3.11)` is one of two required checks and it has been timing out. Raising its cap from 90 to 150 minutes (#4539) did NOT help — the next run consumed 150.3 minutes. That ruled out "slightly too slow" and pointed at the real cause. Per-step timings from the cancelled job on PR #4525 (a four-file, formatting-only change whose reformat was proven AST-identical): step 23 Run Tests with Coverage (Python 3.11) 147.5 min CANCELLED everything else combined ~4 min So the test step is effectively the whole job, and it runs with `-n 0` — fully serial. That override is unconditional, and its stated reason is specific to one runner class: "the memory-constrained local runner fleet has repeatedly crashed xdist workers even at low fan-out". That reason does not hold on a hosted runner. The failing job ran on `GitHub Actions 1000355575` — 4 vCPU / 16 GB, not memory-constrained. And the timings across recent runs line up exactly with runner class rather than with any code change: 77.7 min SUCCESS d-sorg-local-Oglaptop-1 (fleet hardware, fast cores) 150.3 min CANCELLED hosted (slower per-core, same serial run) 93.5 min CANCELLED d-sorg-local-Desktop-3 91.2 min CANCELLED d-sorg-local-Desktop-6 Forcing serial execution onto slower hosted cores is what pushes the job past any cap. So the fan-out is now chosen per runner class instead of globally: PYTEST_FANOUT: ${{ needs.pick-runner.outputs.runner == 'd-sorg-fleet' && '0' || 'auto' }} pytest_args+=(-n "${PYTEST_FANOUT:-0}") Fleet hosts keep `-n 0`, so the crash this override was written to prevent is still prevented. Hosted runners get `-n auto` (4 workers), which should take the test step from ~148 minutes to roughly 40. Two details that make this safer than it looks: - `--dist loadscope` comes from the repo's own addopts and is unchanged, so a module's tests stay on one worker. That matters because the Qt tests use 15 second `waitUntil` timeouts and are load-sensitive; loadscope keeps them from being split across workers. - The default is `0`, not `auto` (`${PYTEST_FANOUT:-0}`). If the variable is ever unset the behaviour is today's serial run, so this fails safe. This follows the conditional the tests job already uses for `PIP_CACHE_DIR`, which switches on the same `needs.pick-runner.outputs.runner == 'd-sorg-fleet'` test. The 150-minute cap from #4539 is left in place as headroom rather than reverted; with parallelism it should be far from binding. If it stops being needed it is a one-line revert. Verified: `scripts/validate_workflows.py`, `scripts/check_workflow_pinning.py` and `scripts/check_blocking_quality_gates.py` all pass; the tests job still parses to 27 steps; and the env var is on the `tests` job and NOT on `quality-gate` (both jobs carry an identical `PIP_DEFAULT_TIMEOUT: "120"` line, so a first-match patch lands in the wrong job — worth knowing when editing this file). Part of #4532 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This was referenced Aug 18, 2026
dieterolson
added a commit
that referenced
this pull request
Aug 19, 2026
… exposed (#4572) * @ fix(tests): repair the order- and load-dependent failures parallelism exposed Two failures that #4548 surfaced by running the suite under xdist. Neither is caused by the diffs that hit them. E-stop state leak. `main.control_context` is a module-level singleton shared by the whole P1AM backend suite, and `test_estop_trigger` cleared the latch as the last statement of its body. Any assertion above it failing left E-stop engaged for whatever xdist scheduled next on that worker, which reported as test_pid_tuning_tag_guards failing with "E-stop active; output writes are inhibited." instead of its own expected message. The clear moves into a `finally`, and a new package conftest clears the latch on both boundaries of every test so the coupling cannot come back - teardown alone would still leave each module first test dependent on whatever ran before it. Reproduced deterministically before fixing (latch E-stop, run the victim: 1 failed; with the guard: 2 passed). Full backend suite: 1252 passed, 6 skipped, both serial and under -n 4. Wall-clock flake. test_100k_rows_remain_bounded_and_fast asserted elapsed < 0.5, which main itself tripped at 0.5146 s under contention. The assertion guards against an accidental super-linear pass, which at 100k rows costs minutes rather than tenths of a second, so the ceiling moves to 5.0 s with the reasoning inline. displayed_count == 2_000 remains the deterministic contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> @ * @ fix(ci): allowlist the fixture-only P1AM backend conftest quality-gate Changed Test Assertion Check flags any changed test file with no behavioral assertion and names the remedy: add a fixture-only pattern to scripts/test_assertion_allowlist.txt. The new conftest holds one autouse fixture and no test functions, so it has nothing to assert about; the behaviour it protects is asserted by the modules it isolates. Entry carries the rationale, matching the surrounding convention. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> @ * @ docs(spec): place the new changelog row in descending order The §12 table is newest-first (1.17.46 above 1.17.45). The row was inserted after its anchor rather than before it, so the column read 46, 48, 45. Moves it above 1.17.46. The remaining disorder on that line is the pre-existing pair of duplicate 1.17.40 rows glued together by a union merge, tracked as #4569; splitting them needs a renumber decision and is deliberately not done here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> @ --------- Co-authored-by: codex-scheduled <codex-scheduled@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ci: run the test suite in parallel on hosted runners (it was forced serial everywhere)
tests (3.11)is one of two required checks and it has been timing out. Raisingits cap from 90 to 150 minutes (#4539) did NOT help — the next run consumed
150.3 minutes. That ruled out "slightly too slow" and pointed at the real cause.
Per-step timings from the cancelled job on PR #4525 (a four-file, formatting-only
change whose reformat was proven AST-identical):
step 23 Run Tests with Coverage (Python 3.11) 147.5 min CANCELLED
everything else combined ~4 min
So the test step is effectively the whole job, and it runs with
-n 0— fullyserial. That override is unconditional, and its stated reason is specific to one
runner class: "the memory-constrained local runner fleet has repeatedly crashed
xdist workers even at low fan-out".
That reason does not hold on a hosted runner. The failing job ran on
GitHub Actions 1000355575— 4 vCPU / 16 GB, not memory-constrained. And thetimings across recent runs line up exactly with runner class rather than with any
code change:
77.7 min SUCCESS d-sorg-local-Oglaptop-1 (fleet hardware, fast cores)
150.3 min CANCELLED hosted (slower per-core, same serial run)
93.5 min CANCELLED d-sorg-local-Desktop-3
91.2 min CANCELLED d-sorg-local-Desktop-6
Forcing serial execution onto slower hosted cores is what pushes the job past any
cap. So the fan-out is now chosen per runner class instead of globally:
PYTEST_FANOUT: ${{ needs.pick-runner.outputs.runner == 'd-sorg-fleet' && '0' || 'auto' }}
pytest_args+=(-n "${PYTEST_FANOUT:-0}")
Fleet hosts keep
-n 0, so the crash this override was written to prevent isstill prevented. Hosted runners get
-n auto(4 workers), which should take thetest step from ~148 minutes to roughly 40.
Two details that make this safer than it looks:
--dist loadscopecomes from the repo's own addopts and is unchanged, so amodule's tests stay on one worker. That matters because the Qt tests use 15
second
waitUntiltimeouts and are load-sensitive; loadscope keeps them frombeing split across workers.
0, notauto(${PYTEST_FANOUT:-0}). If the variable is everunset the behaviour is today's serial run, so this fails safe.
This follows the conditional the tests job already uses for
PIP_CACHE_DIR,which switches on the same
needs.pick-runner.outputs.runner == 'd-sorg-fleet'test.
The 150-minute cap from #4539 is left in place as headroom rather than reverted;
with parallelism it should be far from binding. If it stops being needed it is a
one-line revert.
Verified:
scripts/validate_workflows.py,scripts/check_workflow_pinning.pyand
scripts/check_blocking_quality_gates.pyall pass; the tests job stillparses to 27 steps; and the env var is on the
testsjob and NOT onquality-gate(both jobs carry an identicalPIP_DEFAULT_TIMEOUT: "120"line,so a first-match patch lands in the wrong job — worth knowing when editing this
file).
Part of #4532
Co-Authored-By: Claude Opus 5 noreply@anthropic.com