Skip to content

ci: run tests in parallel on hosted runners (the required lane was forced serial everywhere) - #4548

Merged
dieterolson merged 1 commit into
mainfrom
ci/parallel-tests-on-hosted
Aug 18, 2026
Merged

ci: run tests in parallel on hosted runners (the required lane was forced serial everywhere)#4548
dieterolson merged 1 commit into
mainfrom
ci/parallel-tests-on-hosted

Conversation

@dieterolson

Copy link
Copy Markdown
Collaborator

ci: run the test suite in parallel on hosted runners (it was forced serial everywhere)

tests (3.11) is one of two required checks and it has been timing out. Raising
its cap from 90 to 150 minutes (#4539) did NOT help — the next run consumed
150.3 minutes. That ruled out "slightly too slow" and pointed at the real cause.

Per-step timings from the cancelled job on PR #4525 (a four-file, formatting-only
change whose reformat was proven AST-identical):

step 23 Run Tests with Coverage (Python 3.11) 147.5 min CANCELLED
everything else combined ~4 min

So the test step is effectively the whole job, and it runs with -n 0 — fully
serial. That override is unconditional, and its stated reason is specific to one
runner class: "the memory-constrained local runner fleet has repeatedly crashed
xdist workers even at low fan-out".

That reason does not hold on a hosted runner. The failing job ran on
GitHub Actions 1000355575 — 4 vCPU / 16 GB, not memory-constrained. And the
timings across recent runs line up exactly with runner class rather than with any
code change:

77.7 min SUCCESS d-sorg-local-Oglaptop-1 (fleet hardware, fast cores)
150.3 min CANCELLED hosted (slower per-core, same serial run)
93.5 min CANCELLED d-sorg-local-Desktop-3
91.2 min CANCELLED d-sorg-local-Desktop-6

Forcing serial execution onto slower hosted cores is what pushes the job past any
cap. So the fan-out is now chosen per runner class instead of globally:

PYTEST_FANOUT: ${{ needs.pick-runner.outputs.runner == 'd-sorg-fleet' && '0' || 'auto' }}
pytest_args+=(-n "${PYTEST_FANOUT:-0}")

Fleet hosts keep -n 0, so the crash this override was written to prevent is
still prevented. Hosted runners get -n auto (4 workers), which should take the
test step from ~148 minutes to roughly 40.

Two details that make this safer than it looks:

  • --dist loadscope comes from the repo's own addopts and is unchanged, so a
    module's tests stay on one worker. That matters because the Qt tests use 15
    second waitUntil timeouts and are load-sensitive; loadscope keeps them from
    being split across workers.
  • The default is 0, not auto (${PYTEST_FANOUT:-0}). If the variable is ever
    unset the behaviour is today's serial run, so this fails safe.

This follows the conditional the tests job already uses for PIP_CACHE_DIR,
which switches on the same needs.pick-runner.outputs.runner == 'd-sorg-fleet'
test.

The 150-minute cap from #4539 is left in place as headroom rather than reverted;
with parallelism it should be far from binding. If it stops being needed it is a
one-line revert.

Verified: scripts/validate_workflows.py, scripts/check_workflow_pinning.py
and scripts/check_blocking_quality_gates.py all pass; the tests job still
parses to 27 steps; and the env var is on the tests job and NOT on
quality-gate (both jobs carry an identical PIP_DEFAULT_TIMEOUT: "120" line,
so a first-match patch lands in the wrong job — worth knowing when editing this
file).

Part of #4532

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

…erial everywhere)

`tests (3.11)` is one of two required checks and it has been timing out. Raising
its cap from 90 to 150 minutes (#4539) did NOT help — the next run consumed
150.3 minutes. That ruled out "slightly too slow" and pointed at the real cause.

Per-step timings from the cancelled job on PR #4525 (a four-file, formatting-only
change whose reformat was proven AST-identical):

  step 23  Run Tests with Coverage (Python 3.11)   147.5 min   CANCELLED
  everything else combined                         ~4 min

So the test step is effectively the whole job, and it runs with `-n 0` — fully
serial. That override is unconditional, and its stated reason is specific to one
runner class: "the memory-constrained local runner fleet has repeatedly crashed
xdist workers even at low fan-out".

That reason does not hold on a hosted runner. The failing job ran on
`GitHub Actions 1000355575` — 4 vCPU / 16 GB, not memory-constrained. And the
timings across recent runs line up exactly with runner class rather than with any
code change:

  77.7 min  SUCCESS    d-sorg-local-Oglaptop-1   (fleet hardware, fast cores)
  150.3 min CANCELLED  hosted                    (slower per-core, same serial run)
   93.5 min CANCELLED  d-sorg-local-Desktop-3
   91.2 min CANCELLED  d-sorg-local-Desktop-6

Forcing serial execution onto slower hosted cores is what pushes the job past any
cap. So the fan-out is now chosen per runner class instead of globally:

  PYTEST_FANOUT: ${{ needs.pick-runner.outputs.runner == 'd-sorg-fleet' && '0' || 'auto' }}
  pytest_args+=(-n "${PYTEST_FANOUT:-0}")

Fleet hosts keep `-n 0`, so the crash this override was written to prevent is
still prevented. Hosted runners get `-n auto` (4 workers), which should take the
test step from ~148 minutes to roughly 40.

Two details that make this safer than it looks:

- `--dist loadscope` comes from the repo's own addopts and is unchanged, so a
  module's tests stay on one worker. That matters because the Qt tests use 15
  second `waitUntil` timeouts and are load-sensitive; loadscope keeps them from
  being split across workers.
- The default is `0`, not `auto` (`${PYTEST_FANOUT:-0}`). If the variable is ever
  unset the behaviour is today's serial run, so this fails safe.

This follows the conditional the tests job already uses for `PIP_CACHE_DIR`,
which switches on the same `needs.pick-runner.outputs.runner == 'd-sorg-fleet'`
test.

The 150-minute cap from #4539 is left in place as headroom rather than reverted;
with parallelism it should be far from binding. If it stops being needed it is a
one-line revert.

Verified: `scripts/validate_workflows.py`, `scripts/check_workflow_pinning.py`
and `scripts/check_blocking_quality_gates.py` all pass; the tests job still
parses to 27 steps; and the env var is on the `tests` job and NOT on
`quality-gate` (both jobs carry an identical `PIP_DEFAULT_TIMEOUT: "120"` line,
so a first-match patch lands in the wrong job — worth knowing when editing this
file).

Part of #4532

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@dieterolson
dieterolson merged commit fe78bfc into main Aug 18, 2026
18 checks passed
dieterolson added a commit that referenced this pull request Aug 19, 2026
… exposed (#4572)

* @
fix(tests): repair the order- and load-dependent failures parallelism exposed

Two failures that #4548 surfaced by running the suite under xdist. Neither is
caused by the diffs that hit them.

E-stop state leak. `main.control_context` is a module-level singleton shared
by the whole P1AM backend suite, and `test_estop_trigger` cleared the latch as
the last statement of its body. Any assertion above it failing left E-stop
engaged for whatever xdist scheduled next on that worker, which reported as
test_pid_tuning_tag_guards failing with "E-stop active; output writes are
inhibited." instead of its own expected message. The clear moves into a
`finally`, and a new package conftest clears the latch on both boundaries of
every test so the coupling cannot come back - teardown alone would still leave
each module first test dependent on whatever ran before it.

Reproduced deterministically before fixing (latch E-stop, run the victim:
1 failed; with the guard: 2 passed). Full backend suite: 1252 passed, 6
skipped, both serial and under -n 4.

Wall-clock flake. test_100k_rows_remain_bounded_and_fast asserted
elapsed < 0.5, which main itself tripped at 0.5146 s under contention. The
assertion guards against an accidental super-linear pass, which at 100k rows
costs minutes rather than tenths of a second, so the ceiling moves to 5.0 s
with the reasoning inline. displayed_count == 2_000 remains the deterministic
contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@

* @
fix(ci): allowlist the fixture-only P1AM backend conftest

quality-gate Changed Test Assertion Check flags any changed test file with no
behavioral assertion and names the remedy: add a fixture-only pattern to
scripts/test_assertion_allowlist.txt. The new conftest holds one autouse
fixture and no test functions, so it has nothing to assert about; the
behaviour it protects is asserted by the modules it isolates. Entry carries
the rationale, matching the surrounding convention.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@

* @
docs(spec): place the new changelog row in descending order

The §12 table is newest-first (1.17.46 above 1.17.45). The row was inserted
after its anchor rather than before it, so the column read 46, 48, 45. Moves
it above 1.17.46.

The remaining disorder on that line is the pre-existing pair of duplicate
1.17.40 rows glued together by a union merge, tracked as #4569; splitting them
needs a renumber decision and is deliberately not done here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@

---------

Co-authored-by: codex-scheduled <codex-scheduled@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant