ci: raise the tests job timeout 90 -> 150 minutes (the required check is at its cap) - #4539
Merged
Conversation
`tests (3.11)` is one of only two required status checks on `main`, and its runtime has grown into its own 90-minute cap. Measured across the last ten completed `CI Standard` runs: success 77.7 min <- the ONLY pass, at 86% of the budget cancelled 93.5 min <- timeout kill cancelled 91.8 min <- timeout kill cancelled 91.2 min <- timeout kill cancelled 90.5 min <- timeout kill cancelled 52.3 / 27.7 / 13.7 min failure 26.2 / 10.0 min One pass in ten. The runs at 90+ minutes record **zero failed steps** — they are timeout kills, not test failures, so a healthy run has roughly twelve minutes of headroom and any contention on a shared runner consumes it. Demonstrated concretely by PR #4525, a four-file formatting-only change whose reformat was proven AST-identical: its `tests (3.11)` was cancelled at 90.0 minutes, re-run, and cancelled again at 90.3 minutes. No change to that PR can make it pass. Combined with `strict = true` — which forces PRs to merge one at a time and re-run CI after every sibling merge — a ~10% pass rate on the required check makes a queue effectively unlandable without repeated luck. This is a STOPGAP, deliberately reversible, and it does not pretend to be the fix. 150 minutes is ~2x the observed successful run and ~1.6x the observed maximum, which is headroom rather than a new ceiling to grow into. The actual remedies are tracked in #4532: shard the suite across parallel jobs so the required context no longer has a single-point timeout, scope the required lane to the contract-critical set, and run `--durations` first to establish whether this is broad growth or a handful of slow modules. One known contributor is that `addopts` carries `-n auto --dist loadscope`, so Qt tests with 15-second `waitUntil` timeouts are load-sensitive and wall-clock varies with the runner. Worth noting what this is NOT: #4487 already cached the `tools_core` Rust wheel so the lane stops installing a Rust toolchain and rebuilding the wheel per job, and every measurement above is from AFTER that merged. The remaining runtime is the test suite itself, not the toolchain. Only `jobs.tests.timeout-minutes` changes. `quality-gate` (60) and `rust-quality-gate` (15) are untouched. Verified with the repo's own gates: `scripts/validate_workflows.py`, `scripts/check_workflow_pinning.py` and `scripts/check_blocking_quality_gates.py` all pass, and the tests job still parses to its full 27 steps. Part of #4532 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
dieterolson
added a commit
that referenced
this pull request
Aug 18, 2026
…erial everywhere) (#4548) `tests (3.11)` is one of two required checks and it has been timing out. Raising its cap from 90 to 150 minutes (#4539) did NOT help — the next run consumed 150.3 minutes. That ruled out "slightly too slow" and pointed at the real cause. Per-step timings from the cancelled job on PR #4525 (a four-file, formatting-only change whose reformat was proven AST-identical): step 23 Run Tests with Coverage (Python 3.11) 147.5 min CANCELLED everything else combined ~4 min So the test step is effectively the whole job, and it runs with `-n 0` — fully serial. That override is unconditional, and its stated reason is specific to one runner class: "the memory-constrained local runner fleet has repeatedly crashed xdist workers even at low fan-out". That reason does not hold on a hosted runner. The failing job ran on `GitHub Actions 1000355575` — 4 vCPU / 16 GB, not memory-constrained. And the timings across recent runs line up exactly with runner class rather than with any code change: 77.7 min SUCCESS d-sorg-local-Oglaptop-1 (fleet hardware, fast cores) 150.3 min CANCELLED hosted (slower per-core, same serial run) 93.5 min CANCELLED d-sorg-local-Desktop-3 91.2 min CANCELLED d-sorg-local-Desktop-6 Forcing serial execution onto slower hosted cores is what pushes the job past any cap. So the fan-out is now chosen per runner class instead of globally: PYTEST_FANOUT: ${{ needs.pick-runner.outputs.runner == 'd-sorg-fleet' && '0' || 'auto' }} pytest_args+=(-n "${PYTEST_FANOUT:-0}") Fleet hosts keep `-n 0`, so the crash this override was written to prevent is still prevented. Hosted runners get `-n auto` (4 workers), which should take the test step from ~148 minutes to roughly 40. Two details that make this safer than it looks: - `--dist loadscope` comes from the repo's own addopts and is unchanged, so a module's tests stay on one worker. That matters because the Qt tests use 15 second `waitUntil` timeouts and are load-sensitive; loadscope keeps them from being split across workers. - The default is `0`, not `auto` (`${PYTEST_FANOUT:-0}`). If the variable is ever unset the behaviour is today's serial run, so this fails safe. This follows the conditional the tests job already uses for `PIP_CACHE_DIR`, which switches on the same `needs.pick-runner.outputs.runner == 'd-sorg-fleet'` test. The 150-minute cap from #4539 is left in place as headroom rather than reverted; with parallelism it should be far from binding. If it stops being needed it is a one-line revert. Verified: `scripts/validate_workflows.py`, `scripts/check_workflow_pinning.py` and `scripts/check_blocking_quality_gates.py` all pass; the tests job still parses to 27 steps; and the env var is on the `tests` job and NOT on `quality-gate` (both jobs carry an identical `PIP_DEFAULT_TIMEOUT: "120"` line, so a first-match patch lands in the wrong job — worth knowing when editing this file). Part of #4532 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ci: raise the tests job timeout 90 -> 150 minutes
tests (3.11)is one of only two required status checks onmain, and itsruntime has grown into its own 90-minute cap. Measured across the last ten
completed
CI Standardruns:success 77.7 min <- the ONLY pass, at 86% of the budget
cancelled 93.5 min <- timeout kill
cancelled 91.8 min <- timeout kill
cancelled 91.2 min <- timeout kill
cancelled 90.5 min <- timeout kill
cancelled 52.3 / 27.7 / 13.7 min
failure 26.2 / 10.0 min
One pass in ten. The runs at 90+ minutes record zero failed steps — they are
timeout kills, not test failures, so a healthy run has roughly twelve minutes of
headroom and any contention on a shared runner consumes it.
Demonstrated concretely by PR #4525, a four-file formatting-only change whose
reformat was proven AST-identical: its
tests (3.11)was cancelled at 90.0minutes, re-run, and cancelled again at 90.3 minutes. No change to that PR can
make it pass. Combined with
strict = true— which forces PRs to merge one at atime and re-run CI after every sibling merge — a ~10% pass rate on the required
check makes a queue effectively unlandable without repeated luck.
This is a STOPGAP, deliberately reversible, and it does not pretend to be the
fix. 150 minutes is ~2x the observed successful run and ~1.6x the observed
maximum, which is headroom rather than a new ceiling to grow into.
The actual remedies are tracked in #4532: shard the suite across parallel jobs so
the required context no longer has a single-point timeout, scope the required
lane to the contract-critical set, and run
--durationsfirst to establishwhether this is broad growth or a handful of slow modules. One known contributor
is that
addoptscarries-n auto --dist loadscope, so Qt tests with 15-secondwaitUntiltimeouts are load-sensitive and wall-clock varies with the runner.Worth noting what this is NOT: #4487 already cached the
tools_coreRust wheelso the lane stops installing a Rust toolchain and rebuilding the wheel per job,
and every measurement above is from AFTER that merged. The remaining runtime is
the test suite itself, not the toolchain.
Only
jobs.tests.timeout-minuteschanges.quality-gate(60) andrust-quality-gate(15) are untouched. Verified with the repo's own gates:scripts/validate_workflows.py,scripts/check_workflow_pinning.pyandscripts/check_blocking_quality_gates.pyall pass, and the tests job stillparses to its full 27 steps.
Part of #4532
Co-Authored-By: Claude Opus 5 noreply@anthropic.com