ci: bound the Qt package install that hung for a full job timeout twice - #92
Conversation
…wice Measured, not suspected. `protected execution (ubuntu-latest, agents)` was cancelled at exactly 50 minutes and 27 seconds -- its own timeout -- and the step it was in is "Install Qt offscreen system libraries". It did that twice on 2026-08-19 and never once succeeded in a fourteen-run window, taking the evidence gate of two pull requests with it each time. The tests themselves are not slow: the same agents suite runs 995 tests in 17.45 seconds on the laboratory machine. This is a package install waiting forever, which is what plain apt-get does when the runner image's own unattended upgrades hold the dpkg lock. Three bounds, each closing one way to wait for nobody: a step timeout so a hang becomes a fast named failure a re-run can clear rather than a cancelled job that cannot; a dpkg lock timeout so the common case does not fail at all; and the non-interactive frontend so apt never waits for an answer. Both workflow copies are changed. The evidence-gate copy is documented in its own header as an organization-required workflow, so the organization's copy may be the authoritative one and this change may not take effect there -- said plainly rather than assumed either way.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
The bound worked and revealed the real cause. With the step limited to eight minutes it failed by name instead of vanishing at fifty, and the log shows where the time went: a 2 minute 14 second stall inside `apt-get update` against the Ubuntu archive, not the dpkg lock the lock-timeout was aimed at. So the fix is to not go to the network at all when there is nothing to fetch. The runner image already carries these four libraries, so the step now asks dpkg what is missing, exits immediately when the answer is nothing, and updates the index only for the packages it actually has to install. A step that does nothing when there is nothing to do cannot hang. The eight-minute bound and the lock timeout stay. The bound has already earned its place once: it is the reason this cause is known rather than guessed at.
|
@codex review Head is |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
The bound earned its place immediately: with the step limited to eight minutes it failed by name instead of vanishing at fifty, and the log then said where the time actually goes. So it is The fix is therefore not to wait better but to not go at all. The runner image already carries these four libraries, so the step now asks The eight-minute bound and the lock timeout stay — the bound is the reason this cause is known rather than guessed at, and the next stall of a different kind will name itself the same way. Still true and still worth repeating: the evidence-gate workflow says in its own header that it is an organization-required workflow, so the organization's copy may be authoritative and this change may not reach it. If the gate keeps hanging after this merges, that is where it lives. |
The previous commit skipped the package index update when nothing was missing. Run 32303335407 shows that helps only some runners: one job reported "missing: libegl1" and another reported nothing at all, so the runner images differ and the missing-package path is real. That path then hit the bound again, and its log gives the cause exactly. The step spent 2 minutes 13 seconds on azure.archive.ubuntu.com before apt ignored it, then a further 6 minutes on package indexes from archive.ubuntu.com, and the bound fired with nothing installed. Two changes follow from that measurement. Every apt-get call now carries the same network bounds, not just the index update: a 15 second connect timeout for http and https and two retries. The download of the package itself goes through the same unreachable mirror list, so bounding only the index update would let the first install consume the whole step budget and the fallback would never run. The step now tries the package index already on the image before refreshing it. A mirror that is unreachable cannot delay an install that never asks it for an index. The refresh stays as a fallback for when the on-image index is too old to satisfy the package. The four paths were exercised with stubbed dpkg and apt-get against the step body extracted from the workflow file: nothing missing makes no apt call at all; a missing package with a usable on-image index makes one install call and no update; a stale index falls back to update then install; and a package that cannot be installed either way still fails the step, because a genuinely missing library must not be passed over in silence. The eight minute bound stays. It is the reason this cause is known rather than guessed at, twice now.
The caveat I carried on this pull request is refuted, and by this pull request's own logI wrote twice that the evidence-gate workflow calls itself organization-required, so the organization's copy might be the one that runs and this change might never reach it. That is wrong, and here is the measurement that settles it. Run So the repository's own copy is what executes. Organization-required governs which checks must be green, not which file runs. Merging this pull request really does unblock #89, #90 and #91. There is nothing here for the owner to change at the organization level. What the bound found this timeThe previous commit skipped the package index update when nothing was missing. That helped some runners and not others, and the difference is not flakiness in the step — it is the runner images differing. In that one run:
The missing-package path then hit the bound again, and its log gives the cause to the second: The two changes that follow from itEvery The step tries the package index already on the image before refreshing it. A mirror that is unreachable cannot delay an install that never asks it for an index. The refresh remains, as a fallback for when the on-image index is too old to satisfy the package. The four paths, exercised rather than assumedThe missing-package path has never once completed on CI, so I ran the step body — extracted from the workflow file by a YAML parser, not retyped — against stubbed
The last row is deliberate. A library that genuinely cannot be installed must fail the step, not be passed over in silence. What one green round will and will not proveBecause the images differ, a runner that reports nothing missing exercises none of this. Only a job that reports The eight minute bound stays. It has now named the real cause twice, where the fifty minute silence named nothing. |
…into failure
The package install itself is now fixed and measured: in run 32306229771 all
four Linux jobs found libegl1 missing, installed it from the package index
already on the image in about 7 seconds, and printed the success line. The step
then reported failure anyway, and every one of the four jobs was lost.
The cause is the shell, not the install. These jobs declare `shell: bash -el
{0}`, a login shell, and the runner starts it directly, so SHLVL is 1. That is
the exact condition under which ~/.bash_logout runs its clear_console line;
clear_console fails when there is no terminal, and its status replaces the
argument of an explicit exit. Measured on Ubuntu 22.04:
bash -el script with `exit 0`, SHLVL=2 -> 0
bash -el script with `exit 0`, SHLVL=1 -> 1
bash -el script falling off the end -> 0
clear_console -q with no terminal -> 1
So the step is now if/else and never calls exit. It falls off the end, and the
status the runner sees is the status of the last command it ran.
This is the same defect shape as the two before it: a property measured through
a layer that silently rewrites it. It is one-directional and does not weaken
anything -- exit 1 still fails, so no failure was ever reported as success.
The harness that was supposed to catch this did not, because it ran the step
body under `bash -eo pipefail` while the workflow runs it under `bash -el`. It
now reads the shell from the workflow, scrubs SHLVL so a login shell behaves as
it does on the runner, and refuses a login-shell step that calls exit at all --
a static check, because a Windows host has no ~/.bash_logout and cannot
reproduce the behaviour by running anything. The refusal was confirmed by
putting the defect back into a copy.
The third cause was the shell, not the package — and it is the one that was actually blockingRun These jobs declare Measured on Ubuntu 22.04:
There is also a control in the record: in run So the step is now The trap is one-directional and weakens nothing. The harness I wrote for this step passed while the step failedThat is worth stating plainly. It ran the step body under It now reads the shell from the workflow file, scrubs State of this pull request
|
1 participant
What actually stopped two pull requests
protected execution (ubuntu-latest, agents)was cancelled at 50 minutes and 27 seconds — its owntimeout-minutes: 50— and the step it was sitting in is:It never ran a single test. It did this twice on 2026-08-19 and did not succeed once in a fourteen-run window, and each time it took the evidence gate of a pull request with it — a cancelled required check that, by the gate's own attempt-1 rule, no re-run can ever clear.
It is not the tests
The same suite runs 995 tests in 17.45 seconds on the laboratory machine. Successful jobs across that window take 10 to 28 minutes against the 50-minute limit, margins of 22 to 40 minutes (
evidence/tools/job_durations.py). Nothing here is slow; one step waits forever.That is what plain
apt-getdoes when the runner image's own unattended upgrades hold the dpkg lock.The change
Three bounds, each closing one way to wait for nobody:
timeout-minutes: 8on the step — a hang becomes a fast, named failure that a re-run can clear, instead of a cancelled job that cannot.-o DPkg::Lock::Timeout=180— the common case stops failing at all; apt gives the lock three minutes and then reports it.DEBIAN_FRONTEND=noninteractive— removes the other way apt can block on a machine with nobody at the keyboard.What this does not claim
Both workflow copies are changed. The evidence-gate copy says in its own header that it is configured as an organization-required workflow, so the organization's copy may be the authoritative one and this change may not take effect there. Said plainly rather than assumed in either direction — if the gate keeps hanging after this merges, the organization copy is where it must be fixed.
tests/docs: 68 passed. Both workflow files still parse.Written with assistance from Claude (Anthropic).