Conversation
The short mock-stack soak and the golden-run replay both start the launcher, which imports PySide6. main.yml and the protected evidence gate install libegl1, libgl1, libxkbcommon0 and libdbus-1-3 before doing so. nightly.yml installed none of them. The soak therefore failed on every one of the three nights it has run, with ImportError: libEGL.so.1 in log-launcher.txt, while its own prerequisite gate passed. The same import produced 35 collection errors in the replay job. This copies main.yml's step into the two jobs that start the launcher. The third job does not, so it does not get the step.
|
@codex review Fresh pull request. Please review at head |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf535eea1e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - name: Install Qt offscreen system libraries (Linux) | ||
| if: runner.os == 'Linux' | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3 |
There was a problem hiding this comment.
This change fixes a confirmed three-night CI escape by copying package installs into two nightly jobs, but it adds no deterministic validator or prevention record for the omission. Because this workflow remains outside the default PR lanes, removing either job’s step or a required library can still merge cleanly and recreate the collection/launcher failures being corrected; add a default-CI guard covering both golden-replay and mock-stack-short-soak, with a stable prevention ID and red/green evidence.
AGENTS.md reference: AGENTS.md:L361-L365
Useful? React with 👍 / 👎.
Landed by the batch lander. The lane's own report and the coordinator's verification are recorded on the pull request.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 32df84f6f3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for library in _QT_LINUX_LIBRARIES: | ||
| assert library in step, f"{job_name} omits required Qt library {library!r}" |
There was a problem hiding this comment.
If either apt-get install command is removed or replaced with echo libegl1 libgl1 libxkbcommon0 libdbus-1-3, this guard still passes because it only searches for each package name anywhere in the step. I verified that mutation against both nightly job blocks, so the confirmed prerequisite regression can recur while default CI remains green; parse the step's run command and require the packages to be arguments to apt-get install, with a failing mutation test.
AGENTS.md reference: AGENTS.md:L376-L379
Useful? React with 👍 / 👎.
Landed by the batch lander. The lane's own report and the coordinator's verification are recorded on the pull request.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b31b0fac85
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "NARRATION-GUARD-VACUOUS-PAYLOAD-FALSE-GREEN-305": "sha256:42e04d1f54e633de12ec10c758deb4369c4800aa443d1d399245bcafdea46e8b", | ||
| "NARRATION-GUARD-WATERMARK-OVERRUN-FALSE-GREEN-315": "sha256:89648300a0cc6d8a55828b3a08a40b3aaea0f8e14650fef3e3f080df9020c96f", | ||
| "NARRATION-GUARD-WRONG-BRANCH-FALSE-GREEN-306": "sha256:cd3a5063908083f1ad15b55aa000561a3ba12b7635b0d2b2ae46891d8d4b7705", | ||
| "NIGHTLY-QT-PREREQUISITES-055": "sha256:68a81dde30c299a386faa6ee6c29db62ed78122df711e4c4a41c7eaf1fbcd5be", |
There was a problem hiding this comment.
Adding the second guard changed the digest of NIGHTLY-QT-PREREQUISITES-055, but this baseline still contains the digest generated before that edit (68a81d... instead of the currently rendered 3bb6f3...). Consequently, the default remaining partition fails both test_live_registry_satisfies_the_removal_baseline and test_baseline_is_in_sync_with_the_live_registry, blocking CI until the baseline is regenerated from the final registry.
AGENTS.md reference: AGENTS.md:L445-L448
Useful? React with 👍 / 👎.
| dependencies = job.index("- name: Install dependencies") | ||
| assert install < dependencies, f"{job_name} installs Qt libraries after Python dependencies" | ||
| step = job[install:dependencies] | ||
| assert "if: runner.os == 'Linux'" in step |
There was a problem hiding this comment.
This substring assertion does not establish that the installation step actually has the required condition. I tested a valid mutation that sets if: ${{ false }} and adds # if: runner.os == 'Linux' in the same slice; both new guard tests still pass, while GitHub skips both package-install steps and the nightly Qt failures recur. Parse the workflow step and compare its effective if field rather than accepting the text anywhere in the slice.
AGENTS.md reference: AGENTS.md:L380-L386
Useful? React with 👍 / 👎.
Landed by the batch lander. The lane's own report and the coordinator's verification are recorded on the pull request.
|
@codex review Head is |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1881af2d45
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| run = step["run"] | ||
| assert "sudo apt-get update" in run | ||
| install_commands = [ | ||
| line.strip().split() for line in run.splitlines() if line.strip().startswith("sudo apt-get install ") |
There was a problem hiding this comment.
Fresh evidence beyond the echoed-command fix: inserting exit 0 between apt-get update and the preserved install line in both parsed run blocks is still accepted by _assert_nightly_qt_prerequisites, because it scans individual lines without validating executable control flow; under the workflow's configured Bash shell, the step then succeeds before installing anything. On Ubuntu images missing these libraries, the default remaining guard therefore stays green while the scheduled jobs recreate the collection/launcher failure. Require the exact fail-closed command block or add a mutation covering early exits and control operators.
AGENTS.md reference: AGENTS.md:L380-L386
Useful? React with 👍 / 👎.
|
@codex review Head is 51362ca. Reason for this correction: the previous parsed-workflow guard verified the Linux condition and the install command, but it accepted a successful early exit between apt-get update and apt-get install. That control-flow escape could leave the nightly job green without installing the Qt libraries. Correction: each affected job now permits exactly two non-empty commands in this step: apt-get update, then apt-get install with the four required libraries. The existing registered mutation guard now also inserts exit 0 and requires rejection. Evidence on this exact head: the predecessor helper accepted the early-exit mutation; the corrected test module passed 8 tests; read-only Ruff lint and format checks passed; the derived documentation pair reached a fixed point and was committed; the isolated docs gate passed 67 tests; the changed files passed UTF-8 and byte-order-mark checks. This is software evidence only and does not close a physical laboratory gate. |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Converted to a draft for the laboratory-run focus, owner decision 2026-08-18. The work is not rejected and nothing is lost: the branch is untouched, the Why now. The team is climbing to a week of continuous thermal-conductivity It returns to the queue after the run, or sooner if a measurement shows it |
…20260827 # Conflicts: # docs/architecture-montana-important.svg # docs/current_candidate_metrics.md # governance/agent_preventions.yaml # governance/agent_preventions_baseline.json
|
@codex review this pull request at head |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7656aa76f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| scope: repository | ||
| authority_source: AGENTS.md#primary-ai-first-rule-mistake-to-enforcement | ||
| applies_to: The golden-replay and mock-stack-short-soak nightly jobs' Linux Qt system-library installation steps. | ||
| classification: default_ci_coverage_escape |
There was a problem hiding this comment.
This lone record is being used for both the missing-library incident and its default-CI coverage escape, even though the PR now includes deterministic mutations demonstrating that earlier guards remained green for ineffective installs, disabled conditions, and early exits. The repository contract requires the runtime failure and the false-green escape to receive separate stable IDs with independently enforceable guards; without a linked false_green_pairs entry, the removal baseline cannot independently preserve the coverage disposition. Add the separate false-green record and regenerate the baseline.
AGENTS.md reference: AGENTS.md:L366-L369
Useful? React with 👍 / 👎.
| sudo apt-get update | ||
| sudo apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3 |
There was a problem hiding this comment.
When an Ubuntu mirror stalls, these unconditional apt-get calls can consume the nightly run before either test lane starts because neither new step has a step timeout or per-call network bounds. The current default workflow documents at .github/workflows/main.yml lines 79-130 that this exact installation previously hung for the full job timeout twice, and consequently uses an eight-minute step bound, bounded apt options, and the on-image package index first; the nightly copy reintroduces the confirmed failure shape, while tests/test_ci_safe_sqlite_contract.py lines 165-170 actively requires the unbounded block. Reuse the bounded installation contract in both lanes and update the guard to enforce it.
AGENTS.md reference: AGENTS.md:L391-L394
Useful? React with 👍 / 👎.
Review found that this change reintroduced a failure shape the repository had already met and already solved. The default workflow records that this exact package installation hung for the full job timeout twice, and it therefore uses an eight-minute step bound, bounded package options, and the on-image package index first. The nightly copy used the unbounded form, so a stalled mirror could consume the whole nightly run before either test lane started. Worse, the guard added with it required the unbounded block. The guard was pinning the known-bad shape in place. Both nightly lanes now use the same bounded installation contract as the default workflow. The guard now requires that contract instead of forbidding it: the step must run before dependencies, stay gated to Linux, keep the eight-minute bound, set noninteractive installation, and match the exact bounded command block. It still fails when the install is absent, merely echoed, exited early, or disabled by condition, and now also when the bound is removed. Each of those five mutations was run and each turned the guard red. The coverage escape also gets its own record. One prevention was carrying both the missing-library incident and the default-CI escape that let earlier guards stay green for ineffective installs. The repository contract requires the runtime failure and the false-green escape to hold separate identifiers with independently enforceable guards, so the false-green entry is now registered in its own section and the removal baseline regenerated. The defect this pull request exists to fix is real and current: the nightly run has failed for four consecutive days with `ImportError: libEGL.so.1`, because the two Qt-dependent jobs have no Linux graphics libraries and whole test directories fail at collection. Written with AI assistance.
|
@codex review this pull request at head |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 483d9bd6dd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| commands = [ | ||
| line.strip() for line in step["run"].splitlines() if line.strip() and not line.lstrip().startswith("#") | ||
| ] | ||
| assert commands == list(_BOUNDED_QT_INSTALL_COMMANDS), ( |
There was a problem hiding this comment.
Fresh evidence beyond the prior early-exit mutation: adding shell: /usr/bin/true {0} to both parsed install steps leaves _assert_nightly_qt_prerequisites green because it validates the condition, timeout, environment, and script text but never the effective shell; /usr/bin/true --help confirms that its arguments are ignored and it exits successfully, so the generated script would not run. On images missing these libraries, default CI would still certify the guard while the nightly jobs recreate the prerequisite failure; bind both the step override and inherited job shell, and add this mutation to the guard test.
AGENTS.md reference: AGENTS.md:L380-L386
Useful? React with 👍 / 👎.
Review found one more way the nightly Qt install could be present and inert: giving both parsed install steps a shell of /usr/bin/true leaves the guard green while nothing is installed. The step exists, its commands are exactly right, and none of them runs. The guard now validates the effective shell of the step, so a substituted interpreter turns it red like any other way of not installing. Written with AI assistance.
|
@codex review this pull request at head |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0dea4b7cb8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| assert step.get("timeout-minutes") == 8, ( | ||
| f"{job_name} install step must retain its eight-minute timeout, found {step.get('timeout-minutes')!r}" | ||
| ) | ||
| assert step.get("env") == {"DEBIAN_FRONTEND": "noninteractive"}, ( | ||
| f"{job_name} install step must set noninteractive package installation, found {step.get('env')!r}" |
There was a problem hiding this comment.
Fresh evidence beyond the effective-shell fix: adding continue-on-error: true to either parsed Qt installation step is accepted by _assert_nightly_qt_prerequisites because the effective step options checked here omit that field. If apt then fails on a runner missing these libraries, GitHub Actions continues into the Qt-dependent lane while the default prevention guard remains green, recreating the nightly failure this record is meant to block; require the field to be absent or false and add this mutation to the guard.
AGENTS.md reference: AGENTS.md:L380-L390
Useful? React with 👍 / 👎.
| - node: tests/test_ci_safe_sqlite_contract.py::test_nightly_qt_guard_rejects_libraries_merely_echoed_instead_of_installed | ||
| ci_partition: remaining | ||
| - node: tests/test_ci_safe_sqlite_contract.py::test_nightly_qt_guard_rejects_step_disabled_with_linux_text_echoed | ||
| ci_partition: remaining |
There was a problem hiding this comment.
The newly added test_nightly_qt_guard_rejects_shell_that_does_not_run_the_install_script is absent from this prevention record's guard list. Consequently, deleting or disabling that test changes neither the registry nor its removal-baseline digest, so default CI can again lose the only mutation proving that an inert shell is rejected without reopening NIGHTLY-QT-PREREQUISITES-055; add the exact node to the machine-readable record and regenerate the baseline.
AGENTS.md reference: AGENTS.md:L387-L390
Useful? React with 👍 / 👎.
#72 and #121 are the same workflow change, and their guards are complementaryMeasured, not inferred: the I opened #121 without reading the open queue first; that was my mistake and #72 came first. But closing either one loses something real, so I am not proposing that yet. The two tests take
Notably, #72's approach is immune to the open P1 on #121 — that the test sources the step while Proposal: converge them into one pull request that keeps both guards — the parsed-workflow Until then neither should merge, because merging one leaves the other's guards unwritten. |
1 participant
What is wrong
The nightly workflow starts the launcher in two jobs. The launcher imports PySide6. PySide6 needs Qt system libraries that the runner image does not carry.
main.ymlinstalls them at line 81. The protected evidence gate installs them too.nightly.ymlinstalled none of them.What it cost
The short mock-stack soak has run on three nights and failed on all three. Its own prerequisite gate passed each time, with
exact_sixexit code 0. Only the launcher role failed to start.log-launcher.txtfrom run 31771940946 is 370 bytes and says:The same import produced 35 collection errors in the golden-run replay job on the same run.
What this changes
This copies the step from
main.ymlinto the two jobs that start the launcher:golden-replayandmock-stack-short-soak. The third job does not start the launcher, so it does not get the step. The step is guarded onrunner.os == 'Linux'.Why it matters beyond the red
The short-soak receipt is a blocking clause of the pre-lab milestone item. The gap is an engineering gap, not owner scheduling, and it needs no time at the hardware stand.
What this does NOT do
This does not change
scripts/soak_mock_stack.py. That driver refuses every profile exceptshort, with exit code 3 and a stated reason, and a guard preserves the refusal. The refusal is deliberate. Long-duration soak evidence remains open and is separate work.This does not claim the soak now passes. No nightly run has happened since the change. The claim here is only that the missing libraries are installed.
Verification
git diff --cached --name-onlyshows no.pyfile.git status --shortshowing only untrackeddocs/refactor/.tests/docs: 67 passed.Prepared with AI assistance. Every measurement above was re-run before it was written.