fix: tell the sealed exact-six child where its own interpreter keeps its libraries - #87
Conversation
The endurance run stops before the program starts, on the laboratory machine, at master. The exact-six stage fails to collect, and the error names the storage layer: src/cryodaq/storage/_sqlite.py:31: import sqlite3 as _stdlib_sqlite3 ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.15' not found (required by <prefix>/lib/python3.14/lib-dynload/../.././libicui18n.so.78) It is not a storage defect. The stage COPIES the interpreter into the sealed snapshot, and a relocated interpreter cannot use run-paths that were relative to where it used to live. The first extension module that needs the C++ runtime then loads the SYSTEM one, and every library loaded afterwards is stuck with it. On this machine that first module is pyzmq: `import zmq` then `import sqlite3` fails, while either import alone succeeds. Naming the interpreter's own library directory removes the ambiguity. The value is derived from the interpreter through sys.base_prefix, never inherited from the caller, so the environment stays closed; a directory that does not exist is not named at all, because a wrong entry on the loader path hides a typo rather than reporting it. Measured end to end in a hand-built replica of the child -- extracted git archive, interpreter copied to the same place, same PYTHONPATH, snapshot files read-only: without the entry ImportError, no tests collected, 1 error with the entry both imports succeed; the same collection reports 8 tests Controls: the three new tests fail against master. The existing test that pins the exact key set still passes, because the parameter defaults to absent, so a caller that does not relocate the interpreter is unchanged.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Confirmed in the real runner on the laboratory machine, not only in the hand-built replica. At this branch's head The run then stops at the theme barrier that #86 removes, so the two branches are sequential and both are needed: with both merged locally the launcher starts, loads its fonts, brings up the ZeroMQ bridge subprocess, and reaches engine construction. Full write-up of the combined run, and of the evidence gap it exposes next, is the 2026-08-19 comment on #86. |
# Conflicts: # docs/architecture-montana-important.svg # docs/current_candidate_metrics.md
|
@codex review Head is |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Measured on the laboratory machine: master itself fails these two tests, and this is the fixUntil now the case for this change rested on the soak reaching all four roles with it in the tree. Here is a direct measurement on the target platform instead — Ubuntu 22.04.5 LTS, glibc 2.35, the miniforge Python 3.14.6 the soak worktrees actually use. At The sealed child reaches the system Two things this measurement settles that a green CI run could not:
Why the approach matters, not only the outcome#82 repairs the same thing the opposite way: it validates the library root and refuses — raising I measured that check on the laboratory machine too, and there every one of its conditions passes (uid 1000, mode 0755, resolves to itself). So it is not wrong about what a safe library root is — it is wrong to answer refusal to a root it does not recognise, and it is on the hosted runner that this bites. The owner's standing rule settles which shape wins:
This branch already has that shape: resolve the directory, set the variable when it exists, do not when it does not, never raise. Noted here and on #82 so the two do not collide silently during a rebase. |
# Conflicts: # docs/architecture-montana-important.svg # docs/current_candidate_metrics.md
|
@codex review Head is |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Why this matters for the laboratory week
The endurance run is the evidence that a week of continuous acquisition holds. On the laboratory machine, at master, it stops before the program starts.
What it looks like, and what it is not
It names the storage layer, and it is not a storage defect. Reproduced at
db5584513291f04d11bd4db3424f2ebbadbe15a3— master, with no branch involved.The mechanism, narrowed by measurement
sqlite3imports cleanly in every one of these on the same machine: the conda interpreter in place; the probe virtual environment; a scrubbed environment; a copied interpreter, with and without the original prefix on the loader path; and under the sealed stage's exact environment dictionary.Then the last variable was reproduced — a hand-built replica of the child: extracted
git archive, interpreter copied to<snapshot>/.venv/bin/python, the samePYTHONPATH, every snapshot file read-only. Direct imports still passed; the stage's ownpytest --collect-onlyfailed. Bisecting the module's imports, one process each, found the trigger:import sqlite3pytest,pytest_asyncio,pytest_timeout,psutil,cryodaq,cryodaq.guicryodaq.core.zmq_bridgezmq,zmq.backend,zmq.backend.cythonSo: the stage copies the interpreter into the snapshot, and a relocated interpreter cannot use run-paths that were relative to where it used to live. The first extension module that needs the C++ runtime loads the system
libstdc++, and every library loaded afterwards is stuck with it. Here that module is pyzmq.libicui18nthen asks the already-loaded system runtime forCXXABI_1.3.15, which Ubuntu 22.04's copy does not have.The change
_controlled_test_environmentaccepts the interpreter's own library directory and names it on the loader path. The value is derived throughsys.base_prefix, never inherited from the caller, so the environment stays closed; a directory that does not exist is not named at all, because a wrong entry hides a typo rather than reporting it. The parameter defaults to absent, so a caller that does not relocate the interpreter is unchanged — the existing test that pins the exact key set still passes untouched.Proof, run rather than argued
In the replica, the only difference being the new entry:
ImportError, no tests collected, 1 errorThree new unit tests fail against master.
tests/scripts/test_soak_mock_stack_runner.py: 44 passed, 14 skipped.ruff check src/ tests/clean; the workflow-exact format check clean over 702 changed files.Repeatable rather than a story:
evidence/tools/sealed_child_replica.sh,evidence/tools/interpreter_copy_probe.sh,evidence/tools/import_order_probe.shandevidence/tools/which_zmq.shin the workspace repository.What this does not claim
It does not claim the endurance run now completes. It removes the barrier that stops it at collection; what the run finds after that is the next measurement. It also does not claim the laboratory environment is healthy — mixing a conda prefix with the system C++ runtime is what made this reachable, and the same mixture deserves its own look before the hardware week.
Written with assistance from Claude (Anthropic).