Skip to content

fix: tell the sealed exact-six child where its own interpreter keeps its libraries - #87

Merged
test1card merged 6 commits into
masterfrom
fix/sealed-child-interpreter-library-path
Aug 20, 2026
Merged

test1card merged 6 commits into
masterfrom
fix/sealed-child-interpreter-library-path

Conversation

@test1card

Copy link
Copy Markdown
Owner

Why this matters for the laboratory week

The endurance run is the evidence that a week of continuous acquisition holds. On the laboratory machine, at master, it stops before the program starts.

What it looks like, and what it is not

phase   runner
reason  exact-six collection execution failed: exit code 4
        src/cryodaq/storage/_sqlite.py:31: import sqlite3 as _stdlib_sqlite3
        E ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.15'
          not found (required by <prefix>/lib/python3.14/lib-dynload/../.././libicui18n.so.78)

It names the storage layer, and it is not a storage defect. Reproduced at db5584513291f04d11bd4db3424f2ebbadbe15a3 — master, with no branch involved.

The mechanism, narrowed by measurement

sqlite3 imports cleanly in every one of these on the same machine: the conda interpreter in place; the probe virtual environment; a scrubbed environment; a copied interpreter, with and without the original prefix on the loader path; and under the sealed stage's exact environment dictionary.

Then the last variable was reproduced — a hand-built replica of the child: extracted git archive, interpreter copied to <snapshot>/.venv/bin/python, the same PYTHONPATH, every snapshot file read-only. Direct imports still passed; the stage's own pytest --collect-only failed. Bisecting the module's imports, one process each, found the trigger:

imported first then import sqlite3
pytest, pytest_asyncio, pytest_timeout, psutil, cryodaq, cryodaq.gui imports
cryodaq.core.zmq_bridge ImportError
zmq, zmq.backend, zmq.backend.cython ImportError

So: the stage copies the interpreter into the snapshot, and a relocated interpreter cannot use run-paths that were relative to where it used to live. The first extension module that needs the C++ runtime loads the system libstdc++, and every library loaded afterwards is stuck with it. Here that module is pyzmq. libicui18n then asks the already-loaded system runtime for CXXABI_1.3.15, which Ubuntu 22.04's copy does not have.

The change

_controlled_test_environment accepts the interpreter's own library directory and names it on the loader path. The value is derived through sys.base_prefix, never inherited from the caller, so the environment stays closed; a directory that does not exist is not named at all, because a wrong entry hides a typo rather than reporting it. The parameter defaults to absent, so a caller that does not relocate the interpreter is unchanged — the existing test that pins the exact key set still passes untouched.

Proof, run rather than argued

In the replica, the only difference being the new entry:

result
without it ImportError, no tests collected, 1 error
with it both imports succeed; the same collection reports 8 tests

Three new unit tests fail against master. tests/scripts/test_soak_mock_stack_runner.py: 44 passed, 14 skipped. ruff check src/ tests/ clean; the workflow-exact format check clean over 702 changed files.

Repeatable rather than a story: evidence/tools/sealed_child_replica.sh, evidence/tools/interpreter_copy_probe.sh, evidence/tools/import_order_probe.sh and evidence/tools/which_zmq.sh in the workspace repository.

What this does not claim

It does not claim the endurance run now completes. It removes the barrier that stops it at collection; what the run finds after that is the next measurement. It also does not claim the laboratory environment is healthy — mixing a conda prefix with the system C++ runtime is what made this reachable, and the same mixture deserves its own look before the hardware week.


Written with assistance from Claude (Anthropic).

The endurance run stops before the program starts, on the laboratory machine,
at master. The exact-six stage fails to collect, and the error names the storage
layer:

  src/cryodaq/storage/_sqlite.py:31: import sqlite3 as _stdlib_sqlite3
  ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.15'
  not found (required by <prefix>/lib/python3.14/lib-dynload/../.././libicui18n.so.78)

It is not a storage defect. The stage COPIES the interpreter into the sealed
snapshot, and a relocated interpreter cannot use run-paths that were relative to
where it used to live. The first extension module that needs the C++ runtime
then loads the SYSTEM one, and every library loaded afterwards is stuck with it.
On this machine that first module is pyzmq: `import zmq` then `import sqlite3`
fails, while either import alone succeeds.

Naming the interpreter's own library directory removes the ambiguity. The value
is derived from the interpreter through sys.base_prefix, never inherited from
the caller, so the environment stays closed; a directory that does not exist is
not named at all, because a wrong entry on the loader path hides a typo rather
than reporting it.

Measured end to end in a hand-built replica of the child -- extracted git
archive, interpreter copied to the same place, same PYTHONPATH, snapshot files
read-only:

  without the entry   ImportError, no tests collected, 1 error
  with the entry      both imports succeed; the same collection reports 8 tests

Controls: the three new tests fail against master. The existing test that pins
the exact key set still passes, because the parameter defaults to absent, so a
caller that does not relocate the interpreter is unchanged.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@test1card

Copy link
Copy Markdown
Owner Author

Confirmed in the real runner on the laboratory machine, not only in the hand-built replica.

At this branch's head aa317df2db, the short soak's exact-six stage writes exact-six-result.json with exit_code 0 and status PASS, where master fails collection with the C++ ABI ImportError. The evidence bundle grows from two files to six.

The run then stops at the theme barrier that #86 removes, so the two branches are sequential and both are needed: with both merged locally the launcher starts, loads its fonts, brings up the ZeroMQ bridge subprocess, and reaches engine construction.

Full write-up of the combined run, and of the evidence gap it exposes next, is the 2026-08-19 comment on #86.

# Conflicts:
#	docs/architecture-montana-important.svg
#	docs/current_candidate_metrics.md
@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Head is e3ab73eaafd3f1a949d60908c2a4c884294a2927. A lane closed the live review findings; the coordinator ran the landing gates on this tree -- byte-order-mark, encoding and parse checks on every changed file, a refusal on any tree that deletes more than it adds, ruff check and ruff format --check on the changed Python, the derived documentation pair regenerated to a fixed point, and the documentation gate green.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@test1card

Copy link
Copy Markdown
Owner Author

Measured on the laboratory machine: master itself fails these two tests, and this is the fix

Until now the case for this change rested on the soak reaching all four roles with it in the tree. Here is a direct measurement on the target platform instead — Ubuntu 22.04.5 LTS, glibc 2.35, the miniforge Python 3.14.6 the soak worktrees actually use.

At origin/master, with nothing from this branch:

probe masterctl at 3823d1cb3f on Ubuntu 22.04.5 LTS, Python 3.14.6
selection: tests/scripts/test_soak_mock_stack_runner.py -k strict_exact_six

E   ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.15' not found
    (required by .../miniforge3/envs/cryodaq-lab/lib/python3.14/lib-dynload/../.././libicui18n.so.78)

FAILED test_controlled_environment_genuinely_collects_strict_exact_six
FAILED test_controlled_environment_genuinely_executes_strict_exact_six
2 failed, 56 deselected in 6.27s

The sealed child reaches the system libstdc++ — jammy's, which does not carry CXXABI_1.3.15 — instead of the one beside its own interpreter. That is precisely what this branch repairs by putting the interpreter's own lib directory on the library path.

Two things this measurement settles that a green CI run could not:

  1. It is not hypothetical and it is not a CI artefact. It reproduces on the machine the laboratory week will run on, at master, with no branch in the tree.
  2. Windows cannot see it. 94 of the soak tests skip here, these two among them, so a green local suite says nothing about it. That is why it was measured on the target directly (evidence/tools/posix_tests_before_push.sh, which runs a selection on the real machine at an unpushed commit).

Why the approach matters, not only the outcome

#82 repairs the same thing the opposite way: it validates the library root and refuses — raising _RunnerActivationDisabled — unless the root is a direct directory, not a symlink, resolving to itself, with ownership it judges safe. Both of its Ubuntu remaining tests die in exactly that refusal on the hosted runner.

I measured that check on the laboratory machine too, and there every one of its conditions passes (uid 1000, mode 0755, resolves to itself). So it is not wrong about what a safe library root is — it is wrong to answer refusal to a root it does not recognise, and it is on the hosted runner that this bites.

The owner's standing rule settles which shape wins:

keep the check, guide instead, record the receipt

This branch already has that shape: resolve the directory, set the variable when it exists, do not when it does not, never raise. Noted here and on #82 so the two do not collide silently during a rebase.

# Conflicts:
#	docs/architecture-montana-important.svg
#	docs/current_candidate_metrics.md
@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Head is 4f2fc4732f57e01aaef01e30ded4d72244697f4d. A lane closed the live review findings; the coordinator ran the landing gates on this tree -- byte-order-mark, encoding and parse checks on every changed file, a refusal on any tree that deletes more than it adds, ruff check and ruff format --check on the changed Python, the derived documentation pair regenerated to a fixed point, and the documentation gate green.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@test1card
test1card merged commit fc553d3 into master Aug 20, 2026
28 checks passed
@test1card
test1card deleted the fix/sealed-child-interpreter-library-path branch August 20, 2026 03:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant