fix: put the engine stderr log in the evidence, and show that it holds no diagnosis - #88
Conversation
…s no diagnosis Measured on the laboratory machine, Ubuntu 22.04.5. With the two barriers in front of it removed, the launcher starts, loads its fonts, brings up the ZeroMQ bridge subprocess and reaches engine construction, and then reports CRITICAL Launcher construction failed; phase=engine exception=RuntimeError CRITICAL Launcher retained all construction owners in HOLD after phase engine. That RuntimeError is raised when the engine child exits before its readiness receipt, so the engine died. The evidence bundle held six files and none of them was the engine's own log, so the run reported a CONDITION without its SUBJECT. This publishes that log beside the launcher log, on the success path and on the failure path, and never lets the publication mask the failure it describes: a capture error becomes a note on the primary exception, as the launcher log capture already does. WHAT THE PUBLISHED FILE THEN SHOWED, which is the reason to read this commit rather than only its title. The file arrives, and it carries no diagnosis: engine child stderr record received; phase=runtime (x12) launcher.py:1704 reads each engine stderr line and logs a FIXED string, so the line itself is discarded. The engine's traceback is not kept anywhere. For a week of unattended running that is the gap that matters: an engine that dies at three in the morning leaves a COUNT of stderr lines and no reason. Changing what may be written touches secret handling, so it is a behaviour decision and is raised with the owner rather than made here. Two decisions inside this change. The artifact is ALWAYS written, and says so when the log is absent, because a missing file is indistinguishable from a publisher that silently did nothing. When it exceeds the reviewed ceiling it keeps the END, because a traceback's cause is written last and a bound exists to protect the bundle's size, not to choose which half of the failure survives. The state root parameter is optional, so a caller with no writable root is unchanged. Controls: the three new tests fail with production reverted to master. tests/scripts: 133 passed, 94 skipped.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
# Conflicts: # docs/architecture-montana-important.svg # docs/current_candidate_metrics.md
|
@codex review Head is |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
@codex review Head is |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 participant
The question this answers, and the one it opens
With #87 and #86 in front of it, the endurance run on the laboratory machine gets all the way to the program: the launcher loads its fonts, brings up the ZeroMQ bridge subprocess, and starts the engine. Then:
_wait_engine_readyraises that when the engine child exits before its readiness receipt, so the engine died. The evidence bundle held six files and none of them was the engine's own log — the launcher forwards engine stderr to a rotating log under the writable state root, which for this run is the runner's temporary directory and is deleted with it. The run reported a condition without its subject.What this change does
_launcher_log_capturenow publishes that log besidelog-launcher.txt, on the success path and on the failure path, and never lets the publication mask the failure it describes: a capture error becomes a note on the primary exception, exactly as the launcher log capture already does. Thestate_rootparameter is optional, so a caller with no writable root is unchanged.Two decisions inside it:
What the published file then showed — read this part
The file arrives. It carries no diagnosis:
src/cryodaq/launcher.py:1704reads each engine stderr line and logs a fixed string. The line itself is discarded. So the engine's traceback is not kept anywhere, and this pull request's title would be a false claim if it said otherwise.For a week of unattended running that is the gap that matters. An engine that dies at three in the morning leaves a count of stderr lines and no reason. The operator has nothing to act on and no way to tell one failure from another.
Changing what may be written touches secret handling — the fixed string is presumably there so tokens and chat identifiers cannot reach a log — so it is a behaviour decision and belongs to the owner, not to this pull request. The repository already has redaction machinery (
redact_text,SecretStr), so "redact the line and keep it" is available as a middle path if the owner wants one.Evidence
Measured on Ubuntu 22.04.5, in a worktree cut from the native Linux clone, with all three fixes merged locally and never pushed (
evidence/tools/combine_and_soak.sh). Before: six evidence files. After: seven, includinglog-engine-stderr.txt.Controls: the three new tests fail with production reverted to master.
tests/scripts: 133 passed, 94 skipped.ruff check src/ tests/clean; the workflow-exact format check clean over 702 changed files.Written with assistance from Claude (Anthropic).