Skip to content

fix: deliver the source state to a subscriber that attaches after startup - #116

Closed
test1card wants to merge 12 commits into
masterfrom
fix/source-state-reaches-a-late-subscriber
Closed

test1card wants to merge 12 commits into
masterfrom
fix/source-state-reaches-a-late-subscriber

Conversation

@test1card

Copy link
Copy Markdown
Owner

The Source panel could permanently miss the source's state, and with it the
end-to-end thermal run could never start. This is the last thing that was
stopping the run in phase WAIT_SOURCE_OFF.

What was wrong

SafetyManager.start() records verified-OFF and publishes both source-state
topics. The engine starts SafetyManager before it starts the ZMQ publisher,
so those messages queue and are drained the instant the publisher comes up —
onto a transport that does not retain. A subscriber still completing its
handshake misses that one-shot pair, and nothing republishes it.

Why nobody had seen it

A bug was hiding it. The warm-detector interlock was firing stop_source
against a mock second-stage reading of about 77.5 K, and SafetyManager's stop
path republishes both topics — by which time the GUI bridge was live. So the
panel had been learning the source state as a side effect of a guard firing
wrongly.

Correcting that guard to warn instead of stop, which the owner ruled, removed
the accident and exposed the defect underneath. Bisection between two probe
worktrees that differed only in the interlock configuration named the
configuration change itself as the first bad commit — that change is correct
and stays.
The defect was always here.

What this does not do

The tempting repair is one that reports OFF when it has simply not been told.
This does not do that:

  • it does not restore stop_source for its accidental side effect;
  • it does not let the panel infer OFF from silence;
  • it does not synthesise state in the shell without engine-owned evidence.

A genuinely unknown state stays unknown and the controls stay conservative.
There is a regression for exactly that, because it is the failure mode a
careless fix here would introduce.

Verification

Reversion control, run by the coordinator: with the production change reverted
and the regressions kept, 2 nodes fail; with it applied, 2 pass. The two
nodes are test_late_transport_observer_receives_authoritative_states_without_interlock_trip
and test_late_transport_observer_preserves_genuinely_unknown_source_state. The
first attaches its observer after startup — a subscriber attached before it
does not exercise the defect at all — and asserts no interlock was involved.

End-to-end, measured on a combined worktree carrying this fix plus #110,
#111 and #115, with detector_warmup at action: warning so no guard is
firing:

before this fix:  channel_state="unknown"  cached_topics=[]
after  this fix:  channel_state="off"      cached_topics=['smua','smub']

start_enabled is still false in that tree, gated separately by
"Backend readiness authority". That is #114's subject; #114 is not in this
worktree because it conflicts with #111, and the merge is in progress. This
branch does not by itself make the run pass, and it is not claimed to.

Docs gate green at this head; the derived pair was regenerated as the last
commit.

Not determined

  • The full repository suite. A broad run met
    test_run_engine_registers_safety_tasks_before_installing_startup_backstop,
    which did not settle — and reproduced identically with this repair
    reverted
    , so it is not caused by this change.
  • Physical hardware, and the week-long run itself.

Disclosure

Written by an AI lane (gpt-5.6-sol) under an agent brief; diagnosed by a
separate lane by bisection; verified and landed by the coordinating agent. The
lane could not open a socket in its sandbox and said so rather than guessing —
the end-to-end numbers above are the coordinator's.

soak measurement added 2 commits August 28, 2026 04:55
…rtup

The Source panel could permanently miss the source's state, and then the
instrument could not be driven at all: the end-to-end thermal run timed out
waiting for a confirmed OFF that would never arrive.

SafetyManager records verified-OFF and publishes both source-state topics during
start(). The engine starts SafetyManager BEFORE it starts the ZMQ publisher, so
those messages queue and are drained the moment the publisher comes up, onto a
transport that does not retain. A subscriber still completing its handshake
misses that one-shot pair, and nothing ever republishes it.

This was invisible because a bug was hiding it. The warm-detector interlock was
firing stop_source against a mock second-stage reading of about 77.5 K, and
SafetyManager's stop path republishes both topics — by which time the GUI was
subscribed. So the panel had been learning the source state as a side effect of
a guard firing wrongly. Correcting that guard, which the owner ruled, removed
the accident and exposed this.

Found by bisection between two probe worktrees that differed only in the
interlock configuration; the first bad commit was the configuration change
itself, which is correct and stays.

The repair is in the publication lifecycle, not in the panel and not in the
configuration. A subscriber that attaches at any time can now learn the current
authoritative state. What it must NOT do is manufacture certainty: a genuinely
unknown state stays unknown, and the controls stay conservative. There is a
regression for exactly that, because the tempting fix here is one that reports
OFF when it simply has not been told.
@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Head under review: 34a9ae11ffb3a579b1d1b018d28fe949978904db. Please bind your
verdict to that exact SHA.

This is a publication-lifecycle repair on the critical path, so the useful
review is about what it might have made up rather than what it fixed:

  1. Manufactured certainty is the failure mode. A genuinely unknown source
    state must stay unknown. Please look for any path where this now reports
    OFF because nothing has been said, rather than because the engine asserted
    it. The regression
    test_late_transport_observer_preserves_genuinely_unknown_source_state
    exists for this — check that it would actually catch a violation.
  2. Ordering and races. The defect is that SafetyManager publishes before the
    ZMQ publisher exists, onto a non-retaining transport. Check the repair holds
    when a subscriber attaches during startup, after it, and twice; and that it
    cannot deliver a state that has since been superseded.
  3. Staleness. If a retained value is served to a late subscriber, how does a
    subscriber tell a current assertion from an old one? A wrong answer here is
    worse than no answer, because the panel treats a confirmed OFF as permission
    to act.
  4. It must not depend on any interlock. The whole reason this defect was
    invisible is that a guard firing wrongly used to republish the state. If the
    repair only works while something trips, it has rebuilt the accident.

Stated openly: one broad-suite node
(test_run_engine_registers_safety_tasks_before_installing_startup_backstop)
does not settle — it reproduces identically with this repair reverted, so I do
not believe it is caused here, but I have not root-caused it and am not claiming
a clean full suite.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 34a9ae11ff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cryodaq/core/safety_manager.py Outdated
# subscriber whose handshake completes later. Re-derive this
# bounded-cadence snapshot from the sole safety authority so a
# late observer eventually receives truth, including UNKNOWN.
await self._publish_keithley_channel_states("periodic")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve the age of retained OFF evidence

When a device-reported OFF proof remains cached after connection or emergency-off, this periodic call recreates it with Reading.now() on every monitor tick even though SourceOffEvidence contains no observation time or expiry. A late GUI therefore cannot distinguish an old hardware readback from a current one: keithley_panel.on_reading() ignores off_evidence, apply_state("off") enables Start, and OFF states are exempt from its stale check. Preserve the original evidence time and publish unknown once that proof is stale rather than continually refreshing its timestamp.

AGENTS.md reference: AGENTS.md:L490-L491

Useful? React with 👍 / 👎.

Comment on lines +40 to +44
publisher._queue = queue
publisher._session_id = "0" * 32
publisher._socket = socket # type: ignore[assignment]
publisher._running = True
publisher._task = asyncio.create_task(publisher._publish_loop(queue))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Exercise the real PUB/SUB attachment path

The late-observer regressions replace the ZMQ socket with an attached boolean and invoke _publish_loop() directly, bypassing ZMQPublisher.start(), socket binding, topic subscription, and the PUB/SUB handshake that caused the original loss. These tests can remain green even if a real subscriber attaching after startup never receives the periodic state, so this critical regression needs a real loopback publisher/subscriber test that attaches after publisher startup and repeats the attachment scenario.

AGENTS.md reference: AGENTS.md:L380-L386

Useful? React with 👍 / 👎.

Comment thread src/cryodaq/core/safety_manager.py Outdated
# subscriber whose handshake completes later. Re-derive this
# bounded-cadence snapshot from the sole safety authority so a
# late observer eventually receives truth, including UNKNOWN.
await self._publish_keithley_channel_states("periodic")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep periodic snapshots from disabling adaptive throttling

Every channel-state reading is treated as a transition by AdaptiveThrottle.observe_runtime_signal(), which advances _transition_until by the configured 30-second holdoff. Because this new call emits two such readings every second and adaptive throttling is enabled in config/housekeeping.yaml, the holdoff now remains permanently in the future, so stable temperature and pressure samples are never thinned. This can substantially increase SQLite growth and eventually turn storage pressure into an experiment-stopping persistence failure; periodic retransmissions need to be excluded from transition holdoff updates.

Useful? React with 👍 / 👎.

Comment thread src/cryodaq/core/safety_manager.py Outdated
# subscriber whose handshake completes later. Re-derive this
# bounded-cadence snapshot from the sole safety authority so a
# late observer eventually receives truth, including UNKNOWN.
await self._publish_keithley_channel_states("periodic")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve latched channel faults in periodic snapshots

When _fault() is raised for smua or smub, its immediate publication marks that channel as fault, but the next periodic call supplies no fault_channel; after a successful emergency-off it therefore republishes the same channel as off while the manager is still FAULT_LATCHED. The Keithley panel applies the later state unconditionally, so the channel-specific fault indication disappears about one second after it is raised instead of remaining visible until acknowledgment. Derive periodic channel state from the retained fault latch as well as source activity.

AGENTS.md reference: AGENTS.md:L20-L22

Useful? React with 👍 / 👎.

soak measurement added 2 commits August 28, 2026 07:53
Landed by the batch lander. The lane's own report and the coordinator's
verification are recorded on the pull request.
@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Head under review: f20c42882b4fd5728b01f7b8942ddcb322563939. Please bind your verdict to that exact SHA.

This round answers your four findings and the one red Windows check. The
coordinator verified the lane's claim by reverting the production change and
watching the new assertion redden; this round deletes no assertions.

The property to keep pressing on is the one this change could plausibly break:
a genuinely unknown source state must stay unknown. The tempting repair here
reports OFF because nothing has been said, and that would be worse than the
original defect. test_late_transport_observer_preserves_genuinely_unknown_source_state
is the guard against it — please check it would actually catch a violation
rather than passing for an unrelated reason.

Context: on a tree carrying this branch with the other six in-flight fixes, the
automatic thermal sweep now runs to COMPLETED and both conductance points agree
with the simulator's ground truth to about eight parts in a billion. This branch
is what makes the panel learn the source state at all, so a false OFF here would
propagate into that record.

Docs gate green at this head; derived pair regenerated as the last commit.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: f20c42882b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Head under review: acd81b635117adc53086419a8b822e641c425e67. Please bind your verdict to that exact SHA.

This round fixes a regression this branch introduced: after a hazardous command
succeeded and a SIBLING channel was found unmanaged-on, the per-channel record
reported that channel as fault where master reports off. Three
parametrisations caught it; they pass on master and failed here.

It was resolved as a question about what the record is FOR, not by making the
test green.
The per-channel record drives the operator's ВЫКЛ/ВКЛ/АВАРИЯ badge
and Start/Stop enablement, so it reports PHYSICAL state. Reporting fault there
would show an alarm and disable a channel that is physically off, while telling
him nothing he does not already get from the latched manager fault.

One assertion was changed and I want it looked at rather than waved through:

assert periodic[SMU_CHANNELS[0]].metadata["state"] == "fault"

It guaranteed the per-channel fault badge persisted until acknowledgement. What
now guarantees the incident stays visible: manager.state == FAULT_LATCHED, the
exact reason in manager.fault_reason, the GUI gate reason, and the durable
operator log — with a new test pinning that both channels read off WHILE the
manager remains fault-latched. Please confirm nothing is lost in that move; if
the badge was the only thing an operator would actually see, say so.

Coordinator's measurements: control 4 red without the change, 28 green with it;
tests/core 2204 passed with one failure that also fails on master and is not
this branch's.

Docs gate green at this head; derived pair regenerated as the last commit.

@test1card

Copy link
Copy Markdown
Owner Author

Correction — ignore the SHA in my previous comment.

That comment named acd81b6351. That commit is not on this branch. My push
was rejected as a non-fast-forward and the comment posted anyway, so the review
request pointed at an object that does not exist on the remote. The head here is
still f20c42882.

What happened: the worktree I built the correction in was pinned to an older head
than the one already pushed, so my fix was based behind the branch. That is my
error, not a lane's.

The correction itself is real and verified — the per-channel record reports
physical state (off) rather than fault after a sibling is reconciled, with
the incident kept visible by the latched manager fault, its reason, and the
durable operator log. I will rebase it onto f20c42882 and push properly, then
ask for review again against a SHA that exists.

Please disregard the previous request until then.

soak measurement added 2 commits August 28, 2026 09:13
… fault

After a hazardous command succeeded and a sibling channel was found unmanaged
and on, the system reconciled it -- turned it off -- and latched a fault at the
manager. This branch had also been marking that channel 'fault' in the
per-channel record.

That record drives the operator's badge and the Start/Stop enablement, so it
reports PHYSICAL state. Reporting 'fault' there showed an alarm on a channel
that was physically off and disabled it, while telling the operator nothing the
latched manager fault did not already say.

One assertion changed, which guaranteed the per-channel fault badge persisted
until acknowledgement. What guarantees the incident stays visible now: the
manager stays FAULT_LATCHED, its fault_reason retains the exact incident, the
GUI gate reason shows it, and the durable operator log records it -- with a new
test pinning that both channels read 'off' WHILE the manager remains latched.
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: f20c42882b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Head under review: d6fb6a5616994318e9125ea36ac9255a35587cc1. I have verified
that SHA is the branch head on the remote before writing this — my previous
request named a commit that had failed to push, and this is the corrected one.

This round fixes a regression this branch introduced. After a hazardous command
succeeded and a SIBLING channel was found unmanaged-on, the system reconciled it
and latched a fault at the manager; this branch had also been marking that
channel fault in the PER-CHANNEL record, where master reports off. Three
parametrisations caught it — they pass on master and failed here.

Resolved as a question about what the record is FOR, not by making the test
pass.
That record drives the operator's badge and Start/Stop enablement, so it
reports PHYSICAL state. fault there showed an alarm on a physically-off channel
and disabled it, while adding nothing the latched manager fault did not say.

One assertion changed, and it deserves your scrutiny rather than a nod:

assert periodic[SMU_CHANNELS[0]].metadata["state"] == "fault"

It guaranteed the per-channel fault badge persisted until acknowledgement. What
now guarantees the incident remains visible: the manager stays FAULT_LATCHED,
fault_reason retains the exact incident, the GUI gate reason shows it, and the
durable operator log records it — plus a new test pinning that both channels read
off WHILE the manager is latched. If the badge was the only thing the
operator would realistically see, that trade is wrong and I want to hear so.

Coordinator's measurements: control 4 red without the change, 28 green with it;
tests/core 2204 passed, the single remaining failure also failing on master.

Docs gate green at this head; derived pair regenerated as the last commit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d6fb6a5616

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cryodaq/core/safety_manager.py Outdated
# subscriber whose handshake completes later. Re-derive this
# bounded-cadence snapshot from the sole safety authority so a
# late observer eventually receives truth, including UNKNOWN.
await self._publish_keithley_channel_states("periodic")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the documented channel-state semantics

This periodic overwrite intentionally makes the per-channel badge reflect physical off/unknown while the manager retains the fault, but the canonical component contract still describes a channel fault as an АВАРИЯ badge that disables that channel (docs/design-system/cryodaq-primitives/keithley-panel.md:197-205), and the operator manual does not explain the new physical-state/manager-latch split (docs/operator_manual.md:187-207). Update those specifications and their governed version/changelog evidence in this slice; otherwise future GUI work and tests are directed toward the superseded persistent-fault presentation.

AGENTS.md reference: AGENTS.md:L537-L539

Useful? React with 👍 / 👎.

@test1card

Copy link
Copy Markdown
Owner Author

Correction to my earlier comments on this pull request.

I said that tests/core/test_engine_b3_structure.py::test_engine_wiring_submodules_import_without_engine_reverse_cycle
"also fails on master" and implied master carries a red test. That was wrong.
Master is fine.

The failure was in MY environment. That test spawns a subprocess to prove there
is no import cycle, and the subprocess inherited my shell's library search path
rather than the conda environment's — loading a system libstdc++ older than
the environment's ICU library requires:

ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.15' not found

With LD_LIBRARY_PATH pointed at the environment's lib directory it passes.
Hosted CI never saw it because its interpreter is the system one.

The operational advice was still right — that node is not this branch's problem
and there is nothing here to chase. But the reason I gave for it was not, and it
was a claim about the trunk, so it is worth correcting on the record rather than
leaving to stand.

soak measurement added 2 commits August 28, 2026 11:03
Landed by the batch lander. The lane's own report and the coordinator's
verification are recorded on the pull request.
@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Head under review: b443d6df3e4d2da6e7db71ea2cd2fdbe2370287f (verified equal to
origin/fix/source-state-reaches-a-late-subscriber at the moment of writing).

Reversion control, run this turn on Ubuntu 22.04:

PHASE 1  new guard only, contract text reverted -> 1 failed, 77 passed
  test_keithley_channel_state_contract_separates_physical_badge_from_manager_latch
PHASE 2  with the contract text                 -> 78 passed

Documentation gate: tests/docs 69 passed.

A correction to my earlier brief on this pull request. I previously said the per-channel
snapshot contract on master reads {"smua": "off", "smub": "fault"} on one side and off on
the other, and asked this branch to settle which was right. Measured this turn:
tests/core/test_safety_operator_snapshot_owner.py is byte-identical between master and
this head
(blob 4cb3ebe8326335c2485b74851779fadf7605d5c5), and master itself asserts
{"smua": "off", "smub": "fault"}. No safety assertion was changed by this branch or any
other lane.
What this round adds is documentation binding that contract, not a change to it.

What the round asserts, and what to attack. It writes down that per-channel source state and
manager safety state are two separate truths: the channel badge describes the physical output,
safety_state=fault_latched describes the manager, and a channel returning to a normal badge
does not mean the Safety fault was acknowledged.

The operator-facing risk is the one worth your attention: if a channel is physically off but
its badge reads fault, does anything downstream refuse an action, or show the operator a
source as energised when it is not? And conversely — if the badge reads off, is the fact that
it was once found unmanaged-and-on still visible to him anywhere, or does it survive only
inside the manager's fault reason where he will not look?

The branch's own property must still hold: a subscriber attaching after startup receives the
authoritative source states, and a genuinely unknown state stays unknown rather than being
reported as OFF.

Disclosure: this change was produced and verified with AI assistance.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b443d6df3e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cryodaq/core/safety_manager.py Outdated
# subscriber whose handshake completes later. Re-derive this
# bounded-cadence snapshot from the sole safety authority so a
# late observer eventually receives truth, including UNKNOWN.
await self._publish_keithley_channel_states("periodic")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Let the transition fault survive a GUI drain

When _run_checks() detects a fault, _fault() publishes the channel's fault reading before returning, but this next statement immediately publishes the periodic off/unknown replacement in the same monitor iteration. The GUI drains queued readings every 10 ms and applies them sequentially, so both usually arrive in one drain and the final physical badge is rendered without the documented channel-specific АВАРИЯ cue ever becoming visible. Skip this periodic publication after a transition or retain the fault state for at least one observable cadence.

AGENTS.md reference: AGENTS.md:L529-L535

Useful? React with 👍 / 👎.

Comment thread docs/operator_manual.md

Backend-канал `analytics/keithley_channel_state/{smua,smub}` описывает физическое состояние выхода, а не долговременную защёлку аварии менеджера. `АВАРИЯ` появляется как немедленное событие перехода для затронутого канала. Следующий периодический snapshot публикует `ВЫКЛ`, если есть актуальное подтверждение OFF, или `НЕИЗВЕСТНО`, если такого подтверждения нет.

Эта смена бейджа не означает, что авария Safety квитирована. Отдельный `analytics/safety_state=fault_latched` и его reason сохраняют инцидент, блокируют обычное управление через Safety gating и остаются авторитетными до разрешённого восстановления. Поэтому `ВЫКЛ` нельзя трактовать как отсутствие latched fault, а `НЕИЗВЕСТНО` — как разрешение повторить команду; при живой связи доступным остаётся только аварийное отключение.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Replay the latched fault reason to late subscribers

When a GUI subscribes after the fault transition, this documented claim does not hold: analytics/safety_state and its exact _fault_reason are published only on transitions, while the new periodic path republishes only channel states. The periodic typed operator snapshot exposes a generic “Safety fault recovery is incomplete” blocker rather than the incident reason, so the source panel can show ВЫКЛ plus a generic gate without revealing that the channel was found unmanaged and on. Republish the latched manager state and reason for late observers, or include that reason in the retained typed authority.

AGENTS.md reference: AGENTS.md:L530-L535

Useful? React with 👍 / 👎.

Comment thread src/cryodaq/core/safety_manager.py Outdated
"reason": reason,
"off_evidence": self._reviewed_source_off_evidence.receipt_payload(),
"off_evidence": published_evidence.receipt_payload(),
"is_transition": reason != "periodic",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep retransmissions from clearing unknown command outcomes

When any normal Keithley command returns an unknown outcome, _SmuChannelBlock requires one later source observation and one later Safety observation before clearing its persistent warning. Every packet produced here increments the source observation revision even though is_transition=False, and the independent 1 Hz operator snapshot increments the Safety revision, so the next cadence deterministically declares the outcome reconciled without evidence that the command settled; this is especially invalid for target/limit commands because the repeated on state is identical whether the mutation succeeded or failed. Publish an underlying owner revision or observation time that consumers can compare with the command, rather than allowing retransmitted state to authorize reconciliation.

AGENTS.md reference: AGENTS.md:L530-L535

Useful? React with 👍 / 👎.

soak measurement added 2 commits August 28, 2026 13:44
Landed by the batch lander. The lane's own report and the coordinator's
verification are recorded on the pull request.
@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Head under review: 549704a8a780c48682f267e5b289fd963b516c80 (verified equal to
origin/fix/source-state-reaches-a-late-subscriber at the moment of writing).

All three findings from your review of b443d6df3 are closed, each with a guard that
reddens when the fix is removed.
Measured this turn on Ubuntu 22.04, production reverted with
the new tests kept:

production reverted -> 3 failed, 82 passed
  test_monitor_fault_cue_survives_the_iteration_that_detected_it
  test_late_transport_observer_receives_latched_safety_reason
  test_periodic_source_retransmission_cannot_reconcile_unknown_target_outcome
production restored -> 85 passed

One guard per finding, in the order you raised them:

  1. The alarm cue now survives the iteration that detected it. Previously _fault()
    published the channel's fault reading and the very next statement published the periodic
    replacement in the same monitor iteration, so both reached the interface in one 10 ms drain
    and the АВАРИЯ cue was never rendered. The operator was never shown the alarm at all.
  2. A late transport observer now receives the latched safety reason, not a generic
    "recovery incomplete" blocker. This was the exact question I asked you to check when I
    requested the review: whether the fact that a channel was found unmanaged and on survives to
    somewhere the operator will actually look. It did not; now it does.
  3. A periodic retransmission can no longer reconcile an unknown command outcome. The
    warning existed because the software did not know whether the command took effect; letting
    the next cadence clear it meant the warning was cleared by elapsed time rather than by
    evidence, and worst for target and limit commands where the repeated on state is identical
    whether the mutation succeeded or failed.

Suites, measured this turn:

tests/core                                        -> 2207 passed, 0 failed
module gate: every test file naming `safety_manager`
  or `keithley_panel` -- 60 files                 -> 1512 passed, 3 skipped, 0 failed
documentation gate                                -> 69 passed
ruff check / ruff format --check                  -> clean on all four changed files

No existing test was weakened, skipped or restated; the three new guards are additions.

A note on the earlier verdicts. Two clean reviews on this pull request name f20c42882b.
That is not the head and has not been for some time, so I have not treated either as a verdict
for this branch. This request is bound to 549704a8a.

What I would still like attacked: whether any path can now show the operator a stale latched
reason after the condition has genuinely cleared. Making the reason survive for late observers
is only correct while it is still true; a reason that outlives its fault is the same
misinformation pointed the other way.

Disclosure: this change was produced and verified with AI assistance.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 549704a8a7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

# manager state (including its exact latched reason) and
# re-derive channel state from the sole safety authority so a
# late observer eventually receives truth, including UNKNOWN.
await self._publish_state("periodic")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Exclude periodic safety snapshots from throttle holdoff

In common long-lived non-running states such as SAFE_OFF, READY, and FAULT_LATCHED, this now emits analytics/safety_state every second, while AdaptiveThrottle.observe_runtime_signal() treats every non-running safety packet as a transition and advances its 30-second holdoff. Stable housekeeping samples therefore remain unthrottled indefinitely, increasing archive growth and potentially reaching the persistence-first failure boundary. The fresh evidence beyond the earlier finding is this newly added safety-state retransmission; the existing guard covers only periodic channel-state packets marked is_transition=False.

AGENTS.md reference: AGENTS.md:L478-L481

Useful? React with 👍 / 👎.

Comment on lines +784 to +786
self._source_observation_revision = max(
self._source_observation_revision,
source_observation_revision,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Scope source revisions to an Engine generation

When the GUI survives an Engine/bridge restart after a command acquires an unknown outcome, the replacement SafetyManager restarts these revisions at 0, but the channel block retains the prior producer's value and only accepts max(old, new). The latch requires old + 1, so a new producer's initial/current observations cannot reconcile it; because that latch itself blocks start, stop, target, and limit commands, it can remain stranded until the GUI is restarted or enough unrelated physical state changes occur. Bind the revision to a producer/connection generation rather than comparing counters from different Engine incarnations.

AGENTS.md reference: AGENTS.md:L468-L469

Useful? React with 👍 / 👎.

Comment thread src/cryodaq/core/safety_manager.py Outdated
Comment on lines +2977 to +2979
if is_transition and state != self._source_observed_states[smu_channel]:
self._source_observation_revisions[smu_channel] += 1
self._source_observed_states[smu_channel] = state

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Advance the revision for fresh same-state OFF proof

If a keithley_start command has an unknown outcome because it never reached the manager, the authoritative state remains off and the GUI waits for source_observation_revision + 1. A subsequent emergency_off supplies fresh, decisive OFF evidence but publishes the same off state, so this state-change-only condition leaves the revision unchanged; the Safety half can advance while the source half remains permanently unsatisfied, stranding all normal controls even after the operator used the recovery path. Advance the revision for a genuine new owner observation such as an OFF proof while continuing to exclude mere periodic retransmissions.

Useful? React with 👍 / 👎.

Comment thread src/cryodaq/core/safety_manager.py Outdated
async def _publish_state(self, reason: str = "") -> None:
if self._data_broker is None:
return
published_reason = self._fault_reason if self._state is SafetyState.FAULT_LATCHED else reason

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve an operator-facing reason during manual recovery

After a fault is acknowledged but recovery preconditions are not yet restored, the state is MANUAL_RECOVERY, so every new periodic call selects its literal argument "periodic" instead of a meaningful retained recovery blocker. MainWindowV2._dispatch_safety_evidence() accepts negative legacy states even after typed authority is present and uses this value as the Keithley gate reason, causing the panel to repeatedly show Управление заблокировано: periodic until another typed snapshot happens to replace it. This is fresh evidence outside the corrected FAULT_LATCHED branch: preserve an actionable operator reason for all non-ready periodic states.

AGENTS.md reference: AGENTS.md:L529-L535

Useful? React with 👍 / 👎.

@test1card

Copy link
Copy Markdown
Owner Author

Acknowledged, and this round is not ready. Four findings at 549704a8a, and two of them
strand the operator's controls — which is the failure this branch family exists to prevent.

I asked you to attack the fix rather than take the node names for it. You did, and what you
found is that closing three findings opened four. Recording that plainly rather than arguing
with it.

The archive-growth one is an acceptance criterion, not a nicety. analytics/safety_state
now goes out every second in SAFE_OFF, READY and FAULT_LATCHED, and
AdaptiveThrottle.observe_runtime_signal() counts every non-running packet as a transition,
so its 30-second holdoff never engages and stable housekeeping is never throttled. This
instrument must run for a WEEK without unbounded growth; "increasing archive growth and
potentially reaching the persistence-first failure boundary" is exactly the mode that ends a
week-long measurement. My change to make the latched reason reach a late subscriber created
this. It has to be a retransmission that the throttle can recognise as not a transition.

Two findings leave controls stranded, and that is the owner's standing ruling. His words are
that nothing may disable the Start control — a source may be turned off, but the operator must
never lose the ability to act, because when he does he drives the hardware by hand and nothing
is recorded at all.

  • After an Engine or bridge restart the replacement SafetyManager restarts its revisions at
    0, the channel block keeps the prior producer's value under max(old, new), and the latch
    wants old + 1 — so it can never be satisfied. That latch blocks start, stop, target and
    limit. Over a week the Engine WILL restart; this is not an edge case.
  • An emergency_off is the operator's recovery path, and it supplies genuine fresh OFF
    evidence — but it publishes the same off state, so a state-change-only condition leaves the
    revision unchanged and the controls stay stranded after he has already used the recovery
    path
    . That is the worst shape: the escape hatch does not release him.

The fourth is operator-facing text that says nothing. Управление заблокировано: periodic
is not a reason; "periodic" is the literal argument leaking into the gate reason during
MANUAL_RECOVERY. He should be told what is actually blocking him, and if nothing is, he
should not see a blocker at all.

A lane will take all four. The three guards from the previous round stay and must keep
passing — the fault cue surviving its iteration, the late observer getting the latched reason,
and a retransmission not reconciling an unknown outcome. The fix for the first of those is what
introduced the throttle defect, so it needs to be rebuilt rather than reverted.

I will not merge this on nine-of-nine green. Green is what it looked like before you found
these.

soak measurement added 2 commits August 28, 2026 15:37
Landed by the batch lander. The lane's own report and the coordinator's
verification are recorded on the pull request.
@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Head under review: 045669cdd40bb86015ed2cec1d283fba8cc1496c (verified equal to
origin/fix/source-state-reaches-a-late-subscriber at the moment of writing).

All four findings from your review of 549704a8a are closed, each with a guard that reddens
when its fix is removed.
Reversion control, Ubuntu 22.04, production reverted with the new
tests kept:

production reverted -> 5 failed, 84 passed
  test_late_transport_observer_receives_latched_safety_reason
  test_periodic_safety_retransmission_allows_stable_archive_throttle_to_engage
  test_periodic_manual_recovery_state_publishes_current_blocker_reason
  test_restarted_engine_first_source_observation_reconciles_prior_generation
  test_emergency_off_same_state_observation_reconciles_unknown_start_outcome
production restored -> 89 passed

module gate: every test file naming safety_manager, keithley_panel or housekeeping
  -- 67 files -> 1674 passed, 4 skipped, 0 failed
documentation gate -> 69 passed

Taking your four in order:

  1. Archive growth. The throttle can now tell a retransmission from a transition, so stable
    housekeeping in SAFE_OFF, READY and FAULT_LATCHED no longer defeats the holdoff. This
    was the one I cared about most: the instrument must run for a WEEK, and a packet per second
    that the throttle cannot recognise as routine is exactly how the archive reaches the
    persistence boundary before the measurement finishes. The guard is named for the throttle
    engaging, not for a flag being set.
  2. Controls stranded after an Engine restart. The first source observation from a restarted
    producer now reconciles the prior generation, instead of a max(old, new) revision that a
    latch wanting old + 1 could never satisfy.
  3. The operator's own recovery path. An emergency_off that republishes the same off
    state now reconciles an unknown start outcome. That was the worst of the four: he used the
    escape hatch, it supplied genuine fresh evidence, and he stayed stuck.
  4. The blocker text. MANUAL_RECOVERY publishes the current blocker reason rather than
    leaking the literal "periodic" onto his screen.

Findings 2 and 3 both ended with start, stop, target and limit blocked. The owner's standing
ruling is that nothing may take his controls away — a source may be turned off, but he must
never lose the ability to act.

What I would like you to attack now. Making the latched reason survive for late observers is
only correct while it is still true. If any path can show a stale latched reason after the
condition has genuinely cleared, that is the same misinformation pointed the other way. And on
the throttle: if a real transition can now be mistaken for a retransmission, a change the
operator needed to see would be dropped — that failure is worse than the growth it fixes.

Disclosure: this change was produced and verified with AI assistance.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 045669cdd4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

publish_task = asyncio.create_task(
self._publish_keithley_channel_states(
"emergency_off",
observed_channels=channels,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Retain targeted OFF proof for the observed channel

When cached global OFF evidence is older than stale_timeout_s and the operator successfully runs emergency_off(channel="smua"), _ensure_output_off() confirms that channel but does not refresh the global evidence cache; passing observed_channels only advances its revision. The publisher therefore immediately derives smua from the stale cache and emits unknown, leaving Start, Stop, target, and limit commands unavailable despite the fresh OFF proof. Fresh evidence beyond the earlier emergency-OFF finding is the expired-cache path; retain per-channel evidence and observation time so the exact confirmed channel publishes off.

AGENTS.md reference: AGENTS.md:L20-L22

Useful? React with 👍 / 👎.

Comment on lines +903 to +906
source_is_fresh = self._source_observation_generation > source_generation or (
self._source_observation_generation == source_generation
and self._source_observation_revision >= source_required
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Bind reconciliation to an observation after the command

If an owner transition is dropped before reaching the GUI—ordinary DataBroker telemetry explicitly permits queue drops at core/broker.py:412-432—the producer revision can already be N+1 while the panel still knows N. After a command gets an unknown outcome, the next periodic retransmission carries that pre-command N+1, and this comparison clears the latch once Safety advances even though no source observation occurred after the command; for target/limit mutations, the repeated on state cannot establish whether the mutation settled. Fresh evidence beyond the prior retransmission finding is the replay of a missed pre-command owner revision, so reconciliation must bind to command-time producer evidence rather than last delivered + 1.

AGENTS.md reference: AGENTS.md:L20-L22

Useful? React with 👍 / 👎.

Comment on lines +2936 to +2940
elif self._state is SafetyState.MANUAL_RECOVERY and not is_transition:
recovery_ready, recovery_blocker = self._check_preconditions()
published_reason = "" if recovery_ready else recovery_blocker
else:
published_reason = reason

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Replace the SAFE_OFF periodic placeholder with a real reason

During startup or after an emergency OFF, the manager can remain in SAFE_OFF, but periodic publication falls through here and emits the literal reason "periodic". MainWindowV2._dispatch_safety_evidence() treats every state except ready as negative and forwards that reason to set_safety_ready(False, reason), so the operator repeatedly sees Управление заблокировано: periodic. Fresh evidence beyond the corrected MANUAL_RECOVERY case is this common SAFE_OFF path; publish its current blocker or retained actionable transition reason instead.

AGENTS.md reference: AGENTS.md:L529-L535

Useful? React with 👍 / 👎.

if self._data_broker is None:
return

is_transition = reason != "periodic"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Classify OFF-evidence expiry as a transition

When retained OFF evidence crosses stale_timeout_s, a periodic call changes the published channel state from off to unknown, but is_transition remains false solely because the call reason is "periodic". AdaptiveThrottle.observe_runtime_signal() consequently ignores this real loss-of-authority transition and continues thinning configured temperature and pressure channels instead of retaining the 30-second high-resolution transition window. Determine the throttle transition flag from the derived state change, while keeping mere same-state retransmissions false.

Useful? React with 👍 / 👎.

test1card added a commit that referenced this pull request Aug 28, 2026
…he Start control

Integrates #114 onto the candidate, and fixes one defect that only appears once #114 and #116
are in the same tree.

The severing: main_window_v2 now distinguishes an authoritative blocked state from an
unavailable one. A current interlock condition warns and may stop a running source; it does not
grey out Start. Missing or stale Safety authority still disables ordinary controls, and
emergency OFF stays available throughout. AUTHORITATIVE_NOT_READY and AUTHORITY_UNAVAILABLE stay
distinct -- collapsing them to one boolean is what took the operator's button away.

Conflicts were resolved per file rather than per side, because the two branches do not contest
the same behaviour -- they have different bases. safety_manager and keithley_panel keep this
candidate's A2 content (evidence staleness, the producer revision, replay-is-not-observation);
main_window_v2 and its wiring tests take #114's severing. Taking either side wholesale would
have silently reverted the other.

THE INTEGRATION DEFECT, found by #114's test once both were present: the panel read the
producer revision with `reading.metadata.get("source_observation_revision")`, which returns None
when a producer stamps no revision. #116's stricter handling accepts the sentinel or an int, so
None fell through both branches, the revision never advanced, and a genuine new observation
could never reconcile an unknown command outcome -- leaving the operator's controls latched
indefinitely. Absence now means unspecified, not invalid. Neither pull request could have found
this alone: #116's tests always stamp a revision and #114's path does not.

Verified on the candidate: 298 passed across the wiring, channel-state, panel, safety-manager,
driver and sensor suites -- all three blocks' properties holding at once.

Block A3. Semantic delta only; documentation and generated artifacts reconcile at freeze.
@test1card

Copy link
Copy Markdown
Owner Author

Closing as superseded by master, with the check that establishes it.

Master is 9cd72a91b7efc9d18ba6eacf9c4cbc70d6040d69. Merging master into this pull request's head produces a tree that is byte-identical to master's own tree:

git rev-parse 'origin/master^{tree}'
dd50a9a3d0f09b933176ace91d51df46ca7a240e

merge of master into this head, resulting tree
dd50a9a3d0f09b933176ace91d51df46ca7a240e

Identical trees mean this branch contributes no production, test, governance or documentation change that master does not already have. The behaviour this pull request asked for is in master; the branch is not.

Reopen this if the tree comparison above is wrong. Nothing here is a judgement about the fix itself, which landed.

@test1card test1card closed this Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant