Make a silent point controller detectable (#167) - #175
Merged
Merged
Conversation
Nothing checked that a point controller was still alive. Three correct decisions composed into the gap: a point reading is not retained, it had no re-assert, and a query deliberately arms nothing. So no clock expected to hear from a controller again, and a controller that died quietly left its last confirmedPosition standing while the layout kept setting roads over it. The contract now obliges a 'required' point's controller to re-publish every 30 s, matching sensor/*/reading. evaluateStaleness degrades a point with no reading inside a 90 s window to a seventh confirmation state, 'stale', with confirmedPosition 'unknown'. It latches no PointFault and does not Safe-Stop: silence is a device dying, not lying. It does suspend any route holding it, which is the sensor side's shape exactly - a stale sensor degrades its blocks, latches no SensorFault, and reaches routes through occupancy-unknown. resumeRoute gains no staleness clause on purpose: resuming re-commands every held point, so a recovered controller confirms and a dead one times out into a real fault. The resume is the probe. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #167.
Nothing anywhere checked that a point controller was still alive. A controller that died
quietly left its last
confirmedPositionstanding indefinitely, and the layout kept settingroads over a point whose position nothing had observed since it stopped.
A dead sensor node degrades its blocks to
unknowninside a freshness window (#28). Adead point node degraded nothing.
Why the hole existed
Three decisions, each correct on its own, composed into it: a
point/*/readingis notretained (
point-feedback.mdD1), it had no periodic re-assert, and the confirmation deadlinearms on a command and deliberately not on a query (D6). So no clock anywhere expected
to hear from a point controller again, and silence was indistinguishable from having nothing
new to say.
It is sharper here than on the sensor side, and that is the part worth keeping in view.
Command and feedback are two unrelated devices on two transports: the Cobalt iP motors are
commanded over DCC accessory addresses and have no feedback mechanism of their own, while
position is read from their
S2changeover contacts by a separate MQTT node. There is noshared component whose failure shows up in both, so a fire-and-forget DCC accessory command
keeps succeeding long after the feedback node has stopped. The usual reassurance — that
silence after a command is at least suspicious — does not apply, because the commanded
device is not the reporting device.
The contract changed first
docs/mqtt-contract.mdnow obliges apositionFeedback: 'required'point's controller tore-publish its observed position at least every 30 s, matching
sensor/*/readingexactly.Four shapes were on the table; this one was chosen over a periodic backend query, a
controller-level heartbeat, and accepting the gap knowingly.
The periodic query is the attractive alternative — it keeps the controller dumb and the policy
in one place — and it is refused because it would have had to make an unanswered query into
evidence, when D6 exists precisely to stop a query meaning anything. Reworking what a query
means, on the one path that recovers position at boot, to gain a liveness check that a timer
in the publish loop already provides, is a poor trade. A controller-level heartbeat is cheaper
again and proves less: that the node is running, not that any particular point's sensing path
is intact.
The retention callouts needed rewriting rather than extending, because they justified
non-retention partly on "a point's position is re-asserted by nothing" — which this change
makes false. Non-retention now rests where it always actually rested: re-assertion is
necessary for retention and not sufficient for it. A point's position can change while its
controller is offline, so a retained position is an archived belief with nothing behind it,
and a re-assert says the controller is alive now — a different claim from the archived value
still being true.
Staleness is a degrade, not a fault
evaluateStaleness(pure, indomain/pointConfirmation.ts, applied by the existing 250 mssweep) degrades a point with no reading inside
POINT_FRESHNESS_TIMEOUT_MS— 90 s, 3× theinterval, the same tolerance and reasoning as the sensor window — to a seventh
PointConfirmation,'stale', withconfirmedPosition: 'unknown'.It latches no
PointFaultand does not Safe-Stop. That is #28 D10's split appliedunchanged: a malformed payload is a device lying — immediate, sharp, system-wide; an
unrefreshed reading is a device dying — a freshness window and a scoped degrade.
Three narrowings, each load-bearing:
'required'only. A'none'point has nothing reporting on it and would go staleimmediately and permanently. Every point on Westgate Hollow is
'none'today, so livebehaviour changes by exactly nothing until feedback hardware is fitted.
'confirmed'only — the one state where a reading is actually being trusted.'unreported'is D6's boot case and must stay itself;'pending'belongs to the 8 sdeadline, which fires an order of magnitude sooner;
'mismatch'/'indeterminate'/'timed-out'are already latched faults, and re-labelling one'stale'would replace asharp fact with a vaguer one.
'stale'is its own state, not a reuse of'unreported'. "Never heard from" and"heard, then went quiet" are different facts about the hardware and send an operator to
different places.
The route consequence, and why resume needs no special case
A stale point stops the loco of every route holding it and latches a
'point-not-confirmed'RouteFault— D8's shape minus the point fault. The symmetry is the argument:runSensorTrustSweepalready degrades a stale sensor's blocks, latches noSensorFault, andreaches routes only through
recomputeBlock'soccupancy-unknownpath. Answering the sameevent differently on the other channel would be an inconsistency with nothing behind it.
resumeRoutegains no staleness clause, and that is the interesting decision (D12). Itsprecondition refuses while a held point carries a latched
PointFault— and a stale point hasnone, so a resume proceeds. That is correct rather than a hole:
resumeRoutere-commands everyheld point, which puts each
'required'one back to'pending'and arms the 8 s deadline. Arecovered controller answers and the point confirms; a genuinely dead one does not, times out,
latches a real fault, and the route re-suspends. The resume is the probe, and it settles in
8 s rather than in another freshness window. Adding a clause would have blocked that probe and
stranded the operator until a possibly-healthy device happened to speak again.
Simulator
SimulatedPointControllergained the matching 30 s re-assert. Without it the simulator wouldmanufacture staleness the contract forbids — every
'required'point going'stale'90 safter its last command, which is a simulator defect masquerading as a system one. It re-asserts
through the same
respond()path, so a controller in'silent'mode stays silent, which iswhat lets a scenario move a point from
'confirmed'to'stale'by flipping the mode andletting the clock run.
Tested
npm testfrom the repo root, this session:npm run lintclean.New:
tests/scenario/point-liveness.scenario.test.ts, 7 scenarios — the degrade with nofault latched; a healthy controller re-asserting for ten freshness windows without going
stale; a
'none'point never going stale; the held-route consequence (suspended with locksretained, loco stopped on the wire,
RouteFaultbut noPointFault); recovery on the nextreading with nothing to acknowledge; and both halves of D12's resume probe — recovered
controller confirms, dead controller times out into a real fault. Plus 11 unit tests on
evaluateStalenesscovering the boundary (exactly at the window is fresh), the backwards-clockcase, and each narrowing.
One of them checks on the wire that a re-asserted reading is still published
retain: false—the precise thing this change makes tempting to get wrong.
Docs
Contract amended first, then the record, then the code.
docs/mqtt-contract.md(topic table,both retention callouts, the
readingpayload section, the restart matrix, DegradationTriggers),
docs/point-feedback.md(D11, D12, status, open question 3 now answered in bothdirections),
CLAUDE.md(index line, two new traps),docs/current-state.md,README.md(feature list, Known Limits, Next Milestones).
Unblocks
bazauto/layout-feedback#15— the point position feedback node on board 3, which was heldbecause "publish every N seconds" and "answer a periodic query" are different firmware.
It is the first shape.