Repository navigation
20260916 - Adopt node ingest v1.2.2, and register a node that does not know where it is - #18
Merged
Merged
Conversation
The contract moved three times since we adopted 1.1.1, and this takes all of
it. Behaviour is unchanged: what moves here is the type of things, not what
the node does with them.
What the revisions carry:
1.2.0 the six NodeConfig coordinates became required-and-nullable, so a
node whose owner cannot supply the geometry can still build a
payload. PUT /v1/nodes/contact appeared. NodeState gained no values
but lost its caveat: `stalled` is confirmed, and the one-line revert
we kept ready is retired. The error taxonomy split invalid_request
into invalid_config, invalid_contact and invalid_body, and every
operation gained a 413, an x-retry, an x-terminal and a body cap.
1.2.2 tx_callsign became nullable too, which is what makes the 1.2.0
change reach the case it was made for: a tower's name and its
position are set at the same wizard step, so a node with no position
has no name for one either. The contact document became a named
schema and gained a phone country.
Fourteen fields are now required-and-nullable rather than seven. The canary in
tests/wire/test_serialise.py failed on adoption, which is what it is for, and
the inventory it pins is updated by hand so the growth is a visible edit.
The generator needed help. 1.2.0 onwards is exported from the server's own
FastAPI models, which spell nullable as `anyOf: [X, {type: null}]` where the
hand-written 1.1.1 used `type: [x, "null"]`. The two mean the same thing and
datamodel-codegen treats them very differently: given anyOf it cannot attach a
branch's constraints to a union member, so it manufactures a RootModel wrapper
per field. --collapse-root-models does not reach those, because they are
synthesised from an inline branch rather than declared as a $ref. Adopting
1.2.0 raised the count from 3 wrappers to 19, which is RxLat(root=51.4) at
construction and .root at every read, spreading a generation artefact through
code that should only ever see floats.
So tools/normalise_spec.py rewrites that one idiom on a temporary copy, and
generate-models.sh reads the copy. docs/node-ingest-v1.yml stays byte-identical
to what the server author published, which is the point of keeping it
read-only. A nullable $ref has no type to hoist and is left alone; so is
anything that is not exactly a two-member union with null.
Two renames fall out of the export inlining its named scalar schemas: the
NodeState enum is now generated as State, and Adsb is a plain Enum rather than
a StrEnum now that it is nullable, so it can no longer be compared to a string.
The second is the trap tests/wire/test_heartbeat.py already documented for
Blah2; both are now the same shape and the asymmetry is gone.
Checked NodeConfig.beam_width_deg on adoption, per the working agreement. It
survived, nullable as agreed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
retina-node has shipped location.* null since 3fd4208, so a node between first boot and its owner reaching the tower step has no geometry at all. We tolerated that already: `_optional` reads the seven keys and `is_located` says whether the node knows where it is. What we did with the answer was hold registration, because the wire could not carry a null geometry and asserting a position nobody chose is worse than silence. The consequence was that the fleet had no sight of a node nobody had configured. It could not be counted, its health was unknown, and the only evidence it existed at all was a status document on the node itself. Spec v1.2.0 made the six coordinates required-and-nullable for exactly this, and says what the server does with them: such a node "registers and streams, and its detections are counted, but it places nothing on the map until a position arrives". v1.2.2 did the same for tx_callsign. So the gate goes, and an unsited node now registers with seven explicit nulls. `is_located` stays and gates nothing. It feeds the status document, and that is worth keeping precisely because an unsited node now looks entirely healthy from outside: registered, streaming, heartbeating, contributing nothing. That sentence in `detail` is the only thing telling an operator why nothing of theirs appears on the map, so the wording changed with the behaviour. Nothing is substituted. `m_to_ft` passes None through rather than converting an absent altitude, because 0.0 ft is sea level and a real place to claim to be. Between v1.2.0 and v1.2.2 this branch briefly sent "unknown" for the callsign, since that field was left non-null while the coordinates went nullable; the server author made it nullable instead and the placeholder is gone. Stage 2 now substitutes nothing at all. The mock gains the three rules the contract states and a JSON schema cannot: the six coordinates accept null, a latitude and its longitude are given together or both null, and the degenerate-baseline check only runs once both ends have a position. That last one would otherwise subtract a null and turn the ordinary state of a new node into a 500. `tx_callsign` accepts null but still refuses the empty string, which the schema is explicit about: a node that cannot name its illuminator has exactly one way to say so. Also removes a duplicate `_optional` definition left by 9ed838d. The two bodies were identical so nothing was wrong, and neither ruff nor vulture flags it: F811 does not fire because the name is used from a deferred function body. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
live-service.sh has been a no-op past its first gate since 1f1b990 landed on 2026-08-26. That commit made registration wait on the setup wizard flag at /data/retina-gui/setup-wizard-completed; this script mounts only /data/mender and the scratch directory, and never set WIZARD_FLAG_PATH. So the container could not see the flag whatever the node held, every run since has stopped at `setup_incomplete`, and the end-to-end verification this repo leans on has been quietly verifying nothing for three weeks. Found by running it. It mounts /data/retina-gui read-only now and points the env var at the node's own flag, so the gate is exercised rather than bypassed. Its consent handling was stale in the other direction. The script synthesised three records on the grounds that "the retina-gui wizard persists none", which stopped being true at retina-gui v0.7.0; owl-ded9 has held all three since 2026-08-16. It now uses the node's own file when there is one and says which it used, falling back to synthetic only when there is not. Nothing worth trusting comes from a run that quietly replaced the thing under test. UNSITE=1 nulls the geometry in the scratch copy so the unsited path is reachable on a node that has a tower. The node's own config.yml is never written to; this script still touches nothing outside /tmp on the node, and live-stalled.sh remains the only one that writes to it at all. summarise_run.py printed rx_alt_ft and nothing else of the geometry, which was reasonable when the geometry was always present and is not now that seven nulls are the interesting case. It prints both positions and the callsign. Verified against owl-ded9: unsited, registered, config accepted, 3 heartbeats and 30 frames at 0.9 Hz with nothing dropped, seven nulls on the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A refused registration went into `errors[]` and nowhere else. Nothing reads
that list: retina-gui renders `detail` verbatim for any state outside
{registering, awaiting_config, starting}, and ignores the rest. `unregistered`
is not one of those three, so the channel was open the whole time and we were
writing null into it.
owl-ded9 is what this costs. It has been refused 232 times over nine days with
a 400 naming `tx_lat`, which is the actionable kind: its receiver and
illuminator are at the same point, because the tower's coordinates were typed
into the receiver fields at the wizard's location step. Everything needed to
say so was in hand. Its owner saw a blank line.
Three refusals, three sentences, because they point at different people:
400 names the field and says where to correct it. It cannot clear
without a local edit, so there is nothing to wait for.
403 / 429 deliberately opaque, and the *normal* answer while Mender
acceptance propagates.
5XX / no the network rather than the node. Sending an owner to check a
answer configuration that is fine wastes the one thing they will do.
The 403 is shown from the first refusal rather than after a threshold. A delay
would keep a newly flashed node quiet, but it would also leave a genuinely
stuck one looking healthy for as long as the threshold lasts, and the count of
attempts is not the thing an operator needs. The reassurance goes in the
wording instead: that sentence leads with "this is normal on a newly flashed
node and clears on its own once Mender has accepted it".
`unregistered` gains a baseline sentence too. It was one of three states with
none, so the window before the first attempt was blank as well.
Placed above the unsited line in the precedence order: a node the server will
not accept has a more urgent thing to say than one it would accept but cannot
place. Cleared on success, because a stale refusal on a working node is worse
than none.
The retry loop is deliberately unchanged. A 400 keeps retrying at the 30 minute
ceiling and the payload is rebuilt from the files each pass, so correcting the
configuration recovers with no restart. That spends registration allowance on a
node that is already broken, which is worth less than recovering unattended.
Verified against owl-ded9's own configuration through the tunnelled mock: the
real 400, and `detail` naming tx_lat.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit fixed live-service.sh and stopped there. The same bug is in live-failures.sh, live-stress.sh and live-stalled.sh: none of them set WIZARD_FLAG_PATH or mount /data/retina-gui, so since 1f1b990 landed on 2026-08-26 every one of them has held at `setup_incomplete` and tested nothing past the first gate. Found by going to run them. Four scripts, one bug, and it was worth checking rather than assuming the one I happened to run was the only one carrying it. UNSITE=1 reaches live-failures.sh and live-stress.sh too, for a reason worth recording. owl-ded9's own configuration puts its receiver and illuminator at the same point, so the server refuses its registration with `400: tx_lat` and the node never gets a token. Both scripts need a registered node to test anything at all: run against that config, live-failures.sh produced two registration attempts and none of the five refusal paths it exists to drive. Nulling the geometry in the scratch copy is honest rather than a workaround, because unsited is a real fleet state, and it leaves the node's own config untouched as every script here does except live-stalled.sh. Verified on owl-ded9 against the tunnelled mock, all four now reaching what they were written for: probe stages 1 and 2, every check passed against live config and frames service sited config refused as production refuses it, with the new detail naming tx_lat; unsited registers and streams failures 409, 429, pause, resume and 401 all driven; states error, paused and streaming all seen; detections stop dead after the 401 while heartbeats continue stress one registration across three processes, three boot_ids each restarting seq from 1 with nothing dropped, and a node with an unreadable config still beating with config_version null and the YAML error in errors[] No required-nullable key was dropped from any payload across the whole run, which is the fourteen-field rule holding on real data. live-stalled.sh is fixed the same way but not run: it stops blah2 on a live radar node and needs its own sign-off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes PUT /v1/nodes/contact, whose retina-gui half landed today. Five optional fields, read from /data/retina-gui/telemetry-contact.json and sent unchanged: the stored shape mirrors the wire's NodeContact one-for-one, so stage 2 here is the thinnest builder in the service and converts nothing. The discipline is all in when *not* to send. **A node with nothing to report never calls the endpoint.** The spec says so, and it is not a nicety: an empty document is a valid payload that clears whatever the server holds. That is right for an owner who deleted their details and wrong for one who never gave any, and the two are told apart by whether anything has been sent this process. So an absent file is a complete answer rather than a gap, and the ordinary node sends nothing at all, forever. **On local change only.** Nothing on the server asks for this and no response marks it stale, so a change in the file is the only thing that sends it. Comparison is frozen-dataclass equality, the same mechanism as NodeConfigRaw. Clearing is a change like any other, so an owner deleting their details produces an empty document that travels and removes them server-side. It rides the config loop rather than a sixth thread of its own. Both documents are noticed the same way, by re-reading a file retina-gui wrote on a timer, because nothing in this stack pushes at us. It runs *before* the configuration and is not guarded by it: a node whose config.yml is unreadable can still say who owns it, and that is exactly the node somebody needs to ring. **A refusal goes to errors[] and never to the status document's detail.** Unlike a refused registration, a rejected contact document breaks nothing: the node registers, streams and beats exactly as before, and the only loss is a way to ring the owner. `detail` is for the things stopping a node working, and filling it with something that is not would train an operator to ignore it. Nothing is substituted, the same discipline as the consent records and the beam geometry, and more pointedly here: these values reach a person, so inventing one would put a wrong name or a wrong number against a real node. A non-string is dropped rather than coerced, because str(441234567890) is a phone number nobody typed. Reading never raises. An unreadable contact document is indistinguishable in consequence from an absent one, and neither is a reason to stop streaming. The mock gains the endpoint, its validator and its own refusal slug: `invalid_contact`, not `invalid_config`, which is why v1.2.0 split them. The empty string is refused while null is accepted, because the schema is explicit that a node with nothing to say has exactly one way to say it. Verified on owl-ded9 through the tunnelled mock, both halves: with the node's own state, which has no contact document, nothing was sent across the whole run; with CONTACT=1, one PUT carrying all five fields and no repeat over six poll ticks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
UNSITE=1 reached the other three scripts and not this one, which made it the only script in tools/ that still could not run on owl-ded9. That node's own configuration puts its receiver and illuminator at the same point, so the server refuses its registration and it never streams. `stalled` is "healthy client, radar produced and then stopped", so it is only reachable from a node that registered and streamed first. Without a registerable configuration this script stopped blah2, waited, restarted it, and reported `reached stalled: NO` with nothing having gone wrong: it silently proved nothing while looking like it had run. Found by running it. First confirmation of `stalled` on real hardware: 10:38:06 state=streaming blah2=up 10:38:16 state=stalled blah2=down ...seven beats... 10:39:26 state=streaming blah2=up Which is the distinction v1.1.1 added the value for. The radar stopped, and the node said so against the radar rather than raising `error` against itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The deploy fragment described /data/retina-gui as being there "for the telemetry consent record". It now carries three things this service reads: the consent records, the setup-wizard flag and the contact document. Worth stating because the useful fact is the absence of work: the contact document landed in the same directory, so PUT /nodes/contact needed no compose change at all. A reader comparing this fragment against the contact feature would otherwise go looking for a mount that was never required. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eight commits, four separable strands. Happy to split into separate PRs if you would rather review them apart; they are grouped here because they were found and verified together.
The retina-gui half of the contact work is offworldlabs/retina-gui#90. They are independent: this side treats an absent contact document as a complete answer, so neither breaks the other whichever way round they merge.
1. The contract
a804be4adopts v1.2.2, three revisions on from the 1.1.1 we were implementing.a64e655then does what those revisions were for: an unsited node registers with seven explicit nulls instead of holding.Under v1.1.1 the wire could not carry a null geometry, so registration was held and the fleet had no sight of a node nobody had configured. v1.2.0 made the six coordinates required-and-nullable and v1.2.2 did the same for
tx_callsign, which is what makes the first change reach the case it was made for: a tower's name and its position are set at the same wizard step, so a node with no position has no name for one either.is_locatedstays and gates nothing. It feeds the status document, which is worth keeping precisely because an unsited node now looks entirely healthy from outside: registered, streaming, heartbeating, contributing nothing.The generator needed help, and that is the part most worth a look. v1.2.0 onwards is exported from the server's own FastAPI models, which spell nullable as
anyOf: [X, {type: null}]where the hand-written 1.1.1 usedtype: [x, "null"]. GivenanyOf, datamodel-codegen cannot attach a branch's constraints to a union member, so it manufactures aRootModelwrapper per field, and--collapse-root-modelsdoes not reach those. Adopting 1.2.0 raised the count from 3 wrappers to 19, which isRxLat(root=51.4)at construction and.rootat every read.tools/normalise_spec.pyrewrites that one idiom on a temporary copy;docs/node-ingest-v1.ymlstays byte-identical to what was published.Checked
NodeConfig.beam_width_degon adoption per the working agreement. It survived.2. The contact endpoint
cead54ereads/data/retina-gui/telemetry-contact.jsonand sends it.c86e753corrects the deploy fragment.The discipline is all in when not to send. A node with nothing to report never calls the endpoint — an empty document is a valid payload that clears what the server holds, right for an owner who deleted their details and wrong for one who never gave any. Sent on local change only, compared by frozen-dataclass equality.
It rides the config loop and runs before the configuration, unguarded by it: a node whose
config.ymlis unreadable can still say who owns it, and that is exactly the node someone needs to ring.A refusal goes to
errors[]and never to the status document'sdetail. A rejected contact breaks nothing;detailis for what stops a node working.No compose change was needed —
/data/retina-guiis already mounted read-only for the consent records.3. The live scripts had silently stopped testing anything
Possibly the most urgent strand.
f282541,e3a4fa9andaaa7865.All four
live-*.shscripts set noWIZARD_FLAG_PATHand never mounted/data/retina-gui, so since the setup-wizard gate landed on 2026-08-26 every one of them held atsetup_incompleteand tested nothing past the first gate. Three weeks of verification proving nothing while looking like it ran.live-stalled.shhad a second version of the same problem and reportedreached stalled: NOwith nothing having gone wrong.They also synthesised consent records on the grounds that retina-gui persists none, which stopped being true at v0.7.0.
4. A refused registration now says why
95f59f9. It went toerrors[]and nowhere else, and nothing reads that list: retina-gui rendersdetailand ignores the rest.owl-ded9 is what that costs. It has been refused 232 times over nine days with a 400 naming
tx_lat— its receiver and illuminator are at the same point — and its owner saw a blank line. Three refusals now get three sentences, because they point at different people. The 403 is shown from the first refusal, with the reassurance in the wording rather than in a delay.Verification
All gates pass including
tools/check.sh --tracked.Verified on owl-ded9 against a tunnelled mock, on restored release software (
blah2:v0.4.3) after cleaning development builds off the node:live-probelive-servicedetailnamingtx_lat. Unsited registers and streams at 1.06 Hzlive-failureslive-stressboot_idrestartingseqeach time with nothing dropped, unreadable config still beating withconfig_version: nulllive-stalledstalledon real hardware:streaming → stalledwithblah2=down, then recovery. It reportedstalledrather thanerror, which is the distinction v1.1.1 added the value forNo required-nullable key was dropped from any payload across the whole sweep.
Known, and not addressed here
owl-ded9's receiver is still on the tower's coordinates, so every run above needed
UNSITE=1. Correcting it needs the node's true position. After this lands it will at least explain itself on its own home page.🤖 Generated with Claude Code