Repository navigation
Calibrate empirical coverage from the claim lane, not the emit loop - #368
Merged
Merged
Conversation
Every node's published `empirical_coverage.polygon` is evidence-only, built from calibration points that come from ADS-B hexes bound to tracks. That is right for real hardware — a real node's declared beam azimuth and width are unsurveyed configuration, and drawing them would claim coverage nobody measured — but wrong for the simulator fleet. The simulator emits a detection only for an aircraft inside the node's declared cone, so for a synthetic node the cone IS the detection area by definition, and the evidence is the unreliable half: roughly a third of those hex binds are to the wrong aircraft. Measured on test 2026-09-13, synth-GVL-SCAT-0032 (42 deg beam) held 3,037 calibration points of which 47% lay outside its wedge (34% ignoring the two edge bins), 55 out-of-wedge bins had opened, and the published polygon covered 71 of 72 bearings. Every synthetic node on the test map draws as a disc. So register_node_blocking passes declared_geometry_is_truth=is_synthetic_node to the analytics registration, and the library publishes declared_wedge_polygon() for those nodes instead (retina-analytics 6df7c3d, pinned here). Real nodes are untouched. Both kinds now name their rule in `empirical_coverage.polygon_source`, so no consumer has to guess which shape it was served. is_synthetic_node is imported inside the function: tcp_handler imports this module at import time, the same cycle analytics_refresh and core.state already step around the same way. test_ingest_event_loop's registration stubs take **kwargs — they stand in for the analytics registry, which now receives one more keyword. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The site popup quotes measured coverage and nothing else — the declared beam azimuth, width and range were deliberately removed from it, because most nodes' aim was never surveyed and beside a calibration-point count they read as measurements. That stays true for real nodes. A synthetic node is now served its declared cone as its detection area (the simulator emits a detection only inside it, so it is the detection area by definition), and the backend says which rule it used in `empirical_coverage.polygon_source`. NodeInfo carries it through as `empirical_polygon_source`, and the popup line moves into a pure `coverageLine(node)` helper in nodeSites.ts so the wording is unit-tested rather than buried in JSX: the declared beam is quoted only when the source reads "declared", and "learned" is treated as measured, because it is. Missing on a payload from an older server, where it defaults to "evidence" — the conservative reading, and what every node published then. CoverageLayer needs no change: it draws whatever polygon it is served. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
services/calibration.py has held the one age rule and the one skew rule since the frame and solve paths disagreed on them. It gains a third call site's entry point — record_claim_calibration — and a fourth rule in the docstring saying which call site may record at all. The claim lane's recorder shares the age rule (at 250 m/s a 10 s fix is 2.5 km stale against a 5-degree grid) and deliberately does not share the skew rule. That is not an omission: its caller dead-reckons the fix to the frame instant, so the fix and the detection it describes are the same instant by construction and there is no skew left to bound. The test says so explicitly, because a later reader would otherwise "fix" the missing gate. The four constants are the gates the caller applies. 3 µs / 8 Hz against the claim gate's 10 µs / 25 Hz: the simulator's measurement noise is sigma 0.1-0.2 µs and 2-4 Hz, so this is still over 5 sigma at the noisy end while shrinking the delay x Doppler area a wrong aircraft can land in by ~10x. Three claims with no gap over 10 s is the maturity bar, the counterpart of the emit path's n_detections >= 3 — a wrong bind can be a coincidence once, not frame after frame at a consistent (delay, Doppler). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Empirical-coverage calibration has been dead since 2026-08-25 and nobody noticed, because the thing that killed it was a default flip. KNOWN_LANE_MODE went to "binding" in #240, and binding mode strips every detection the claim lane binds out of the frame before the tracker sees it. The emit loop records only when track.last_detection_adsb_hex is set, and that is set only from a tagged detection the tracker actually saw — so for a synthetic node, whose every detection is claimed, it is never set at all. Measured on the test deployment's persisted bins 2026-09-13: every synthetic node's newest calibration point is dated exactly 2026-08-25, 19 days old. Real nodes get a trickle (3-6 days) because a few tagged detections survive claiming — but those are precisely the binds claiming refused, which is adverse selection, and it shows: across eight nodes 5-41 % of the points sit outside the node's own declared wedge, and the simulator only ever emits a detection inside it. synth-GVL-SCAT-0032, a 42-degree beam, held 3,037 points of which 47 % were out of wedge. So the claim lane becomes the one source whenever it runs, and the emit-loop block is gated on mode "off" (where it is unchanged, rules and all). In shadow mode the reason is different and just as decisive: nothing is stripped, so both paths would record the same detection twice and the bin-count gates would be reading a doubled denominator. A claim is not automatically a calibration point. Claiming binds what it can explain; calibration records what it could not have explained any other way, and _calibration_from_claim charges every claim to exactly one of six outcomes: recorded, or one of the five rules, in order. No hold or follow claim (neither has a fresh transponder fix behind it). Fresh fix at the frame instant. Tight residual, unscaled by fix age. Uncontested — no dark projection inside the claim gate, and no OTHER known hex's prediction inside it either. Mature link — three claims, no gap over 10 s. Rule 4's second half is what the path-2 restructure is for. The candidate list is now built for every frame carrying detections, and without skipping the hexes paths 1 and H already took; the claimed_hexes skip moved to the assignment's column build, so the cost matrix and every claim are exactly what they were, while a path-1 tag's detection can finally be judged against the rest of the cache. The published reject tallies are pinned to what the ASSIGNMENT rejected, so widening the population does not inflate them. The streak store is deliberately not the hold store: KNOWN_HOLD_MAX_GAP_S <= 0 is a supported rollback and calibration must not switch off with it. Every non-hold claim advances the streak, including ones that fail rules 3-4 — those are still evidence of the link, just not clean samples, and withholding them would make a node's first clean sample its third, which on a noisy link may never arrive. The recorded position is the fix dead-reckoned to the frame instant, carried from the assignment that computed it rather than re-derived, on a private `extra` key filtered back out before the claim record is written — so state.known_claims is byte-identical. KNOWN GAP, pinned by a test rather than left to be rediscovered: path H outranks path 2 and every claim creates a hold, so a node that sends no frame["adsb"] is claiming through path H from the second frame of each link onwards, and rule 1 refuses those. Such a link never matures. The whole synthetic fleet sends tags (path 1 outranks path H) so it is unaffected; KNOWN_HOLD_MAX_GAP_S=0 isolates it. See docs/pipeline.md for the numbers and the fix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The five rules are only worth their cost if they buy attribution, so this
measures it rather than asserting it. Truth is exact, not inferred: the
simulator staples the producing aircraft's hex to each detection index, so the
frame read BEFORE the tags are stripped says which aircraft every detection
belongs to, and None means no transponder produced it — clutter or a dark
target — against which any hex is wrong by construction.
Three rules are scored side by side on the same frames. "greedy" is the
literal old source: associate_detections_to_adsb's tags, which is what set
track.last_detection_adsb_hex, recorded at the reported position, with no
one-to-one, no world gate and no visibility gate. "claims" is every claim the
lane makes recorded ungated — the fairer baseline, since the new rule is a
filter on exactly that population. "new" is the five rules, scored through
_calibration_from_claim itself so the bench reads the hex and the position the
rule actually accepted rather than re-deriving either.
20 synthetic nodes, 3 seeds, 5 simulated minutes each, ~26-30 aircraft:
wrong-hex out-of-wedge points/node-min
greedy 8.9-11.0 % 8.1-10.0 % 69-100
claims 0.9-3.5 % 0-1.3 % 57-92
new 0.0-0.1 % 0.0 % 51-82
...except blind with path H enabled, where the yield collapses to 0.11-0.13
points per node-minute with 97 % of claims charged to rejected_hold. That is
the gap the previous commit documents, and --hold-gap is the lever that
isolates it; the 51-82 figures above are the tagged legs and the blind legs
with KNOWN_HOLD_MAX_GAP_S=0.
The bench swaps in its own in-memory NodeAnalyticsManager: the module-level
one is wired to backend/coverage_data, so registering there would load a
deployment's persisted bins into the run and write the run's points back out.
docs/pipeline.md section 7 is rewritten around the new source — the five
rules, the counters, the dead-since-#240 finding and the path-H gap — keeping
the emit-loop rules for mode off. The runbook gains the funnel and how to read
a flat `recorded`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The previous commit shipped the claim-lane calibration rule with a measured
hole in it: a node that sends no frame["adsb"] yielded 0.11-0.39 calibration
points per node-minute against the 51-68 a tagged node gets, with ~97% of its
claims charged to rejected_hold. That is every hardware receiver without its
own ADS-B correlation, which is the population the empirical polygon exists
for.
The cause was an ordering, not a judgement. Path H runs before path 2 and
every claim creates a hold, so from the second frame of a link onwards the
claim is a hold claim; rule 1 refused it AND returned before the maturity
streak was touched, so the link could never reach CAL_CLAIM_MIN_CLAIMS. Rule
1 was written for a hold with nothing behind it, and it was catching holds
with a live transponder behind them.
A hold carrying extra["fix_refreshed"] is not that hold. The consistency rule
in _claim_holds has already compared this very detection against the LIVE
cached fix inside path 2's own age-scaled gate, and the claim carries that fix
rather than the hold's frozen one. It is a path-2 claim in everything but
which path found the detection, so calibration now judges it as one: rule 2
against the fresh fix's fix_ts_ms (which is what the claim already carries),
rule 3 against the FRESH FIX's prediction rather than the hold's propagated
one, rule 4 against the same candidate population as any other claim, rule 5
advancing on it, and the position recorded is that fresh fix dead-reckoned to
the frame instant.
Rule 3's reference is the part that is not cosmetic. A hold predicts this
node's next measurement from this node's last measurement, so its residual is
near zero whatever aircraft is actually out there — judging a calibration
point on it would be circular, and the 3 us gate would be measuring the
propagator rather than the attribution. _fresh_fix_prediction therefore hands
back the dead-reckoned position it already computes alongside the prediction,
for the same reason path 2's candidate loop does: re-deriving it at the
recording site would be a second offset_latlon_m free to disagree with the one
the prediction was built from. Both ride on the claim's private `extra` and
are filtered back out before the registry record is written, now through
_CAL_EXTRA_KEYS rather than a single-key comparison, so state.known_claims is
byte-identical and a test pins its exact key set.
A hold whose fix is gone, or has aged past KNOWN_CLAIM_MAX_FIX_AGE_S so
_fresh_fix_prediction stops answering, carries no fix_refreshed and is still
refused by rule 1 — both halves are tested, the second through the window
where the fix is merely stale (rejected_stale_fix) before it ages out
entirely (rejected_hold).
Re-measured with calibration_attribution_bench.py, 20 synthetic nodes, seeds
42/43/44, 5 simulated minutes, same seeds old and new (the old module run from
a shadow tree so both legs see identical traffic):
old rule (new) this commit
blind, holds ON 0.11-0.39 /node-min 54.5-61.3 /node-min
blind, holds OFF 51.2-60.6 51.2-60.6 (unchanged: no holds)
tagged (path 1) 57.6-67.5 57.6-67.5 (+0.1, 39-78 hold
rejects become
residual rejects)
Attribution did not pay for the yield: wrong-hex 0.0-0.3% and out-of-wedge
0.0% on the blind+holds leg, against 7.3-10.1% / 5.5-8.6% for the greedy tags
that fed the old emit path and 1.1-3.5% / 0-1.3% for every claim recorded
ungated, on the same frames. The blind+holds leg's residual rejects rise
(363-394 vs 186-296 with holds off), which is exactly the fresh-fix residual
doing the work the hold's own residual could not.
docs/pipeline.md section 7 gains the refreshed-hold rule and loses the "known
gap" note; the runbook's funnel entry for a flat `recorded` with a large
`hold` now points at the transponder rather than at path H.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
jehanazad
changed the base branch from
feat/synthetic-nodes-declared-wedge
to
main
September 15, 2026 20:18
|
Claude finished @jehanazad's task in 9m 47s —— View job Claude finished reviewing PR #368Todo list
What changed since the last reviewThe only new commit since the prior review round is New finding — funnel invariant has an uncounted exit (
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Empirical-coverage calibration has been effectively dead since #240 (2026-08-25) made
KNOWN_LANE_MODE=bindingthe default. The only recorder was the tracker's emit loop, which needs an ADS-B-tagged detection to reach the tracker, and binding strips every claimed detection before it gets there. Measured on the test deployment's persisted bins on 2026-09-13: every synthetic node's newest calibration point is dated exactly 2026-08-25; radar3/radar3a get a trickle (newest 3–6 days old) made only of detections the claim lane refused but the greedy tagger accepted; the ret* TCP nodes have no calibration file at all. Every coverage polygon on every environment is a frozen pre-#240 picture drawn by the old greedy binder, and that binder was wrong about the aircraft roughly a third of the time (47 % of one 42° synthetic node's points lay outside a wedge the simulator never emits detections outside of).This moves calibration into the claim lane under a rule much stricter than claiming, and silences the emit-loop path unless the lane is off. Real and synthetic nodes are treated identically here; which polygon is published is #367's business.
Stacked on #367 (base branch = its head); merge order: retina-analytics #28, #32, then #367, then this.
The rule (
services/known_claiming.py::_calibration_from_claim)A claim records a calibration point only if all five hold:
_claim_holds' consistency check against a live fix); an unrefreshed hold or a follow claim never records;CAL_MAX_ADSB_AGE_S(10 s) old at the frame instant;CAL_CLAIM_DELAY_US(3 µs) andCAL_CLAIM_DOPPLER_HZ(8 Hz), unscaled by age — about a tenth of the claim gate's area;CAL_CLAIM_MIN_CLAIMS(3) claims of this hex by this node with no gap overCAL_CLAIM_STREAK_GAP_S(10 s).The position recorded is the fix dead-reckoned to the frame instant, so the old fix-vs-detection skew rule does not apply;
services/calibration.py::record_claim_calibrationdocuments that as the module's fourth rule. Per-reason counters (calibration_points_recorded,calibration_claims_rejected_{hold,stale_fix,residual,contested,immature}) are in the stats payload beside theknown_claimsblock, and the runbook says how to read them.Measured offline (
backend/scripts/calibration_attribution_bench.py)Simulator world, 20 nodes, truth = the aircraft that actually produced each detection.
points/node-min · wrong-hex · out-of-wedge, seed 7 / 240 s (seeds 42–44 in the commit body agree):The first cut of rule 1 refused all holds and starved blind nodes to ~0.1 points/node-min because path H outranks path 2 from the second frame of every link; judging refreshed holds on their fresh fix (commit 2242780) restored the yield without loosening any constant.
Changes
services/known_claiming.py: candidate list built per frame for every claim's rival check;_fresh_fix_predictionreturns the DR position; refreshed holds carry the fresh-fix prediction and DR position onextra(filtered out of the registry entry, which stays byte-identical);_calibration_from_claim+ streak store; per-frame counter flush.services/calibration.py:record_claim_calibration, fourth rule in the docstring.services/track_gates.py: emit-loop recording only whenKNOWN_LANE_MODE == "off".config/constants.py: the fourCAL_CLAIM_*constants with derivations.core/state.py,routes/test.py,vulture_whitelist.py: counters.scripts/calibration_attribution_bench.py: the measurement above,--both,--hold-gap.docs/pipeline.md§7 rewritten for the claim-lane source;docs/runbook.mdfunnel entry.Test coverage
tests/test_calibration_from_claims.py(20 new): each rule's reject, the third-claim record, DR position, rival candidate, dark contention, node tags, streak reset, refreshed hold records / unrefreshed does not / refreshed judged on the fresh residual / registry entry carries no calibration keys; lane on ⇒ emit loop records nothing, lane off ⇒ unchanged.test_solver_stats.pyupdated for the two new keys in theknown_claimsstats block.Review notes
calibration_points_recordedshould climb within minutes on every claiming node, and the newestbin_pos_tsincoverage_data/empirical_<id>.jsonshould move; live check on test owed.coverage_dataon deploy would give clean polygons faster; not done here — operator's call.🤖 Generated with Claude Code