Calibrated, universal tracker: hold zero-Doppler crossings, measured R and Q, no min_snr by Purple10101 · Pull Request #37 · offworldlabs/retina-tracker

Purple10101 · 2026-10-15T14:40:15Z

Summary

Makes the tracker's filter physically honest and the same on every node. Tracks now survive zero-Doppler crossings, stop jumping kilometres, and run on noise values measured from real recordings instead of guesses. The last per-site setting (min_snr) is gone, and the legacy anomaly detection is removed so it can be rebuilt on the improved tracker.

Measured on jonathan-node-1 (VHF), fairforest B (VHF) and jonathan-node-2 (UHF) by replaying each node's own recorded detections through old and new code.

What changed

Hold tracks through blah2's zero-Doppler blank. Aircraft sit inside the ±15 Hz blank for 4-90 s at closest approach, and a confirmed track was deleted after 5 s. A miss the blank can explain no longer counts towards deletion, held no longer than the track had been observed (max 120 s). The blank is read from each node's process.detection.minDoppler.
Measurement noise R measured on three nodes. 26 hours of innovation recordings, measured from smoother residuals: delay noise is a tenth of a resolution cell everywhere, Doppler noise about 0.4 m/s of range rate on both bands. Real noise has heavy tails, so R is twice the robust width (0.25 cells at 10 dB, 0.8 m/s), the factor every node needed for the gate to admit 98.9% of real measurements. Delay noise scales as SNR^-0.4, as measured.
Process noise Q refitted by replaying recorded aircraft tracks: white jerk with a 1e-6 floor; the range-rate random walk added on 5 October is removed.
No min_snr. Every detection with a finite SNR is tracked; non-finite SNR is rejected and counted.
Legacy anomaly detection removed (it judged ADS-B rather than the radar, and on live data only flagged the tracker's own mistakes). Wire fields kept, neutral.
Scoring and recording fixes: live_score judges each side of a zero-Doppler crossing separately; the synthetic scorer reports blank_crossing_rate; innovation records carry the true prediction interval; retina_tracker.calibrate is retired (its method cannot separate R from Q at 2 frames/s).

Measured effect

4 hours per node, identical input, v0.4.0 → this PR:

	jn1	fairforest B	jn2 (UHF)
Jumps over 1 km	88 → 4	28 → 4	79 → 2
ADS-B zero-Doppler crossings kept as one track	5/26 → 20/26	1/53 → 17/50	5/6 → 6/6
Tracks per ADS-B aircraft	1.87 → 1.56	2.32 → 2.27	1.07 → 1.07
ADS-B-labelled detections tracked	99.7% → 99.5%	99.4% → 98.8%	98.6% → 95.0%
Filter self-check, median NIS (matched 1.39)	0.03 → 0.45	0.02 → 0.35	0.17 → 0.45

Live on jn1 since 12 October.

Compatibility

Wire unchanged. events.jsonl, GET /frame and the history stream keep every field retina-telemetry and retina-gui read: is_anomalous is always false, anomaly_types always empty, max_velocity_ms is the radar speed floor |range rate| / 2, and the below_snr detection class is always empty.
Config. tracker.min_snr, adsb.initial_covariance and process_noise.range_rate are removed; a config that still sets them loads and ignores them (checked with the CLI). process_noise.range_jerk default 1e-7 → 1e-6.
Test fixtures that moved their delay against their own Doppler, which the old kilometre-wide gate accepted, now move as their Doppler says.

Known limitations and follow-ups

False reacquisitions after long blank holds, about 1-3 per 4 h per node: a check that refuses all of those seen in replay is designed and costed in docs/zero-doppler-blank.md, for the stitching phase.
retina-gui still offers min_snr in its config editor and draws an always-empty "below SNR" layer; retina-node's default.yml still sets min_snr.
The synthetic harness cannot validate this filter: its noise is fitted to ADS-B residuals, about 9x real radar noise.
The Doppler-noise finding (a fixed range rate, not a share of a bin) rests on one UHF node.

Docs

docs/zero-doppler-blank.md, docs/measurement-noise.md and docs/anomaly-detection.md (new), plus README, CLAUDE.md and the config comments.

Verification

ruff check, ruff format --check and pytest (364 passed) clean on every commit; vulture unchanged.

🤖 Generated with Claude Code

…crossing live_score fitted one slope to a track's delay and compared it with the track's mean Doppler. Across a zero-Doppler crossing the range is at an extremum, so both are near zero with signs that need not agree, and every track that held a crossing read as impossible. On a jn1 replay that put two ADS-B-labelled aircraft with clean crossings among the implausible tracks. Each side of a crossing is now judged on its own, and a track is impossible if either side is. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

blah2 blanks Doppler near zero, so an aircraft's detections change sign across a gap. blank_crossing_rate is the share of crossings, of those a track reached, where the same track holds the aircraft on the far side. It is the number the blank work is judged on, and it was 1-6% on the harness. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

blah2 never reports within process.detection.minDoppler of zero, and an aircraft crossing zero Doppler sits there for 4-90 s because its range is at an extremum. A confirmed track was deleted after 10 missed frames, 5 s, so nearly every crossing ended one track and started another. A miss the blank can explain is no longer counted towards deletion: the prediction is inside the blank, has crossed zero since the last detection, or its gate reaches into the blank. Once inside, the hold latches until the next association, because the acceleration estimated going in is too poor to time the exit. A track is held no longer than it had been observed, and never longer than 120 s. The observation limit is what keeps the hold from making false tracks. Without it, jn1 tracklets of 5-7 detections were held for 27-51 s and then took detections 3-11 km away. The blank is read from the node's blah2 config through --blah2-config. Unset means unknown, and then nothing is held. On a replay of 2.5 h of jn1: labelled crossings held 0 of 10 to 8 of 10, implausible tracks 1 to 1, tracks per aircraft 1.82 to 1.50. On the synthetic harness, crossings held 1-6% to 29-30%. Design and measurements are in docs/zero-doppler-blank.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

… stop jumping A confirmed track would take one stray detection kilometres off its path and come straight back. R was a fixed 1 km of delay noise, about 7 cells, and the SNR scaling took it past 2 km at the detection floor, so the gate was about +/-6 km for exactly the detections most likely to be false. Real scatter is a small fraction of a cell: consecutive steps in jn1 tracks miss constant acceleration by a median 15 m. R is now one resolution cell on each axis, c/fs by 1/cpi from the node's own capture. Innovations from jn1 and fairforest B both put R 25-33x too large in delay but disagree on a measured k by 3.7x, so one cell is the conservative value that needs no fitting. Q gains a range-rate random walk, process_noise.range_rate = 1e-5 km^2/s^3, because a tighter R without it loses aircraft where Doppler slews. The ADS-B start covariance squares its position setting. It was a standard deviation used as a variance, which would now be looser than a radar-only start. Replayed on three nodes under their own configs, jumps over 1 km went from 24 to 0 on jn1, 328 to 27 on fairforest B and 30 to 0 on jn2 (UHF), with live_score's implausible tracks falling at all three. No real aircraft was lost from tracking; the "aircraft" that went were a wide-gate artefact on jn1 and CW-tone tracks with coincidental ADS-B labels on fairforest B. One real jn1 crossing was lost. On the harness, ID switches halved and clutter absorption fell from 10-12% to 3-4%. Fourteen tests failed on fixtures that were physically impossible, mostly a delay moving against its own Doppler, which the old gate accepted. They now move as their own Doppler says, and the intermittent ADS-B test that was a strict expected failure for that reason now passes. Design and measurements are in docs/measurement-noise.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Two goals, agreed with Josh: an ADS-B-verified aircraft shows as a single track, and the filter only has to model well-constrained physics, leaving dramatic changes of velocity or direction to stitching after the fact. The second is a design rule for the filter: it is why R can be one resolution cell and why Q should not be raised to catch manoeuvres. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ratch Nine of its ten checks judged ADS-B rather than the radar, so a target with no transponder got one fixed-threshold test. On live data it only caught the tracker's own mistakes: over 34.5 hours of v0.4.0 on jn1 its four flags were all identity swaps from tracks that had jumped onto a second aircraft, and with the gate sized from the resolution cell it flagged nothing in 8.8 hours. Removed: every per-track check and its state, their thresholds, the synthetic scorer's anomaly metrics, the anomaly test suites and the integration compose file, which existed to confirm anomaly flags and no longer ran. The wire is unchanged, because retina-telemetry requires the fields on every track: is_anomalous is always false, anomaly_types always empty, and max_velocity_ms is now the radar's speed floor, |range rate| / 2, which holds in any bistatic geometry. A track's adsb_hex no longer switches when a second aircraft's hex appears on it. The reasons and the rebuild proposal are in docs/anomaly-detection.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

R was one resolution cell, a conservative bound. Recorded for 26 hours on jn1, fairforest B and jn2 under the same code, and measured from smoother residuals of ADS-B and physics-checked tracks, delay noise is a tenth of a cell on every node and Doppler noise is about 0.4 m/s of range rate on both VHF and UHF, 2.2x apart in bins. Delay variance falls as SNR^-0.4, not SNR^-1; Doppler noise barely moves with SNR. Delay noise is now 0.12 cells at 10 dB, scaled by max(SNR/10 dB, 0.1)^-0.4; range-rate noise is a fixed 0.4 m/s. One function scales it for both the update and the gate, which had carried its own copy of the formula. A track's start no longer takes ADS-B's 100 m position uncertainty: its state always came from the radar detection, whose measured noise is 18 m. The adsb.initial_covariance setting goes with it. Six test fixtures moved their delay against their own Doppler, jumped back to their start, or sat 80 m off their true position; the old 150 m floor hid it. They now move as their Doppler says. Not shippable alone: Q is corrected in the next commit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…en per site With R measured, each node's ADS-B tracks were replayed offline over a grid of white jerk and range-rate random walk, choosing the Q whose median NIS sits at 2 ln 2 across prediction intervals. Every node's best fit had no range-rate walk: it had been compensating for an R ten times too large, so it goes. White jerk fits at 1e-6 to 3e-6 km^2/s^5 at far-traffic jn2 and 1e-5 at close-traffic jn1 and fairforest B. The floor becomes 1e-6, and the adaptive scaling, which widens Q by however far the filter's own NIS says the model is failing, carries it to each site's value with nothing set per site. It never engaged before, because the oversized R kept NIS near zero. No single Gaussian Q matches every interval: right at one frame, overconfident at 1-6 s, overcautious beyond. Manoeuvres are left to adaptive widening and to stitching rather than widening every gate for every track. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

tracker.min_snr set detections below a hand-set SNR aside before the tracker saw them: 4 dB on jn1 and jn2, 7 dB on fairforest B. That is a per-site tuning, and with measured noise the gate and the physics are what reject noise, so it goes. Low-SNR detections still carry more delay noise, by the measured SNR^-0.4 scaling, so the gate admits them less readily. A non-finite SNR is now rejected and counted: it used to be routed into below_snr, and with no threshold it would reach the covariance. below_snr stays on the wire, always empty, because retina-gui draws it as a layer. A config that still sets min_snr loads and is ignored, checked against the tracker and its CLI. retina-gui's config editor and retina-node's default.yml still carry the field; that is follow-up work there. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ation tool An innovation record gave the prediction interval only as dt * (1 + n_missed), and frames held through the zero-Doppler blank are not misses: 2-4.5% of the October records were off by up to two minutes. Each record now carries interval, the time since that track's previous update. retina_tracker.calibrate extrapolated innovation scatter to zero interval to separate R from Q. At 2 frames per second nothing is shorter than one frame and scatter rises by half between 0.5 s and 1 s, so its intercept moved from 2 m to 18 m with the choice of tracks. It ran only offline and nothing imported it, so it goes. docs/measurement-noise.md now describes the method that worked: smoother residuals for R and a replay fit for Q. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Replayed on jn1, fairforest B and jn2, the measured noise as R lost 5-7% of ADS-B-labelled detections against the deployed code, and raising Q recovered little of it. The widths were robust estimates of the typical detection, and real radar noise has heavy tails: at those widths the chi-squared gate of 9, which should admit 98.9% of genuine measurements, refused 7.7-12.2%. The scale that makes the gate admit 98.9% of real aircraft measurements was 2.00x, 2.05x and 2.10x at the three nodes, on both bands, so R is now twice the robust width: 0.25 cells of delay at 10 dB and 0.8 m/s of range rate. Against the deployed code on the same 4 hours per node, jumps over 1 km fall 6 -> 4, 8 -> 4 and 4 -> 2, crossings and fragmentation hold, and recall is 99.5%, 98.8% and 95.0% against 100%, 99.9% and 100%; the aircraft no longer mostly tracked are a 454 m jump the gate is right to refuse, a one-second three-detection appearance and a sparse fast-slewing target. Adaptive widening of Q did not engage with this R; a 1e-6 floor did as well as higher values, which brought jumps back. The comments now say so. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Live on jn1, a track held through the zero-Doppler blank for 30 s or more occasionally took an unrelated detection when it reacquired, because its gate grows for as long as it is held. The constant-acceleration check that would refuse them, and its replayed cost, are written down for the stitching work. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

claude · 2026-10-15T14:40:30Z

Claude finished @Purple10101's task in 23s —— View job

Review of PR #37

I read the core diff (track.py, kalman.py, tracker.py, config.py). I did not run the test suite, and I could not launch the /code-review skill, so this is a manual pass. I found no blocking bugs.

What looks right

delay_noise_scale is shared by update() and the association gate (tracker.py _associate), so the gate and the update weight a detection with the same covariance.
is_occluded and blanked reset on association and only apply to non-tentative tracks. The max_occluded_s cap (120 s, and no longer than the track was observed) bounds how long a blank hold can run.
last_interval_s is computed from timestamps rather than n_missed, which is needed now that held misses are not counted.
The wire fields stay present and neutral (is_anomalous=False, anomaly_types=[], below_snr=[]).

Points to consider

Comment rule. CLAUDE.md and .claude/rules/code-style.md say "no comments". The PR adds several docstrings and # comments in track.py, kalman.py, tracker.py and config.py. Most of the explanatory text already lives in docs/. Either trim them or relax the rule.
Test coverage.
- The PR deletes tests/test_integration_synthetic.py (1,085 lines), test_anomaly_*, test_calibrate.py and docker-compose.integration-test.yml.
- Removing the anomaly and calibrate tests follows from removing the features.
- The synthetic integration test was the only end-to-end tracker check. The PR says the synthetic harness is not valid for this filter, which justifies dropping it. Say explicitly that nothing replaces it, or add a small replay-style regression test.
- New coverage (test_blank_occlusion, test_measurement_noise, test_speed_floor) looks appropriate.
delay_noise_scale bounds. The scale is capped at 2.5 at low SNR (0.1**-0.4) but has no floor at high SNR. At 40 dB it is about 0.063, so delay variance shrinks to about 6% of its 10 dB value. That follows from the SNR^-0.4 fit. A strong close aircraft therefore gets a very tight delay gate. Check that against the heavy-tail finding for strong returns, or add a floor on the scale.
mark_missed timing. self.occluded_s += self.last_dt or 0.0 uses the last predict dt, not the real frame gap. Normally these are equal. After a gap or a backwards-timestamp resync they can differ, so occluded_s could under-count and extend a hold. Using timestamp - previous timestamp would be more robust.
Config. Removed keys (min_snr, initial_covariance, range_rate) are ignored silently. A one-time warning log when a config still sets them would help operators, since retina-node's default.yml still sets min_snr.
Known limitations. The false reacquisitions after long holds (1-3 per 4 h per node) are already documented, with a design in docs/zero-doppler-blank.md. They are fine to defer, but add a tracking issue.

Verdict: sound approach with good measurement backing. Points 1, 3 and 4 are worth a look before merge.
· branch 20261011-calibrated-noise

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

claude · 2026-10-15T14:42:59Z

Claude finished @Purple10101's task in 25s —— View job

Review of PR #37

I read the core diff (track.py, kalman.py, tracker.py). The /code-review skill errored when I called it, so this is a manual pass. I did not run the tests or ruff. I found no blocking bugs.

Checked and correct

delay_noise_scale feeds both Kalman.update and the gate innovation covariance, through measurement_noise(). The gate and the update use the same covariance.
last_interval_s comes from timestamps (track.py:247), so it stays right now that held misses don't increment n_missed.
is_occluded returns False for tentative tracks. blanked and occluded_s reset on association (track.py:272-273). max_occluded_s caps a hold at 120 s and at the time the track was observed.
The wire fields stay present and neutral.

Points to consider

High-SNR floor on delay noise (kalman.py, delay_noise_scale). The scale is capped at 0.1**-0.4 ≈ 2.5 at low SNR, but has no floor at high SNR. At 40 dB it is about 0.063, so delay variance falls to about 6% of its 10 dB value. The PR says real noise has heavy tails and that the 2x factor was needed to admit 98.9% of measurements. Please confirm the SNR^-0.4 fit holds above the SNR range you measured. Otherwise add np.maximum(..., floor) on the scale.
Covariance growth during a blank hold. predict() stops growing the covariance once n_missed > N_COAST (track.py:227). Held misses never increment n_missed, so a track held for up to 120 s keeps inflating its covariance, and the gate widens with it. That may be intended, since it matches the false reacquisitions you list. If it is intended, a one-line note in docs/zero-doppler-blank.md would help.
occluded_s accumulates last_dt (track.py:295), the dt of the last predict. If predict ever skips a frame, or the timestamp resyncs backwards, the hold can be mis-timed. Using timestamp - history["timestamps"][-1] would be more robust. This is minor if the tracker always predicts with the real frame gap.
Comment rule. CLAUDE.md and .claude/rules/code-style.md say "no comments". The diff adds docstrings and # comments in track.py, kalman.py and config.py, for example delay_noise_scale, update() and is_occluded. Most of that text is already in docs/. Either trim it or relax the rule.
Test coverage. Deleting test_integration_synthetic.py removes the only end-to-end tracker test. The reason is sound, since the synthetic noise is about 9x real noise. Please state in the PR that nothing replaces it, or add a small replay-style regression test. The new test_blank_occlusion, test_measurement_noise and test_speed_floor cover the new logic.
Silent config removal. min_snr, initial_covariance and range_rate are now ignored without notice. retina-node's default.yml still sets min_snr, so a one-time warning log would help operators.
Follow-ups. File tracking issues for the false reacquisitions after long holds (the design is in docs/zero-doppler-blank.md), for the retina-gui min_snr editor and the empty "below SNR" layer, and for the retina-node default.yml min_snr setting.

Verdict: the approach is sound and well backed by measurement. Points 1 and 2 are worth a look before merge. Point 4 conflicts with a repo rule.
· branch 20261011-calibrated-noise

…hat no end-to-end test remains Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

claude · 2026-10-15T14:47:59Z

Claude finished @Purple10101's task in 24s —— View job

Review of PR #37

The /code-review skill errored when I called it, so this is a manual read of track.py, kalman.py and tracker.py, checked against docs/measurement-noise.md. I did not run pytest or ruff. I found no blocking bugs.

Checked and correct

delay_noise_scale feeds both Kalman.update and the gate (tracker.py _associate), so both use the same covariance.
The scale is clamped at low SNR (max(..., 0.1), so at most about 2.5). It has no floor at high SNR. docs/measurement-noise.md says this openly: the data supports it to 32 dB, and at 40 dB the extrapolation gives 9 m. A flooring instruction is already written there for a node whose strong returns get refused.
last_interval_s comes from timestamps, so it stays correct now that held misses don't increment n_missed.
blanked and occluded_s reset on association. is_occluded returns False for tentative tracks. The hold is capped at min(120 s, time observed).
Non-finite SNR is rejected in _reject_impossible before it can reach delay_noise_scale. The wire fields stay present and neutral.

Points to consider

Covariance keeps growing during a blank hold.
- predict() freezes covariance once n_missed > N_COAST (track.py:227).
- Held misses never increment n_missed, so a track held for up to 120 s keeps inflating its covariance and the gate widens with it.
- This is plausibly the mechanism behind the 1-3 false reacquisitions per 4 h per node.
- If it is intended, a line in docs/zero-doppler-blank.md would say so. If not, freeze growth on occluded_s as well.
occluded_s accumulates last_dt (track.py:295), the dt of the last predict, not the measured frame gap. The two normally match. After a backwards-timestamp resync, dt falls back to 0.5, so the hold can be mis-timed. This is minor.
Comment rule.
- CLAUDE.md and .claude/rules/code-style.md say "no comments".
- The diff adds docstrings and # comments in track.py, kalman.py and tracker.py (for example delay_noise_scale, is_occluded, max_occluded_s, _classify_frame).
- Most of that text is already in docs/.
- Either trim it or relax the rule.
Test coverage. Deleting test_integration_synthetic.py removes the only end-to-end tracker test. The reason is sound, since the synthetic noise is about 9x real noise, and the latest commit already records that no end-to-end test remains. test_blank_occlusion, test_measurement_noise and test_speed_floor cover the new logic.
Silent config removal. min_snr, initial_covariance and process_noise.range_rate are now ignored without notice. retina-node's default.yml still sets min_snr. A one-time warning log would help operators.
Follow-ups worth tracking as issues:
- false reacquisitions after long holds
- the retina-gui min_snr editor and its empty "below SNR" layer
- the retina-node default.yml min_snr setting

Verdict: the approach is sound and well backed by measurement. Point 1 is worth a decision before merge. Point 3 conflicts with a repo rule.
· branch 20261011-calibrated-noise

Purple10101 and others added 12 commits October 5, 2026 18:32

20261015 - Sort an import the way the repo's ruff config wants it

0deb1e9

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

20261015 - Record how far up in SNR the delay fit was measured, and t…

9a59499

…hat no end-to-end test remains Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Purple10101 merged commit 851cf9f into main Oct 15, 2026
2 checks passed

Navigation Menu

Repository navigation

Calibrated, universal tracker: hold zero-Doppler crossings, measured R and Q, no min_snr - #37

Calibrated, universal tracker: hold zero-Doppler crossings, measured R and Q, no min_snr#37
Purple10101 merged 14 commits into
mainfrom
20261011-calibrated-noise

Purple10101 commented Oct 15, 2026

Uh oh!

claude Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

claude Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

claude Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Purple10101 commented Oct 15, 2026

Summary

What changed

Measured effect

Compatibility

Known limitations and follow-ups

Docs

Verification

Uh oh!

claude Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #37

Uh oh!

claude Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #37

Uh oh!

claude Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #37

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

claude Bot commented Oct 15, 2026 •

edited

Loading

claude Bot commented Oct 15, 2026 •

edited

Loading

claude Bot commented Oct 15, 2026 •

edited

Loading