Skip to content

Remove blah2's built-in tracker - #69

Merged
Purple10101 merged 1 commit into
mainfrom
20260918-remove-blah2-tracker
Sep 19, 2026
Merged

Purple10101 merged 1 commit into
mainfrom
20260918-remove-blah2-tracker

Conversation

@Purple10101

Copy link
Copy Markdown

Summary

Removes blah2's built-in tracker outright: Track, Tracker, their tests, the
track socket and port, /api/tracker and its TCP listener, and the config
keys. -1104 lines.

It leaked, and nothing consumed it.

Why remove rather than fix

Track::remove() erased six of its seven parallel per-track vectors and missed
nInactive, which add() pushes for every track created. Tracker::process
then returned std::make_unique<Track>(track) — a deep copy of the whole Track,
every CPI — so that ever-growing vector was allocated, copied and freed twice
a second with more than one instance resident at peak.

Measured on fairforest-b over a 50 minute window at ~78 h uptime: 100% of RSS
growth landed in exactly those two mappings
, at 2.00 MB/h. No other mapping
moved. On a 2 GB board with no swap and ~870 MB free, that is 18 days to
exhaustion.

Reproduced offline against pristine origin/main (real Tracker/Track,
synthetic detections, 20,000 CPIs = 2.78 simulated hours):

    iter    simh   RSS kB    dRSS   nInactive    id   assoc
    2500    0.35     4304    1328       37500    30      30
   20000    2.78     8400    5424      300000    30      30

id holds correct steady state at 30. nInactive climbs linearly to 300,000.

Fixing the missing erase alone was viable, but nothing reads this tracker:

  • blah2-api's /api/tracker was the only reader of the track socket, and the
    only reference to that endpoint anywhere was a debug link in a collapsed
    section of web/html/controller/index.html.
  • retina-gui's Tracker page proxies retina-tracker's SSE feed, not this.
  • It never emits a non-tentative track anyway: the association gate is pinned at
    the origin (delayPredict/dopplerPredict are initialised to 0.0 and never
    assigned from the prediction), so with minDelay: 5 nothing can ever
    associate and Track::to_json omits tentative tracks. /api/tracker has
    always returned data: [].

Live verification on owl-ded9

Deployed with the matching retina-node change (see below) and left overnight.

check result
starts against the stripped config restarts=0, exit=0, zero errors or aborts
RSPduo NumDevs=1, normal startup
tracker key in /api/timing gone
/api/tracker 404
listeners on port 3003 0
map / detection / timing / iqdata / timestamp all 200
retina-tracker sidecar restarts=0, still consuming the detection feed
duty 92.9%

Memory over 12.84 h and 86,919 CPIs:

  0.36h  281.98 -> 283.83   +1.84     warm-up ends
  1.36h  283.83 -> 284.44   +0.61     gap 1.00h
  4.36h  284.44 -> 284.98   +0.55     gap 3.00h
 10.78h  285.05 -> 285.81   +0.77     gap 6.25h

After warm-up: +1.98 MB over 11.87 h = 0.167 MB/h, a 12x reduction on
the 2.00 MB/h it replaces. The character changed too: the old leak moved on
every sample, this is a flat line with three isolated steps whose gaps are
doubling (1 h, 3 h, 6.25 h) — the signature of a geometrically-growing
structure rather than linear accumulation. VmHWM == VmRSS at the end, so
nothing transient is hiding above it.

Runway on a 2 GB board goes from 18 days to roughly 7 months.

Deployment: three images move together

blah2, blah2-api and config-merger must be deployed as a set.

The matching retina-node change strips process.tracker and
network.ports.track from the merged config. Against that config:

  • an old blah2 reads process.tracker.enable through ryml, whose default
    error handler calls abort() — it dies rather than warns;
  • an old blah2-api calls server_tracker.listen(config.network.ports.track)
    with the key gone, so it binds a random free port, silently.

The second one is easy to miss: during verification /api/tracker kept
answering 200 after the config was stripped, purely because the API had not
restarted and was still holding its original bind on 3003.

Recreating blah2-api also takes blah2 down with it (exit 0, Error sending data: Broken pipe, recovers). Pre-existing and unrelated to this change, but it
means the rollout costs a brief blah2 restart.

Not addressed here

The association bug that makes the tracker useless is moot now, but the same
defects exist in the retina-tracker sidecar's lineage if it was ported from
this code — worth a look separately.

🤖 Generated with Claude Code

blah2's own tracker has never been part of the product. Tracking is done by
the retina-tracker sidecar, which blah2-api feeds detections over
network.tracker_forward and which runs its own association. Nothing consumed
blah2's tracker output: retina-gui's Tracker page proxies retina-tracker's SSE
feed, and the only reference to blah2-api's /api/tracker anywhere was a debug
link in web/html/controller's collapsed API section.

Leaving it in was not free. Track::remove() erased six of its seven parallel
per-track vectors and missed nInactive, so one uint64_t leaked per track ever
created, and Tracker::process() then deep copied the whole Track every CPI. On
fairforest-b that took the tracker stage from 0.19 ms at 4,472 CPIs to 30-43 ms
at 558,381, and the process from 285 MB to 510 MB RSS over 77 h. Nodes have no
swap, so a 2 GB board exhausts memory in roughly a fortnight of uptime. The
desynchronised nInactive also silently corrupted the deletion logic, which is
gated on get_nInactive(i) > nDelete.

It was a standing source of confusion too: four interacting defects in
Tracker::update meant it never associated anything, so /api/tracker always
returned an empty data array, which people kept rediscovering.

Removes the sources, both CMake targets, the /api/tracker endpoint and its
port-3003 listener, the tracker keys from the repo configs, and the comments
that explained the pipeline ordering in terms of the tracker's state.
Detection output and the retina-tracker feed are untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Purple10101

Copy link
Copy Markdown
Author

Pairs with offworldlabs/retina-node#40, which makes the config change reach deployed nodes. Deploy blah2, blah2-api and config-merger together.

@Purple10101
Purple10101 merged commit 794bed3 into main Sep 19, 2026
14 checks passed
@Purple10101
Purple10101 deleted the 20260918-remove-blah2-tracker branch September 19, 2026 10:17
Purple10101 added a commit that referenced this pull request Sep 23, 2026
/api/tracker went out in #69, but the api tag description still listed
"tracks" among what the tag covers, so the reference advertised a route
that no longer exists.

openapi:check cannot catch this: the tag text is hand-written and lands
identically on both sides of the comparison, so it stays green while the
prose goes stale.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant