Repository navigation
Make blah2-api ingest linear, hoist ADS-B geometry, serve JSON as JSON - #81
Merged
Merged
Conversation
Each TCP listener tested data_x.slice(-1) after appending a chunk. The append builds a lazy V8 rope, and reading its last character flattens the whole rope, so every chunk copied the entire frame so far. Ingest was quadratic in frame size, and a +-1000 Hz map is 2.2 MB. Test the chunk's last character instead. A chunk is never empty, so after appending it the buffer's last character is the chunk's, and the frames assembled are identical. Applied to map, detection, timing and iqdata. Verification: - test_ingest.js boots server.js, writes frames the way blah2 does (persistent connection, 1 KiB writes, irregular splits, a lone closing brace) and checks each GET serves the sent bytes exactly. 15/15 on the old server.js and 15/15 on the new; 6 checks fail if framing is broken. - bench_ingest.js, node 20.20.2 arm64 in blah2-api:v0.6.1 on a Pi 5, 2.21 MB frame: 8 KiB chunks 135.1 -> 3.0 ms, 1 KiB 1057 -> 3.6 ms, identical frames at every chunk size. - Live A-B-A on jonathan-node-1, cpi 0.5, +-1000 Hz, 360 s windows, with this branch's server.js mounted over blah2-api:v0.6.3: blah2-api 48.9% -> 42.4% -> 49.6% of a core, blah2 at 2.0 CPI/s throughout, every sampled map parsed with 1001 rows. The saving is about what the bench predicts at 64 KiB chunks; an earlier estimate of ~72% of a core assumed 8 KiB chunks and does not hold for current builds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The detection handler recomputed every aircraft's expected bistatic delay and Doppler inside the per-detection loop, detections x aircraft times (~280 aircraft on jn1), though neither depends on the detection. Move the matcher into lib/adsb_match.js and compute the geometry once per aircraft, then compare it against every detection. The output is unchanged: same functions on the same inputs, same aircraft order, and ties still go to the first aircraft to reach the best score. Verification: - test_adsb_match.js runs a verbatim copy of the old inline code and the new matcher over 300 randomised frames (missing and zero fields, many aircraft competing for a detection, custom tolerances) and compares serialised output: identical. An exact-tie case pins first-wins. A changed rounding fails 145 of the 300 frames; `<=` fails the tie case. - End to end, old and new server.js against a fake tar1090 (280 aircraft, 40 frames of 30 detections, 624 matches): /api/detection byte-identical. - 280 aircraft x 50 detections: 3.58 -> 0.15 ms per frame (x86). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/api/map, /api/detection, /api/timing and /api/iqdata held JSON text but res.send(string) labelled it text/html. Set application/json and send the string unchanged; res.json would have re-encoded it as a string literal. /api/timestamp is a bare integer and becomes text/plain, so the front end's $.get keeps receiving a string rather than a parsed number. Consumers checked: the web front end reads the JSON routes with $.getJSON and retina-gui with requests' resp.json(), neither of which looks at the type; the stash modules read raw bytes over http.get. type and send are separate statements because swagger-autogen degrades a chained res.type().send() to a "default" response; openapi.json is unchanged and openapi:check passes. Verification: test_ingest.js now also checks the headers. Against the previous server.js its 16 body checks pass and its 5 type checks fail; against this one all 21 pass. Bodies, and so ETags, are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three behaviour-preserving changes to blah2-api, ahead of moving the default Doppler span to +-1000 Hz (where each map is 2.2 MB instead of 0.65 MB).
Changes
data_x.slice(-1)after appending a chunk, which flattens V8's lazy rope and copies the whole frame so far on every chunk. It now checks the chunk's last character. A chunk is never empty, so the frames assembled are identical. Applies to map, detection, timing and iqdata.api/lib/adsb_match.jsand no longer recomputes every aircraft's expected delay and Doppler inside the per-detection loop. Output is unchanged, includingposition_agefrom Match detections to where each aircraft is, not where it was #76'se53f528and first-wins ties.application/json./api/map,/api/detection,/api/timingand/api/iqdatawere labelledtext/html./api/timestampbecomestext/plain. Bodies and ETags are unchanged.Every consumer was checked for the content type change: blah2's web front end reads these routes with
$.getJSON(the$.getcalls are on/api/timestamp, which stays a string), retina-gui and retina-telemetry parse the body themselves, and the stash modules read raw bytes.Verification
api/test_ingest.js(new): bootsserver.js, writes frames the way blah2 does and checks every GET serves the exact bytes sent and the rightContent-Type. 21/21.api/test_adsb_match.js(new): compares the new matcher against a verbatim copy of the current inline code over 300 randomised frames, includingposition_age. Identical. A deliberate change to the rounding fails 118 of them.api/test_*.jspass.npm run openapi:checkpasses.blah2-api:v0.6.3:Every sampled map parsed with 1001 rows. No restarts. The node was restored to production afterwards.
The saving is smaller than the earlier estimate of about 72% of a core, which assumed blah2 sends the map in 8 KiB chunks. The live result matches the benchmark at 64 KiB chunks. The rest of blah2-api's cost at +-1000 Hz is not explained yet. The likely source is
api/stash/maxhold.js, which fetches and parses the full map every CPI, but that has not been measured and is not part of this PR.Docs
None changed. The README describes only
openapi.json, which is unchanged.🤖 Generated with Claude Code