Skip to content

fix(observers): an observer's own transmissions are not receptions (#667) - #2220

Merged
efiten merged 2 commits into
Kpa-clawbot:masterfrom
efiten:fix/667-tx-observations
Oct 21, 2026
Merged

efiten merged 2 commits into
Kpa-clawbot:masterfrom
efiten:fix/667-tx-observations

Conversation

@efiten

@efiten efiten commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Refs #667 (M1, the first part).

Problem

Observations with direction = 'tx' are packets the observer's own radio sent, published by the MQTT bridge. Nothing in cmd/server or the ingestor's neighbor builder filtered on direction, so these rows counted as receptions.

Two shapes, both seen in a production DB:

  • Forwarded flood. A repeater-observer appends its own hash before retransmitting (firmware Mesh.cpp), so the tx copy's path ends on the observer itself, for example tx ["2979","2029"] next to rx ["2979"] for the same transmission.
  • Own advert. The observer's own advert goes out with an empty path and no rx copy.

Effects:

  • The node detail "Heard By" card listed the observer as having heard itself. On one repeater-observer that was the only row in the card (2,107 packets, no SNR).
  • The neighbor graph counted a forwarded advert twice, once from the rx copy and once from the tx copy (originator ↔ path[0]). This happened in both the in-memory build and the ingestor's neighbor_edges builder.

Scale

Last 2,000,000 observations on that DB, about 7.5 days:

  • 1,385 tx rows against 1,805,378 rx rows;
  • all tx rows have SNR and RSSI NULL;
  • every tx row with a non-empty path has an rx row from the same observer for the same transmission.

Change

  • StoreObs.isOwnTransmission() (store.go).
  • These now skip tx rows:
    • buildDirectHeardIndex (direct_heard.go);
    • phase 1 of the neighbor graph build (neighbor_graph.go);
    • the ingestor's buildAndPersistNeighborEdges SQL, via o.direction IS NOT 'tx' (NULL rows from older bridges still count).

Deliberately not changed

  • Reach (node_reach.go). A tx row with path [A, X] from observer X is the only row that shows X received A and forwarded it, because X is absent from the rx copy [A]. There it is valid evidence.
  • Home regions (computeNodeHomeRegions). A tx row of the observer's own zero-hop advert counts the observer's own IATA, which is where it is.
  • Counters and region filters. These include observation counts, byObserver, packet_count and the SQL region joins. There the effect is double counting of about 0.08% of rows, not a wrong fact. That is left for a follow-up if wanted.
  • Existing neighbor_edges rows. The builder is incremental from its watermark, so edges persisted before this change keep their counts. The Heard By card is rebuilt from the store, so it is correct after a restart.

Tests

  • Red then green:
    • TestBuildDirectHeardIndex_SkipsOwnTransmissions: before the fix the observer was credited for itself;
    • TestBuildNeighborGraph_SkipsOwnTransmissions: X↔R1 count 2 instead of 1, plus a zero-hop edge from an empty-path tx row;
    • TestNeighborEdgesBuilderSkipsOwnTransmissions: 2 neighbor_edges rows from a single tx observation.
  • go vet and go test . pass in cmd/server and cmd/ingestor.

No frontend change. I have not yet checked the "Heard By" card in a browser; that will be on a staging deploy.

efiten and others added 2 commits October 15, 2026 15:23
…pa-clawbot#667)

Observations with direction "tx" are packets the observer's own radio sent.
A repeater-observer appends its own hash before forwarding a flood, so the
tx copy ends on the observer itself, and its own advert goes out with an
empty path. Counted as receptions, the node detail "Heard By" card listed
the observer as having heard itself, and the neighbor graph counted each
forwarded advert twice (once per rx and tx copy).

Skip tx observations in buildDirectHeardIndex, the in-memory neighbor graph
build and the ingestor's neighbor_edges builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	cmd/server/neighbor_graph.go
@efiten
efiten merged commit e8f6d41 into Kpa-clawbot:master Oct 21, 2026
12 checks passed
@n30nex

n30nex commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

I checked 11e06295 against current master and the firmware's flood-forwarding/TX logging path; I found no production regression in the TX exclusion.

One useful regression-test addition: TestBuildNeighborGraph_SkipsOwnTransmissions checks the X↔R1 count only if that edge is encountered. An entirely empty graph also passes. Explicitly assert that the RX X↔R1 edge exists with count 1, while keeping the prohibited zero-hop-edge assertion. That catches accidentally dropping valid RX evidence along with TX rows.

efiten added a commit that referenced this pull request Oct 22, 2026
Release notes for v3.15.0: `docs/release-notes/v3.15.0.md` and the
`CHANGELOG.md` entry. The release workflow uses the notes file as the
GitHub release body.

## What is in it

- 77 commits since v3.14.0 (76 merged pull requests plus this one): 27
fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs, 1 chore. Nineteen feats, so
minor.
- "Read this before upgrading" first. It covers:
  - the liveness change (#2261, #2262);
- the `users.db` v6 migration and what it means for a rollback (#2174);
- the backfill at every ingestor start (#2133) and the 24 h
planner-statistics refresh (#2158);
- the caches that trade freshness for latency (#2165, #2222, #2147,
#2156);
- changes in hop names and region placement (#2188, #2190, #2234, #2245,
#2220);
  - the role thresholds on favorites (#2215);
  - the geo-filter save on a single-file mount (#2137);
  - the `/api/stats` polling in hidden tabs (#2162).
- Features, fixes, performance, then tests/CI/docs.
- 29 issues closed, taken from each pull request's
`closingIssuesReferences`.
- Contributors: pull request authors and issue reporters with counts.
The commit co-author trailers only name the pull request's own author or
tooling.

## How it was checked

- I read every pull request body. Every number in the notes comes from a
pull request body.
- I checked the config keys and defaults against `config.example.json`
on `2bb000f2`.
- I checked the UNASKED and opt-in claims for #2125, #2133, #2235 and
#2239 against the source.
- For the other pull requests I relied on their bodies and did not open
the diffs.

## Not covered

- The uncertain count is a single count on one staging instance on the
day of the change (1,682 repeaters and rooms). It is not a before/after
comparison.
- #2133 gives no measured cost for its backfill on a large database, and
the notes say so.

## After merge

Tag `v3.15.0` on the merge commit once its `:edge` image exists. Then
check in the registry that `v3.15.0`, `v3.15`, `v3` and `latest` share
one digest, and that the release has the notes as its body and both
`corescope-decrypt` binaries.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants