fix(nodes): place nodes in regions by zero-hop home regions (#1879)

[anieto]

Red commit: cd546b2 (fails on assertions, e.g. region "SAT": got [pk_aus pk_edge pk_far], want [pk_edge pk_far])
Red commit (fewest-hops fallback): a66a7f3 (fails on assertions, e.g. region "AUS": got [pk_cap pk_gulf pk_near pk_tie], want [pk_cap])
Red commits (review round 1): d4154ee (region_placement missing), 862a451 (zero: RegionFallbackMaxHops() = 3, want 0), 16dcd8d (pk_mixed:{fewest_hops [AUS] hops=0}, want [SAT] hops=2), 95f2aca (pk_late not placed after the startup load finished)

Fixes #1879

What this does

/api/nodes?region= put a node in every region where any observer heard any of its adverts. Adverts flood, so a node showed up in every region its adverts got relayed into, not just where it actually is.

Now, as agreed in the issue:

• Nodes heard zero-hop are placed by their home regions. Same rule as Bug/feature: Use actual repeaters region data and not observer based #804: a zero-hop DIRECT advert can't have been relayed, so the observer that heard it is in RF range. This is the tally computeNodeHomeRegions() already does, pulled out into nodeZeroHopRegionCounts() so analytics and the filter share one definition.

• It's a set, not a winner. A node heard zero-hop in two regions is in both. So per-region counts don't add up to the node total anymore, on purpose.

• Everything else goes by fewest hops. A node never heard zero-hop is in the region(s) whose observers heard it in the fewest hops, and in no region when even that is more than regionFallback.maxHops (config, default 3); it still shows unfiltered. The old fallback (every region that heard it at all) turned out to be most of the leak: see the numbers at the bottom. Hearings with no path_json (legacy NULL rows) are unknown, not zero hops, and are skipped.

• Only nodes the last recompute didn't cover (first advert newer than it, or no hearing with a path) still use the old observer check.

• region_placement on every node row (list and detail) says which rule placed it, so an operator can tell the four outcomes apart:

  {"basis": "zero_hop", "regions": ["AUS"]}
  {"basis": "fewest_hops", "regions": ["HOU", "SAT"], "hops": 2}
  {"basis": "none", "hops": 5}
  {"basis": "observer_scan"}

  It's a named struct (NodeRegionPlacement), documented as its own OpenAPI component. It replaces the earlier home_regions field.

How

• node_home_regions.go: a recomputer in the bridge_recomputer.go shape (sync prewarm, ticker on AnalyticsDefaultRecomputeInterval(), atomic snapshot, panics swallowed), started from main.go next to the bridge one. It also recomputes once when StartupLoadDone() closes, like recomputeWhenLoaded does for the analytics recomputers, because the prewarm only sees the first chunk.
• Each recompute builds one snapshot, pubkey → NodeRegionPlacement, from nodeZeroHopRegionCounts plus nodeFewestHopsByRegion. Hop counts come from path_json without decoding (pathHopCount: hop hashes hold no commas).
• RegionNodePubkeys loads the snapshot (atomic, no lock), skips placed nodes in the observer scan, then adds the ones whose regions intersect the request. The 30 s region cache is unchanged.
• computeNodeHomeRegions() keeps its plurality output for analytics; only the counting moved. The Bug/feature: Use actual repeaters region data and not observer based #804 / fix(#804): attribute analytics by repeater home region, not observer #1025 tests are unchanged and pass.
• No new map[string]interface{}: the node rows set region_placement to the struct, the OpenAPI component uses a small typed openAPISchema, and the old enrichNodeHomeRegions helper is gone. Touched files are at the upstream count.

Performance

• Request path: one atomic load plus map lookups. RegionNodePubkeys adds one pass over the snapshot per cache miss, so O(nodes), well under the existing advert scan it sits next to. BenchmarkRegionNodePubkeys stays ~19 ms.
• Recompute, off the request path, once per interval under s.mu.RLock: BenchmarkRecomputeNodeHomeRegionSets times the whole function at 220k adverts × 8 observations over 0-7 hop paths (one in eight zero-hop direct) plus 1M channel messages: ~54 ms/op on an Apple M5 (Docker, linux/arm64), of which the fewest-hops pass (BenchmarkNodeFewestHopsByRegion) is ~40 ms.
• That's after two small speedups in 3dc3cb7: nodeZeroHopRegionCounts walks byPayloadType[ADVERT] instead of every packet (same counts), and the fewest-hops pass looks up a node's map once per advert instead of once per observation. Before them the whole recompute was ~72 ms on the same machine.

Tests

• TestRegionNodePubkeysUsesZeroHopHomeRegions: a node whose relayed advert reached SAT stays out of SAT; the boundary node is in both; the never-zero-hop node is placed by the fallback.
• TestBoundaryNodeStaysInEachHomeRegion: the two-region case on its own, so a later refactor can't quietly collapse it back to one winner.
• TestNodeRegionPlacementsZeroHop: the snapshot for zero-hop and fallback nodes.
• TestHandleNodesRegionPlacement: /api/nodes returns zero_hop, none and observer_scan placements for seeded nodes.
• TestRegionFallbackUsesFewestHops: fewest hops wins, ties keep both, exactly the cap is placed, past it is in no region (and the observer scan doesn't re-add it), a closer later advert counts.
• TestRegionFallbackMaxHopsConfig: the config accessor (unset or negative keeps 3, 0 allowed, parsed from JSON) and the recompute honoring it.
• TestRegionFallbackIgnoresUnknownPaths: an empty path_json doesn't count as zero hops; [] still does.
• TestRegionFilterObserverScanForNodesNewerThanRecompute: the observer-scan path for a node newer than the snapshot.
• TestRegionFilterListingEveryRegionOmitsUnplacedNodes: naming every region explicitly still leaves out a node in no region.
• TestNodeRegionsRecomputeWhenStartupLoadDone: placements cover a later chunk once the startup load is done.
• cd cmd/server && go test ./... and cd cmd/ingestor && go test ./... pass (CGO_ENABLED=1). gofmt and go vet are clean.

No UI change; the API change is covered by the handler test above.

Config

• regionFallback.maxHops (default 3), documented in config.example.json.
• Per AGENTS.md rule 8, this should be exposed in the customizer in a later milestone. Not done here.

Not in this PR

• Transport-direct zero-hop adverts aren't counted as zero-hop (pre-existing, nodeZeroHopRegionCounts): for ROUTE_TYPE_TRANSPORT_DIRECT, RawHex[2:4] is the first transport-code byte, not path_len. The fallback still places such nodes (their path_json is []), but as fewest_hops, not zero_hop. I'll send that as its own PR since it changes Bug/feature: Use actual repeaters region data and not observer based #804 analytics too.
• Memory window. The answer only covers adverts still in the packet store, so quiet nodes fall back. That's the long-tail point from the thread. A node_regions table kept by the ingestor (the UpdateNodeDefaultScope pattern) would remove the window; happy to do that as a follow-up if you want it.
• One-off long-range hearings. A single zero-hop hearing during a ducting event still adds that region. On our deployment we run the same rule with a 20% share threshold per region (ignore a region holding under a fifth of a node's direct hearings), which drops those. Easy to add here if you'd like it.
• Flood adverts heard with an empty path are direct evidence too, and much more common than zero-hop DIRECT adverts. Left out of the zero-hop rule to keep one definition with Bug/feature: Use actual repeaters region data and not observer based #804; the fallback places them at 0 hops.
• Moving Bug/feature: Use actual repeaters region data and not observer based #804 / fix(#804): attribute analytics by repeater home region, not observer #1025 analytics to the set version, as you suggested, after this lands.

Fallback numbers. On our network only ~27% of nodes are ever heard directly, so the fallback decides most of the filter. With "every region that heard it", our San Antonio filter showed 241 nodes whose GPS puts them outside SAT against 149 inside, and Austin picked up Louisiana and Gulf Coast nodes heard 3-10 hops out. We've been running the fewest-hops/cap-3 rule in production since 2026-10-14; nodes with GPS outside the selected region went SAT 241 → 7, DFW 76 → 7, HOU 47 → 2, AUS 226 → 48. About 240 nodes are in no region (they still show unfiltered).

For context, on our network (Texas, ~1,500 nodes) the same idea with the broader "heard directly" rule took San Antonio from 543 nodes to 482. The 61 removed were all Austin-area nodes whose adverts get relayed south, and only 3 nodes statewide are heard directly in two regions.

[anieto]

Pushed one more change before review: nodes never heard zero-hop now go by fewest hops instead of every region that heard them (red 998219f1, green fb81de23).

On our network that old fallback was most of the leak, since only ~27% of nodes ever get heard directly. Now a node goes in whichever region heard it in the fewest hops, or in no region if even that's more than 3 hops (it still shows unfiltered). We've been running it in production today: nodes outside the selected region dropped SAT 241 → 7, DFW 76 → 7, HOU 47 → 2, AUS 226 → 48. Description is updated with the details and a benchmark (~50 ms per 5-min recompute at 220k adverts).

[efiten]

Thanks for the production numbers. SAT going from 241 to 7 out-of-region nodes makes the case, so I accept the fewest-hops fallback with the 3-hop cap, even though it differs from what we agreed in #1879. Two things before merge.

What I checked

• Shape. It follows what #1879 settled on:
  • zero-hop counting is one function (nodeZeroHopRegionCounts) shared with #804 analytics;
  • the filter returns a set, not a single winner;
  • TestBoundaryNodeStaysInEachHomeRegion pins a node that sits on a region boundary.
• Red/green. At 752c9540 three tests fail with the stated assertion, at 998219f1 the two fallback tests fail, and at fb81de23 everything passes. go vet and gofmt are clean, and go test ./cmd/server passes.
• The recomputer follows bridge_recomputer.go: it prewarms, uses an atomic snapshot and recovers from panics. On the request path this adds two atomic loads and one pass over all nodes on a miss of the 30 s cache. Observers are read before s.mu (#2146).
• Scope. There are no DB writes. Only /api/nodes?region= changes meaning. The "show all" toggle (nodesRegionQueryString) still bypasses it, and "All regions" sends no parameter.

Requested changes

1. Show which rule placed a node. With the cap, a node can now be in no region at all, and from the response alone an operator can't tell that apart from the other outcomes. There are four:

   • zero-hop home regions;
   • fewest-hop regions;
   • no region (more than 3 hops);
   • the observer scan, for nodes newer than the last recompute.

   Only the first one sets home_regions (node_home_regions.go:216); the other three all look like "field absent". #1879 asked that the response say which rule answered. A small region_basis enum, or a fallback_regions array (empty meaning no region), would cover it. If it becomes more than one field, please use a named struct.

2. The recompute cost in the description covers only half of the work.

   • The "~50 ms per 5-min recompute" comes from BenchmarkNodeFewestHopsByRegion.
   • But recomputeNodeHomeRegionSets (:95-111) also runs nodeZeroHopRegionCounts over all of s.packets, and both run under s.mu.RLock.
   • I measured the whole function on 220k adverts × 8 observations plus 1M non-advert transmissions: about 162 ms/op. On the same machine BenchmarkNodeFewestHopsByRegion took 71 ms/op.
   • Please time recomputeNodeHomeRegionSets itself and state that number.

Also

• Please make regionFallbackMaxHops a config value, as you offered. Per AGENTS.md rule 8, note the customizer as a later step.

Notes, not blocking

1. Transport-direct zero-hop adverts are not counted as zero-hop (store.go:3977-3989, from before this PR).
   • For ROUTE_TYPE_TRANSPORT_DIRECT, RawHex[2:4] is the first transport-code byte, not path_len (firmware docs/packet_format.md). The comment that it matches the analytics decoder is not accurate: store.go:9161 uses pbOffset.
   • The fallback still places such a node, because path_json [] counts as 0 hops, but home_regions stays empty.
   • This could be its own PR.
2. pathHopCount("") returns 0 (node_home_regions.go:150), so a legacy observation with NULL path_json counts as a direct hearing.
   • The current ingestor always writes "[]", so this is rare.
   • The fixture relies on it, though: pk_far in node_home_regions_test.go:39 has a relayed raw packet with an empty path_json.
   • Treating "" as unknown and giving pk_far an explicit path would be more accurate.
3. The prewarm covers only the first chunk. It runs after FirstChunkReady. For up to one interval, a node whose zero-hop advert sits in a later chunk can get a partial or empty set. The other recomputers behave the same way, so a recompute when the background load finishes, or a comment, would do.
4. This adds about two map[string]interface{} occurrences: the enrichNodeHomeRegions signature and the OpenAPI entry. They follow the existing node-row pattern, but AGENTS.md asks the count not to grow in touched files.
5. Two tests would help:
   • the observer-scan path for a node newer than the snapshot;
   • a request that lists every region code explicitly (nodes in no region drop out even then).

[anieto]

Review feedback addressed (commits 3dc3cb7..de0fe4c, rebased onto master 2cdb18e). Thanks for the thorough read.

Requested changes

1. Which rule placed a node. Every node row now has region_placement, a named struct (NodeRegionPlacement): basis is zero_hop, fewest_hops, none or observer_scan, plus regions and the fewest hops where they apply. It replaces home_regions. Red d4154ee, green 261178f.

2. Recompute cost. Added BenchmarkRecomputeNodeHomeRegionSets, which times the whole function at your scale (220k adverts × 8 observations plus 1M channel messages). It was ~72 ms here, so your 162 ms looks right for your machine. 3dc3cb7 brings it to ~54 ms:

   • nodeZeroHopRegionCounts walks byPayloadType[ADVERT] instead of every packet (same counts, so Bug/feature: Use actual repeaters region data and not observer based #804 is unchanged).
   • The fewest-hops pass looks up a node's map once per advert instead of once per observation.

   The description now gives the whole-function number.

3. Config. regionFallback.maxHops (default 3) is in config.example.json, and the customizer is noted as a later step. Red 862a451, green 47b763f.

Notes

• Empty path_json is now an unknown hop count, and pk_far has an explicit path (red 16dcd8d, green b8a8c1f).
• The recomputer recomputes once when StartupLoadDone() closes, like recomputeWhenLoaded (red 95f2aca, green de0fe4c).
• No new map[string]interface{}. enrichNodeHomeRegions is gone, and the OpenAPI component uses a small typed openAPISchema, so the touched files are back at the master count.
• Added both tests you suggested: the observer scan for a node newer than the snapshot, and a request listing every region.
• I'll do the transport-direct path_len offset as its own PR, since it changes Bug/feature: Use actual repeaters region data and not observer based #804 analytics too.

go test passes for cmd/server and cmd/ingestor; gofmt and go vet are clean.

[efiten]

Thanks, all three requested changes are in. Approving and merging.

What I checked at de0fe4cd

• Red/green. Each pair fails for the stated reason and then passes: d4154ee/261178fe, 862a451/47b763f1, 16dcd8d/b8a8c1f8 and 95f2aca/de0fe4cd.
• region_placement is on list and detail rows (routes.go:1627, :1834). Nothing in public/, docs/ or tests/ still reads home_regions.
• The byPayloadType[ADVERT] walk gives the same counts as the walk over all packets. Every insertion path appends to s.packets and byPayloadType together (store.go:1047, :1563, :3071, chunked_load.go:598), and eviction removes from both. Only the order differs, and the tally does not depend on order.
• Config. A missing or negative regionFallback.maxHops gives 3.
• map[string]interface{}. The count in each touched file is the same as on master.
• Checks. go vet and gofmt are clean. go test . passes in cmd/server and cmd/ingestor.

For the description, not blocking

1. The benchmark is machine-dependent. BenchmarkRecomputeNodeHomeRegionSets takes 159-171 ms/op here (-benchtime 3x, then 10x -count=2), against the 54 ms in the description, and all of it runs under s.mu.RLock. Please state which machine gave 54 ms, so readers know the read lock can be held about three times longer on slower hardware.
2. Two of the four tests in d4154ee were already green at that commit: TestRegionFilterObserverScanForNodesNewerThanRecompute and TestRegionFilterListingEveryRegionOmitsUnplacedNodes. They are the characterization tests I asked for, so that is fine, but they are not red tests.
3. The deferred transport-direct path_len offset is now tracked in #2225.

Minor, no action needed

• regionNodesCache (30 s TTL) is not cleared when a recompute installs a new snapshot. A filter can therefore show the previous placements for up to 30 s, which is within the existing TTL contract.