Skip to content

data,web: key Reticulum nodes on identity; discover host aspects - #897

Merged
l5yth merged 4 commits into
mainfrom
l5y-data-reticulum-e2e
Aug 28, 2026
Merged

l5yth merged 4 commits into
mainfrom
l5y-data-reticulum-e2e

Conversation

@l5yth

@l5yth l5yth commented Aug 28, 2026 •

Copy link
Copy Markdown
Owner

Ref #888

Findings from a live RNode + rnsd test, where the shipped behaviour from #889/#890/#893 did not survive contact with a real stack.

One node record per identity (RE7, superseding RE1). A peer announcing lxmf.delivery, lxmf.propagation and nomadnetwork.node is one node with three destinations, not three nodes. RE1 had split them to stop a merged row being named from one aspect and roled from another; the destinations table (RE2) already carries the per-aspect name and role, so the split bought nothing and cost the node count its meaning. Verified against RNS: Destination.hash reproduces all three field destination hashes from identity 27716218... alone.

Destinations become a table, and are served. destinations (id, node_id, identity_hash, name, aspect, role, interface, first_heard, last_heard, protocol) replaces the nodes.dest_hash JSON column -- a column and a table modelling one relationship would drift -- and is served by GET /api/destinations. The write is its own forward-only statement outside the upsert's freshness guard, so a stale record's new destination is not dropped, and it still cannot regress last_heard or write through a synthetic placeholder.

A 0-hop announce is always ingested. Transport.inbound adds a hop to every inbound packet and takes it back for a local-client or shared-instance interface, so zero hops can only mean "announced by an app on this machine". Filtering purely on interface name hid the operator's own nodes, because every local destination legitimately reads LocalInterface[rns/default]. RETICULUM_INTERFACES now applies from one hop out; empty (the default) still ingests everything.

Share the operator's RNS stack. RETICULUM_CONFIG_DIR defaults to ~/.reticulum again; the config seeding and the allowlist's fail-open, both of which existed to work around an app-owned directory, are removed.

Derive the host node id the Reticulum way. canonical_node_id keeps the low 32 bits of a hex value -- right for a Meshtastic node num, wrong for a 16-byte identity hash, which it truncates from the opposite end: the ingestor registered !86c39940 while its own peer row read !27716218. Raw identity hashes now map through _reticulum_node_id. With INGESTOR_NODE_ID unset the id is derived from the transport identity rather than warned about.

Log the transmit gate only when it blocks something. The announcement policy check moves after the due check, so the suppression line names the gate when it actually blocks a due announcement instead of once per 60 s loop; the startup "Transmit policy resolved" line already carries that.

@codecov

codecov Bot commented Aug 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@eknuth

eknuth commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Pulled e0b7913 onto my Pi 5 (RNode + rnsd, shared instance, enable_transport = False) and ran it against a live stack. The host-id discovery works, and I like it better than what I did in #896: deriving from the primary announced identity means the ingestor's row is one an operator can actually recognise, instead of a synthetic key. Three things to report.

1. The README documents the approach you rejected

README.md tells operators the id comes from the transport identity, and hands them a command to predict it:

The ingestor derives its own node id from the config dir's transport identity

python -c "import RNS; RNS.Reticulum(configdir='$HOME/.reticulum'); \
  print('!' + RNS.Transport.internal_identity().hash.hex()[:8])"

The code does not do that. _derived_host_node_id() returns _reticulum_node_id(_primary_local_identity()), and your own module docstring is explicit that "the transport identity is deliberately not the node id".

Both on my box, same config dir:

README snippet says:     !1254d60f     (transport identity 1254d60f17f92697...)
code actually derives:   !e492a9cd     (primary local identity e492a9cd6fe8f0e0...)

So the README is describing the !fbf8e338 vs 27716218… bug this PR fixes, and its copy-paste command prints the wrong id. Line 340 (the env table) and line 446 both need it. Happy to push that fix to your branch if it is easier.

2. The PR title and body describe a different design than the head commit

The title says "key Reticulum nodes on destinations" and the body opens with "Key rows on the destination hash again (reverses RN1)". The head commit e0b7913 says "key Reticulum nodes on identity; discover host aspects", and the docstring still describes identity keying with aspects collapsing onto one row. Looks like the body did not get updated after the approach changed. Worth a pass before merge, since RN1 and the SPEC entries key off which way this went.

3. Changing the id rule strands the previous ingestor row

Not a defect, but it has now bitten twice on the same host. /api/ingestors on my instance:

!edabc002  reticulum    <- what I had pinned via INGESTOR_NODE_ID
!887391ba  reticulum    <- #896's generated identity
!e492a9cd  reticulum    <- this PR's primary local identity

Each id-scheme change leaves the old row behind, unheartbeated, until it ages out. I had drafted an upgrade caveat for #896 but did not push it, since you had this in flight and its premise changed. If you want, I will send a short note for the README and CHANGELOG against whatever lands: leaving INGESTOR_NODE_ID set is inert, removing or changing it moves the ingestor's node id, and the old row ages out.

Otherwise clean

  • The RN10 line still earns its keep. My reticulum unit shares the meshtastic .env via EnvironmentFile, so it correctly reports the inherited TBeam CONNECTION as inapplicable.
  • Config dir back to ~/.reticulum behaves as described, and it attaches to my running rnsd as a shared instance.
  • No warnings or errors in the run, including against a web tier several commits behind, so the ingestor half degrades quietly.
  • 1426 passed locally, black clean.

One small thing: at connect the startup line reads node_id=None, because the path table has nothing local yet. It resolves on a later daemon loop and registered fine, but an operator reading the startup log sees None and no indication that a retry is coming. A word in that log line, or logging the id once it resolves, would save someone a false alarm.

@l5yth l5yth changed the title data,web: key Reticulum nodes on destinations; fix e2e field findings (#888) data,web: key Reticulum nodes on identity; discover host aspects Aug 28, 2026
@l5yth

l5yth commented Aug 28, 2026

Copy link
Copy Markdown
Owner Author

Good findings, I fixed all three.

@l5yth
l5yth merged commit 96e0dbf into main Aug 28, 2026
15 checks passed
@l5yth
l5yth deleted the l5y-data-reticulum-e2e branch August 28, 2026 19:11
l5yth pushed a commit that referenced this pull request Aug 28, 2026
.env.example still described the #896 model: the ingestor owning an RNS
identity and deriving its !xxxxxxxx from "the transport identity already
in RETICULUM_CONFIG_DIR". #897 shipped the opposite. RE8 excludes the
transport identity from the pick, because RNS generates it as an
independent keypair that matches none of the operator's announced
destinations; the id is the discovered primary identity, the one
announcing the most destinations on the machine. README says this
correctly and .env.example contradicted it, pointing operators at the
hash rnstatus prints as "Transport Instance".

The strands-the-old-row caveat #897 added to README and CHANGELOG is
carried over too, since .env.example is where an operator is standing
when they set the variable that moves the id.

Two guards in the deployment-surface class, both negative-tested against
main, where they correctly fail.

pytest 1443 passed; black clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants