Publish the data archive under node_ref, not node_id

[Babissimo]

/api/data/archive was the one public route still handing out the private node_id: in every listed key, in its filter, in the download path and in the downloaded body. The Data Explorer on app.retina.fm showed those ids as its file tree. Its node picker is built from /api/radar/nodes, which is keyed on node_ref, so on prod every node appeared twice (16 known nodes, 32 rows on the timeline).

What changes

The store stays keyed on node_id (re-keying it is phase 2 of the archive programme plan in claude-shared). The route translates at the publication boundary, as every other public feed does:

	Before	After
Listed key	…/node_id=ret…/part-….parquet	…/node_ref=nde…/part-….parquet (bare <ref> for the legacy layout)
Filter	?node_id=	?node_ref=
Download	stored key	published key, resolved back to the stored one
Body	"node_id": "ret…"	"node_ref": "nde…"
Node with no published identity	listed	not listed

• Only published keys download. A stored node_id= key now 404s, so the route no longer confirms a guessed id.
• A filter naming a private node's ref gets the same answer as an unknown ref, total included.
• The Data Explorer sends node_ref, reads node_ref from the preview, and its search box says "node ref".
• The admin Storage page reads the same listing. It maps refs back to ids through the admin-only node-refs map, so its file table matches the per-node table beside it.

A second commit fixes the reverse lookup for mirrored nodes. These are nodes relayed from another environment, with no registry row here, whose ref arrives with their detections (#373). public_identity published them under that ref, but id_for_identity could not resolve it back. That left their archive files listed but impossible to download or filter, and left /api/radar/analytics/{node_ref} and the custody routes unable to answer for them.

#610 (the owner pages: no Node ID column, and names that fall back to the ref rather than the id) is merged into this branch as its third commit, so both ship in one deploy.

Breaking for scripts: anything holding an old node_id= key or passing ?node_id= gets 404s or an unscoped listing. The route is public and appears in the API reference.

Verification

• New route tests in test_publication.py cover listing, filtering, both key layouts, download, stored-key refusal and private nodes. All of them fail against the old route.
• Full backend suite (-m "not external" -n 4 --dist worksteal): 5078 passed. Dashboard: 1394 passed (both on the combined head, rebased on current main). Typecheck, eslint and pre-commit are clean.
• Tested locally against the real app on a seeded SQLite DB, with a public node, a private node, a synthetic node and one with no registry row. The listing showed only the ref and the synthetic id. A stored key, a private node's key and the id as a filter each got 404 or an empty listing. In the browser, the Data Explorer file tree, the preview's "Reported node" and a ?node= scoped listing all used the ref, and no ret… id appeared on the page.

🤖 Generated with Claude Code

[Babissimo]

1. Fixed in the amended commit: the download now names the node from the key it has already resolved, and falls back to the body's own field only for a key with no node segment. test_an_empty_file_is_named_by_its_key covers the zero-row case.
2. Left as it is. _key_node_id reads stored keys and mirrors storage._partition_value. The two new helpers deliberately accept different spellings: _published_key rewrites only node_id=, and _stored_key accepts only node_ref=, so a stored key is never taken as a published one. A shared parser would have to carry that asymmetry as a parameter. Phase 2 of the archive plan replaces this layout anyway.

[Babissimo]

• StoragePage node segment: fixed in the archive commit. The Storage page now reads the segment before the filename, as the route and the Data Explorer do, and its test uses a key with an extra leading segment, which fails against parts[3].
• _mirrored_id scan: accepted. It runs only when the registry map has no entry for a ref, and it walks the connected nodes (tens) once, which costs less than serialising the response it guards.
• Nodes with neither a name nor a ref: as intended. Such a node is on no public surface, and the console doesn't fall back to the private id.

[Babissimo]

Both fixed, each folded into the commit it belongs to:

1. _mirrored_id now snapshots connected_nodes under connected_nodes_lock and iterates outside it, as node_sites.py does. No caller of id_for_identity holds that lock.
2. A key with no node segment now checks the node its body names against the private set before responding, and 404s if the node is private. test_a_file_with_no_node_in_its_key_is_withheld_if_its_body_names_a_private_node covers both outcomes and fails without the check.

[claude]

Claude finished @Babissimo's task in 3m 49s —— View job

---

Code Review: PR #609 — Publish the data archive under node_ref, not node_id

• Gather context (PR body, prior review thread, CLAUDE.md)
• Review the diff against origin/main (all 3 commits: cfd26c3, 4a7a436, 7431f57)
• Post review findings

Summary

This re-review checked the current HEAD against the two issues raised in the prior automated review round (unlocked _mirrored_id scan, and the download route's body-fallback path skipping the privacy check). Both are already resolved in the code currently on the branch:

_mirrored_id locking — backend/services/node_refs.py:219-220 takes the snapshot under state.connected_nodes_lock before iterating:

with state.connected_nodes_lock:
    snapshot = list(state.connected_nodes.items())
for node_id, known in snapshot:
    ...

This matches the snapshot-under-lock pattern used elsewhere (node_sites.py, analytics_refresh.py, frame_processor.py). No unlocked iteration over connected_nodes remains.

Download route's body-fallback privacy check — backend/routes/archive.py:131-140 checks the resolved node_id (key segment or body fallback) against private before building the response:

node_id = stored[1] or body_id
if node_id in private:
    raise HTTPException(status_code=404, detail="Archive file not found")
data["node_ref"] = public_identity(node_id)

and this exact path is covered by test_a_file_with_no_node_in_its_key_is_withheld_if_its_body_names_a_private_node in test_publication.py, which asserts a 404 for a path-less key whose body names a private node. The route's stated invariant ("no key, parameter or body it hands out names the private id") holds for this case.

Rest of the diff

Went through the remaining changed surface (auth.py's my_nodes/public_name wiring, the OverviewPage/OnboardingPage/TunnelLinkPage renames, StoragePage's node-segment parsing and ref→id mapping, and the mechanical node_id→node_ref rename across the Data Explorer) — nothing new stood out. The node_ids closure variable used in public_name(cfg.get("name"), ref, node_ids) (auth.py:427) resolves correctly to the function-scoped list from get_user_nodes, and ref is correctly scoped in each of OverviewPage's three render blocks that use it.

Test coverage for this diff (test_publication.py, test_node_refs.py, test_auth_routes.py) exercises listing, filtering, both key layouts, download, stored-key refusal, private nodes, mirrored nodes in both directions, and the name-fallback behavior — consistent with what the PR description claims.

Confirmed clean

No outstanding issues found. Both concerns from the previous review round are already fixed and tested on this branch.

Branch: archive-publishes-node-ref