TUI windows ignore /nodes-discovered peers: dwight (and any non-bootstrap node) invisible across most tabs

[vm-wylbur]

Symptom

Live nodes that are not in config/tfcs-tui.toml bootstrap_peers are silently absent from most TUI windows, even when the cluster knows about them. Concrete current example: dwight (IJLA-NOLa anchor, live and heartbeating, per server-documentation/machines/dwight.yaml) is not in bootstrap_peers and therefore does not appear in any per-node widget.

Root cause

Every widget that is keyed on a hostname is constructed from self._peer_hosts (sourced from bootstrap_peers in TOML) rather than from the live node list returned by /nodes. There is no auto-discovery; the static bootstrap list is treated as the cluster.

Source: src/tfcs_tui/app.py:120 (self._peer_hosts = peer_hosts) is then passed to every widget that needs node axes (OrgNodeTable, TrafficHeatmap, LatencyHeatmap, HeartbeatMatrix) and used to build _get_ntx_hosts() fallbacks.

Secondary cause for traffic/latency/heartbeats: load_tailscale_ip_map() in src/tfcs_tui/data.py:348 only learns short → FQDN mappings for hosts in peer_hosts. When a peer reports /traffic containing a non-bootstrap IP (e.g. dwight's tailscale IP), the IP gets mapped to the bare short name \"dwight\" instead of \"dwight.hrdag.net\", and the heatmap matrices key on FQDNs.

Per-window impact

Window	Widget	How nodes are sourced	dwight present?
1 Replication	ClusterOverview	counts len(node_status) from /nodes	counted but not displayed
1 Replication	ReplicationVelocity (by_source)	server-supplied velocity.by_source	yes, if dwight is a source
2 Nodes	NodesTable	iterates store.statuses (polled peers only)	missing row
2 Nodes	SourceUtilization	aggregates over statuses[*].claims[*].source	yes if other nodes pull from dwight
2 Nodes	TransfersTable	dest = polled peer, source = claim[\"source\"]	as source yes; as destination no
3 Orgs	OrgsTable	totals only	n/a
3 Orgs	OrgNodeTable	columns = self._peer_hosts	no column → dwight's copies invisible
4 Traffic	TrafficHeatmap	axes = self._peer_hosts	missing row + column
5 Latency	LatencyHeatmap	axes = self._peer_hosts	missing row + column
6 Heartbeats	HeartbeatMatrix	axes = self._peer_hosts; observers from peer_hosts only	missing row + column
7 Ingest	IngestOverview / IngestNodeTable / IngestPipeline	_get_ntx_hosts() = config ntx_hosts (5 hosts, no dwight) or fallback to statuses[node_class=='active'] (bootstrap-only)	missing

Fix plan

The fix is the auto-discovery work already listed in TODO.md, but extended to cover every affected widget. We are restructuring TODO.md into a per-widget checklist so progress is visible. Items:

• Return raw nodes list from poll_cluster return tuple
• _merge_discovered_peers() on TfcsDashboard, called from _do_poll and do_full_refresh
• Hybrid model: config bootstrap peers never evicted; discovered peers can be added/evicted
• Track consecutive failures per host; back off after 3, evict after 10
• Re-run load_tailscale_ip_map after merging new peers
• Make BaseHeatmap.node_names updatable (axes rebuild every render, so just update the list) → fixes Traffic / Latency / Heartbeats
• Rebuild OrgNodeTable columns on node list change (DataTable columns are structural)
• Drive NodesTable from node_status ∪ statuses keys, not just statuses
• Drive ntx host list from discovered nodes with node_class == 'active', not bootstrap
• Remove explicit ntx_hosts from config (let _get_ntx_hosts() auto-discover)
• Fix load_tailscale_ip_map: build short → FQDN from discovered FQDNs too, not just bootstrap

Success condition

This issue is resolved when:

• dwight (and any other non-bootstrap node that the cluster knows about via /nodes) renders correctly in every affected window listed above.
• Verified by running uv run tfcs-tui -c config/tfcs-tui.toml with dwight removed from bootstrap_peers and observing: dwight row in NodesTable, dwight column in OrgNodeTable, dwight row+column in Traffic / Latency / Heartbeats heatmaps, and dwight in Ingest tab if it runs ntx.
• A node taken offline is evicted from auto-discovered windows after the configured backoff threshold; bootstrap peers stay listed regardless.

Related

• Cross-repo: hrdag/tfcs#110 (API contract for heartbeat_age_seconds: null on copy-only /nodes rows) — independent but in the same surface area.

---

📡 cc-TFC