Skip to content

warn when the cluster peer mesh is incomplete and fix the seed-node docs - #139

Merged
fabracht merged 2 commits into
mainfrom
cluster-mesh-warning
Sep 20, 2026
Merged

fabracht merged 2 commits into
mainfrom
cluster-mesh-warning

Conversation

@fabracht

@fabracht fabracht commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The README told users to start each node with a single seed peer, which produces a cluster that fails silently.

  • inter-node messages go over direct peer connections and are never relayed, so a cluster where node 2 and node 3 both peer only with node 1 leaves those two unable to reach each other
  • on such a cluster, cross-node publishes between them are dropped and a read on one cannot see data whose partition primary is the other, with no error, warning or metric
  • README and docs/distributed-design.md now peer every node with all the others and state the requirement next to --peers
  • each node now compares the members it knows about (partition map plus voter gossip) against the peers it can actually reach, and warns once a minute naming every unreachable node
  • mqdb cluster status shows an UNLINKED line, and the status payload gained known_members and unlinked_nodes

This makes the failure visible, not impossible: a node still cannot dial a peer it was not configured with, because nothing propagates node addresses. Detection also covers a node that was never linked, not a link that dies in flight, since a peer is never removed from the connection map when its stream fails. Both are tracked in #140.

Test plan

  • cargo make clippy passes with zero warnings
  • cargo make test: 24 suites, 1200 passed, 0 failed
  • unit tests: unlinked nodes names only members missing a link, and is empty with no members or no direct-peer concept
  • live 3-node hub-and-spoke: node 2 reports UNLINKED: [3], node 3 reports UNLINKED: [2], hub silent, both spokes log the warning with unlinked_nodes
  • live 3-node full mesh: no warning in status or logs on any node
  • regression on full mesh: dev test --all 19/19, dev test --presence 7/7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant