fix(fleet): let a deliberate re-claim outrank an older signed eviction - #199
Conversation
Unclaiming a device and re-claiming it left it permanently unreachable. The device looked healthy — owner, fleet key and attachment all intact locally — while every peer refused it with `evicted`, so it served its own LAN UI and was invisible to the fleet. ensure_fleet_network prunes any locally listed device the signed governance has removed, so one owner's admit loop can't resurrect a device another owner evicted. It decided on set membership alone, which cannot tell that case apart from a device this owner has deliberately re-claimed SINCE the eviction. The prune runs before the admit loop, so the re-claimed device was dropped first, every time, and the superseding RoleGrant was never authored. Membership itself converges last-writer-wins on the entry stamp and myownmesh-core already honours exactly this (`re_admitting_an_evicted_member_supersedes_the_tombstone`) — the grant just never got written, which made unclaim → re-claim a one-way door. So compare stamps. OwnedMember records claimed_at when the owner first adds it — first add only, since a later upsert is a label refresh driven by roster gossip and would otherwise push a device past every tombstone just by being talked about. signed_evicted returns pubkey → removal stamp instead of a bare set, dating each removal from the member log the daemon already returns alongside the verdict: `evicted` stays the authoritative verified answer to WHETHER a device is removed, and the log is read only for WHEN, so nothing here re-decides membership or trusts an unverified entry. Both unknowns fail toward the old behaviour rather than toward resurrection: a removal the log can't date sorts at u64::MAX, and a record written before claimed_at existed sorts as older than any eviction. A device stranded by this bug therefore recovers by being re-claimed once, which stamps it. The member-log JSON shape is load-bearing and pinned by test: myownmesh-core serialises Transition with the variant NESTED, not flattened. If that changed, eviction_stamps would silently date nothing, every removal would sort at u64::MAX and the prune would quietly go back to being unconditional. cargo test (200 node + protocol) and clippy -D warnings pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012vfwNJWeimvaQ9FGHqjM7i
|
Why it isn't mine:
Everything else is green: 12 of 13 completed checks passed, with I don't have permission to re-run failed jobs ( One observation, deliberately not acted on here: raising the bound past 24.6 ms would defeat the test's purpose, since what it exists to catch is the ~15.6 ms quantum disaster — a threshold above that detects nothing. Making it robust would mean retries or a load-aware skip, which is its own change and doesn't belong in a fleet-governance PR. Generated by Claude Code |
The symptom
Unclaiming a device and re-claiming it left it permanently unreachable, in a shape that hides the cause well. The device looks healthy from its own side — owner, fleet key and attachment all intact in its local record — and keeps serving its own LAN UI, while every peer refuses it:
Found on a NanoKVM that had been reset from its own web UI and then unclaimed from the app, which is
fleet_evict: aReleaseto the device followed byGovernanceProposeEvict. Re-claiming it appeared to work and changed nothing.Why a re-claim could never win
ensure_fleet_networkprunes any locally listed device the signed governance has removed, so one owner's admit loop can't resurrect a device another owner evicted. That guard is right, but it decided on set membership alone:which cannot tell that case apart from a device this owner has deliberately re-claimed since the eviction. The prune runs immediately before the admit loop, so the re-claimed device was dropped first, every time, and the superseding
RoleGrantwas never authored. Membership itself converges last-writer-wins on the entry stamp, and myownmesh-core already honours exactly this —re_admitting_an_evicted_member_supersedes_the_tombstone. The grant simply never got written, which made unclaim → re-claim a one-way door.The fix
Compare stamps instead of set membership.
OwnedMembergainsclaimed_at, recorded when the owner first adds the member. First add only: a later upsert is a label refresh driven by roster gossip, and moving the stamp there would drift a device past every tombstone just by being talked about.signed_evictedreturnspubkey → removal stamprather than a bare set, dating each removal from the member log the daemon already returns alongside the verdict.evictedstays the authoritative, already-verified answer to whether a device is removed; the log is read only for when, so nothing here re-decides membership or trusts an unverified entry.Both unknowns fail toward the old behaviour rather than toward resurrection:
u64::MAXclaimed_atexistedNoneSo an upgrade never silently un-prunes anything. A device already stranded by this bug recovers by being re-claimed once, which stamps it.
The load-bearing detail
eviction_stampsparsesstate.member_log, and myownmesh-core serialisesTransitionwith the variant nested, not flattened:{"at":1700000000,"variant":{"kind":"evict","target":"devicekey"},"signers":["ownerkey"],"signatures":["sig"]}I verified that against the real type rather than inferring it, because the failure mode is silent: if that shape changed,
eviction_stampswould date nothing, every removal would sort atu64::MAX, and the prune would quietly revert to unconditional with no error anywhere. The tests pin the shape as much as the logic.Testing
cargo test— 200 node tests + protocol suite, all passingcargo clippy --all-targets -- -D warnings— clean on both cratesNew coverage:
claimed_atstill deserialisesRelated
The device this was found on is covered by the companion PRs on NanoKVM, NanoKVM-Pro and CECSupport (same branch name), though this bug is independent of them — it strands any fleet member, not just a KVM.
Worth noting separately: CECSupport's Unclaim button calls
fleet_kick, and its confirm text says the device "offers itself for setup again". With this fix that becomes true. Before it, the button was the one-way door.Generated by Claude Code