Skip to content

fix(fleet): let a deliberate re-claim outrank an older signed eviction - #199

Merged
mrjeeves merged 1 commit into
mainfrom
claude/kvm-wifi-interface-auth-l96vcx
Jul 30, 2026
Merged

fix(fleet): let a deliberate re-claim outrank an older signed eviction#199
mrjeeves merged 1 commit into
mainfrom
claude/kvm-wifi-interface-auth-l96vcx

Conversation

@mrjeeves

Copy link
Copy Markdown
Owner

The symptom

Unclaiming a device and re-claiming it left it permanently unreachable, in a shape that hides the cause well. The device looks healthy from its own side — owner, fleet key and attachment all intact in its local record — and keeps serving its own LAN UI, while every peer refuses it:

WARN auth: peer denied us: dqbx4v… (reason: Some("evicted"))
WARN peer: dqbx4v…mmoa dropped (Denied)

Found on a NanoKVM that had been reset from its own web UI and then unclaimed from the app, which is fleet_evict: a Release to the device followed by GovernanceProposeEvict. Re-claiming it appeared to work and changed nothing.

Why a re-claim could never win

ensure_fleet_network prunes any locally listed device the signed governance has removed, so one owner's admit loop can't resurrect a device another owner evicted. That guard is right, but it decided on set membership alone:

for member in self.ownership.fleet_member_ids() {
    if signed_evicted.contains(pubkey_part(&member)) {
        let _ = self.ownership.kick_member(&member);   // ← the re-claim, undone

which cannot tell that case apart from a device this owner has deliberately re-claimed since the eviction. The prune runs immediately before the admit loop, so the re-claimed device was dropped first, every time, and the superseding RoleGrant was never authored. Membership itself converges last-writer-wins on the entry stamp, and myownmesh-core already honours exactly this — re_admitting_an_evicted_member_supersedes_the_tombstone. The grant simply never got written, which made unclaim → re-claim a one-way door.

The fix

Compare stamps instead of set membership.

  • OwnedMember gains claimed_at, recorded when the owner first adds the member. First add only: a later upsert is a label refresh driven by roster gossip, and moving the stamp there would drift a device past every tombstone just by being talked about.
  • signed_evicted returns pubkey → removal stamp rather than a bare set, dating each removal from the member log the daemon already returns alongside the verdict. evicted stays the authoritative, already-verified answer to whether a device is removed; the log is read only for when, so nothing here re-decides membership or trusts an unverified entry.
  • The prune keeps a device whose claim is strictly newer than its eviction, and logs that it is doing so. The admit loop then authors the superseding grant.

Both unknowns fail toward the old behaviour rather than toward resurrection:

Unknown Sorts as Effect
Removal the member log can't date u64::MAX outranks every claim — still pruned
Record written before claimed_at existed None older than any eviction — still pruned

So an upgrade never silently un-prunes anything. A device already stranded by this bug recovers by being re-claimed once, which stamps it.

The load-bearing detail

eviction_stamps parses state.member_log, and myownmesh-core serialises Transition with the variant nested, not flattened:

{"at":1700000000,"variant":{"kind":"evict","target":"devicekey"},"signers":["ownerkey"],"signatures":["sig"]}

I verified that against the real type rather than inferring it, because the failure mode is silent: if that shape changed, eviction_stamps would date nothing, every removal would sort at u64::MAX, and the prune would quietly revert to unconditional with no error anywhere. The tests pin the shape as much as the logic.

Testing

  • cargo test — 200 node tests + protocol suite, all passing
  • cargo clippy --all-targets -- -D warnings — clean on both crates

New coverage:

  • a removal is dated from the latest member-log entry targeting it, and another device's entries don't leak into that stamp
  • an undatable eviction outranks every claim
  • display-suffixed and bare-pubkey forms collapse, so the lookup can't miss and silently prune anyway
  • the first add is stamped and a label refresh never moves it
  • a record without claimed_at still deserialises

Related

The device this was found on is covered by the companion PRs on NanoKVM, NanoKVM-Pro and CECSupport (same branch name), though this bug is independent of them — it strands any fleet member, not just a KVM.

Worth noting separately: CECSupport's Unclaim button calls fleet_kick, and its confirm text says the device "offers itself for setup again". With this fix that becomes true. Before it, the button was the one-way door.


Generated by Claude Code

Unclaiming a device and re-claiming it left it permanently unreachable. The
device looked healthy — owner, fleet key and attachment all intact locally —
while every peer refused it with `evicted`, so it served its own LAN UI and was
invisible to the fleet.

ensure_fleet_network prunes any locally listed device the signed governance has
removed, so one owner's admit loop can't resurrect a device another owner
evicted. It decided on set membership alone, which cannot tell that case apart
from a device this owner has deliberately re-claimed SINCE the eviction. The
prune runs before the admit loop, so the re-claimed device was dropped first,
every time, and the superseding RoleGrant was never authored. Membership itself
converges last-writer-wins on the entry stamp and myownmesh-core already honours
exactly this (`re_admitting_an_evicted_member_supersedes_the_tombstone`) — the
grant just never got written, which made unclaim → re-claim a one-way door.

So compare stamps. OwnedMember records claimed_at when the owner first adds it —
first add only, since a later upsert is a label refresh driven by roster gossip
and would otherwise push a device past every tombstone just by being talked
about. signed_evicted returns pubkey → removal stamp instead of a bare set,
dating each removal from the member log the daemon already returns alongside the
verdict: `evicted` stays the authoritative verified answer to WHETHER a device is
removed, and the log is read only for WHEN, so nothing here re-decides
membership or trusts an unverified entry.

Both unknowns fail toward the old behaviour rather than toward resurrection:
a removal the log can't date sorts at u64::MAX, and a record written before
claimed_at existed sorts as older than any eviction. A device stranded by this
bug therefore recovers by being re-claimed once, which stamps it.

The member-log JSON shape is load-bearing and pinned by test: myownmesh-core
serialises Transition with the variant NESTED, not flattened. If that changed,
eviction_stamps would silently date nothing, every removal would sort at
u64::MAX and the prune would quietly go back to being unconditional.

cargo test (200 node + protocol) and clippy -D warnings pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vfwNJWeimvaQ9FGHqjM7i

Copy link
Copy Markdown
Owner Author

Node · windows-latest is red on a timing flake, not on this diff — and I can't re-run it myself.

---- os_perf::tests::precise_sleep_holds_sub_quantum_accuracy stdout ----
panicked at src\os_perf.rs:489:9:
overshoot 24.6267ms looks quantized, not precise
test result: FAILED. 220 passed; 1 failed

Why it isn't mine:

  • The diff touches crates/allmystuff-protocol/src/app.rs, node/src/mesh.rs and node/src/ownership.rs only. There's no path from fleet-eviction stamps to precise_sleep.
  • Node · ubuntu-latest and Node · macos-latest ran the same cargo test --lib and passed. The failing test is Windows-gated (TimerResolutionGuard, GetThreadSelectedCpuSets), so Windows is the only runner that executes it.
  • It's a wall-clock assertion whose own comment says the 2 ms bound is "deliberately lenient … so a loaded CI box can't flake it". A 24.6 ms overshoot is the scheduler not returning promptly on a busy runner — the thing being measured is the runner, not the code.

Everything else is green: 12 of 13 completed checks passed, with GUI backend · windows-latest still running at the time of writing.

I don't have permission to re-run failed jobs (403 Resource not accessible by integration), so this needs either a manual re-run or the next push to clear it.

One observation, deliberately not acted on here: raising the bound past 24.6 ms would defeat the test's purpose, since what it exists to catch is the ~15.6 ms quantum disaster — a threshold above that detects nothing. Making it robust would mean retries or a load-aware skip, which is its own change and doesn't belong in a fleet-governance PR.


Generated by Claude Code

@mrjeeves
mrjeeves merged commit e393abf into main Jul 30, 2026
24 of 25 checks passed
@mrjeeves
mrjeeves deleted the claude/kvm-wifi-interface-auth-l96vcx branch July 30, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants