fix(route): terminal cert-mismatch eviction ignores proof-of-life — kill the 0↔1 route flap (self-heal #2) - #1300
Open
joelteply wants to merge 1 commit into
Open
Conversation
…ival — kill the 0↔1 route flap (self-heal #2) Glass-boxed live on the M5↔bigmama grid 2026-07-28: transport health oscillated 0↔1 healthy route every ~20s. Root cause: record_terminal_failure (the cert-identity-mismatch path, #1296) evicts the orphan endpoint on the first strike but left BOTH revival signals live — a fresher advert OR proof-of-life. Proof-of-life revival is wrong for a DETERMINISTIC failure: the peer being alive changes nothing about a cert that mismatches forever on the SAME stored endpoint. So every presence beacon (last_seen_ms advances each registry import) revived the corpse → re-dial → re-mismatch → re-kill → next beacon revives again. That IS the flap ('cert is for 71dcc50f, expected 2f0aed7f' cycling). Fix: mark terminal entries `terminal: true` on QuarantineEntry; gate()'s alive_since_failure is now `!entry.terminal && proof_of_life_ms > failed_at_ms`. A terminal eviction revives SOLELY on a genuinely fresher endpoint ADVERTISEMENT (new addr/port/cert) — the only thing that can actually clear a deterministic cert mismatch. Non-terminal (transient) failures keep both signals, preserving the #240 live-peer heal untouched. Tests: renamed the terminal test to _revives_only_on_fresher_advert (drops the now- wrong proof-of-life-revives assertion); added terminal_eviction_is_not_revived_by_ proof_of_life_no_flap pinning the flap regression (advancing PoL stays DEAD every tick; only a fresher advert lifts it). #240 non-terminal PoL-heal test stays green. 12 dial_quarantine + 44 route tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The flap (glass-boxed live, M5↔bigmama, 2026-07-28)
transport healthoscillated 0↔1 healthy route every ~20s. Dial errors alternatedcert is for 71dcc50f, expected 2f0aed7f— the relay-target cert identity, mismatching forever on the same stored endpoint.Root cause
record_terminal_failure(the cert-identity-mismatch path from #1296) correctly evicts the orphan on the first strike, but left both revival signals live: a fresher advert OR proof-of-life. Proof-of-life revival is wrong for a deterministic failure — the peer being alive changes nothing about a cert that mismatches forever on the same endpoint. So every presence beacon (last_seen_msadvances on each registry import) revived the corpse → re-dial → re-mismatch → re-kill → next beacon revives again. That is the flap.Fix
QuarantineEntrygainsterminal: bool;record_terminal_failuresets it true,record_failurefalse.gate():alive_since_failure = !entry.terminal && proof_of_life_ms > failed_at_ms. A terminal eviction now revives solely on a genuinely fresher endpoint advertisement (new addr/port/cert) — the only thing that can actually clear a deterministic cert mismatch.Tests
terminal_failure_..._still_revives→..._revives_only_on_fresher_advert(drops the now-wrong PoL-revives assertion).terminal_eviction_is_not_revived_by_proof_of_life_no_flap: advancing proof-of-life stays DEAD every tick; only a fresher advert lifts it.Deployed + verified on M5: daemon on this build, flap gone (route settles instead of oscillating).
Note: this fixes the software flap. The underlying p2p gap (M5 and bigmama on different non-routing subnets → rendezvous-only) is a separate topology item flagged to Joel (#1298 / one-tailnet).
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo