Skip to content

daemon: address watcher follow-ups — own retry backoff, Stop not held by a redial, no duplicate heartbeat recovery - #519

Merged
TeoSlayer merged 6 commits into
mainfrom
fix/addrwatch-followups
Oct 8, 2026
Merged

TeoSlayer merged 6 commits into
mainfrom
fix/addrwatch-followups

Conversation

@TeoSlayer

@TeoSlayer TeoSlayer commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Three follow-ups to the address watcher from #495, found in its review and confirmed by probes on main.

(a) Registry retries no longer push back the next address move

Retries shared the local-path backoff, so retries at +10 s, +30 s and +70 s left the next real move waiting 80 s, up to the 2-minute cap. They now have their own backoff.

Test: TestAddrWatchRegistryRetriesDoNotHoldBackTheNextMove fires at [1 11 31 71 81], against [… 151] on main.

(b) Stop no longer waits on a registry redial or a heartbeat backoff

The registry dial backoff and the heartbeat's re-register sleep ignored Stop. With a refused registry, a reconnect outlived Stop by 47 s, and Stop logged "leaked". A new sleepOrStop helper returns early on stop, and every attempt checks stopping().

Test: TestForceReconnectRegistryGivesWayToStop.

(c) The heartbeat no longer repeats a recovery that just ran

  • The problem: the heartbeat's serialised reconnect and re-register never skipped a duplicate and never recorded success. When its call on the old connection timed out during a slow recovery, it replaced the recovery's fresh connection and ran a second full re-registration: 2 registers and 4 report_trust.
  • The fix:
    • Shared helpers reestablishedRecently and noteReestablished.
    • The reconnect skips if the connection was already replaced.
    • The re-register skips within 5 s of a full run and records its own.
    • The reconnect still has the heartbeat re-register when the recovery that replaced the connection did not register (its own re-registration failed): nothing registered after the heartbeat's call began.
    • A full run counts as full only if every visibility, hostname and trust write succeeded. Otherwise it counts as endpoint-only, so the heartbeat's full re-registration is not skipped.
    • RegisterWithBeacon in reestablishTransport now runs before the "done recently" check, so the beacon registration happens even when the registry half is skipped. An endpoint-only re-registration does not register with the beacon, and the rx watchdog's soft recovery counts on the beacon's reply as inbound traffic.
  • Tests:
    • TestHeartbeatDoesNotRepeatARecovery (1 register, connection kept; both orders).
    • TestHeartbeatReregistersWhenTheRecoveryDidNot.
    • TestPartialRestoreDoesNotCountAsFull (the heartbeat re-syncs both trust pairs after a recovery whose trust writes failed).

Each test fails on main. Results:

  • go test -race ./pkg/daemon passes;
  • go test -race -run 'Addr|Recover' ./tests -count=2 passes.

AdvertiseEndpoint is untouched.

🤖 Generated with Claude Code

Teo Calin and others added 6 commits October 8, 2026 12:43
The address watcher's registry-only retries shared the local-address
backoff, and each one bumped its streak. After a move the registry did
not take, the three retries (+10 s, +30 s, +70 s) left the streak at 4,
so the next real move waited 80 s after the last retry instead of 10 s.

The retries now have a backoff of their own, on the local-address
schedule and started over by every move, so they keep their +10/+30/+70
spacing and leave the move's cooldown alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The registry dial retries slept through Stop. A recovery that was
reconnecting to a registry that refused connections kept dialling for
its whole backoff, ten attempts over 47.5 s: Stop gave up on it after
its 5 s wait and logged a leaked goroutine. The heartbeat's sleep
before a re-registration (up to 30 s) did the same.

Both backoffs now end when Stop is called (sleepOrStop, which the
address watcher's follow-up already used as a closure), and each dial
attempt checks for Stop first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The heartbeat's reconnect and re-registration took reestablishMu, but
never skipped work a recovery had just done and never recorded their
own. When the heartbeat's call timed out (8 s) on the connection a
resume recovery was replacing, it waited for the recovery, replaced
the fresh connection again and re-registered in full a second time:
two registers and two ReportTrust per trusted peer.

- reestablishedRecently and noteReestablished are now shared by
  reestablishTransport and the heartbeat.
- The heartbeat's reconnect takes the connection its call timed out on
  and does nothing if that has already been replaced; the heartbeat
  then does not escalate to a re-registration.
- The heartbeat's re-registration is skipped when a full one the
  registry accepted finished under 5 s ago (wall clock), and records
  its own, so a recovery right after it skips its registry half.
- reestablishTransport registers with the beacon before deciding to
  skip: the recent run may now be the heartbeat's, which does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…registration

Two cases where the heartbeat's re-registration was skipped although
nothing had restored the node:

- A recovery replaced the connection the heartbeat's call waited on,
  but its own re-registration failed. The heartbeat took the replaced
  connection as a sign the recovery had re-registered and did not
  escalate. It now re-registers unless something registered after its
  call began.
- A full re-registration whose visibility, hostname or trust writes
  partly failed was recorded as full, so a heartbeat re-registration
  within 5 s skipped re-syncing them. It now counts as endpoint-only.
  restoreRegistryState reports whether every write succeeded, and its
  log no longer says trust pairs were re-synced when some failed.

Also corrects the comment on the beacon registration: the heartbeat's
full re-registration does register with the beacon; an endpoint-only
one does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TeoSlayer
TeoSlayer force-pushed the fix/addrwatch-followups branch from cf2ee6b to d8677ca Compare October 8, 2026 09:46
@TeoSlayer
TeoSlayer merged commit d27a368 into main Oct 8, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant