rendezvous: stop holding the global write lock across the whole reap

[TeoSlayer]

Problem

reapStaleNodes (runs every 10s) took the global write lock and held it while:

• building a ~200k-entry id slice from s.nodes,
• sorting it (O(n log n)),
• emitting slog.Info + s.audit lines,

…then kept holding it for the whole delete pass.

The mutex profile shows this as the #3 contention site (reapStaleNodes.deferwrap1, ~160,000 s cumulative wait — low count, huge delay = long holds), stalling every concurrent heartbeat/register.

Fix

• Snapshot node ids under RLock (cheap) and sort outside the lock.
• Scan one chunk under RLock, collecting stale candidates without mutating or logging.
• Delete candidates in short write-locked batches (reapDeleteBatch = 128), re-checking staleness (a node may have heartbeated between scan and delete).
• Advance reapCursor and do the slog/audit outside the lock.

Behaviour is unchanged: same deletions (backbone membership, hostname index, pubkey/owner index preserved), same cursor semantics, same save() trigger.

Verification

• go test ./... (20 packages) green; reap tests green; -race on reap/stale tests green.

Context: after the merged encoder + a 1 TB pd-ssd, the live registry is ~88% CPU-bound on 16 vCPUs (pilot-rendezvous ~11.4 cores); the profile is now dominated by network syscalls + lock contention (beacon relay, then this reap, then report_trust).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!