Repository navigation
rendezvous: take the remaining hot-path locks off the request path - #135
Merged
Merged
Conversation
added 3 commits
October 2, 2026 01:51
IsWhitelisted took the same mutex as the token buckets and re-parsed the CIDR list on every call; the live mutex profile showed it as the #2 contention site (~3.5k s of wait per 5 min). The whitelist is written rarely (operator) and read on every connection, so publish it copy-on-write behind an atomic.Pointer and drop the lock. The JSON and binary connection loops also re-armed a 5-minute read deadline on every message, one runtime timer modification per request (~43k/s under load). Re-arm at most once a minute; the deadline keeps at least 4 minutes of headroom, so idle-connection semantics are unchanged.
The default slog handler formats and writes under its own mutex on the caller goroutine; the live mutex profile showed it as a top contention site (~1.1k s of wait per 5 min). Wrap the configured handler in a buffered async handler so hot request paths only enqueue, with formatting and writing on a dedicated goroutine. Overflow drops are counted instead of blocking callers. The dynamic log-level endpoint reinstalls the wrapper so it survives level changes.
The handler held the global write lock (and the node shard) across st.cb.Save(). Save is a non-blocking trigger and reads state under its own locks when it runs, so moving it after the unlock shortens the critical section without changing ordering.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the registry CPU work (#130–#134). Live 5-minute mutex profile after the v0.2.9-beta.6 deploy still listed four avoidable hot-path sites; this PR removes them.
Changes
IsWhitelistedtook the rate-limiter mutex (the same one guarding token buckets) and re-parsed the CIDR list on every call — ~1.1M acquisitions / 3.5k s of wait per 5 min. The whitelist is written rarely and read per connection, so it is now published copy-on-write behind anatomic.Pointer[[]whitelistRule].SetReadDeadlineon every message — one runtime timer modification per request (~43k/s live). Deadline is now re-armed at most once a minute, keeping ≥4 minutes of headroom; idle semantics unchanged.HandleSetHostname.st.cb.Save()is a non-blocking trigger, so it no longer runs inside thest.mu+ node-shard critical section.Testing
go build ./...,go test ./...green.go test -race -run "Whitelist|RateLimit" ./accept/green.Verification plan
Deploy to
pilot-rendezvous-new, capture 5-minute mutex + CPU profiles, and compare against/tmp/locks5m.json//tmp/cpu5m.pprof(before). Snapshot output remains byte-identical (no encoder changes).