Repository navigation
rendezvous: faster snapshot saves (hand-written encoder + trust-pairs cache) - #131
Merged
Merged
Conversation
added 2 commits
October 1, 2026 22:24
flushSave re-encoded the entire trust-pair list (millions of "min:max" strings at fleet scale) on every save, single-threaded — the dominant cost of snapshot encoding. The set is near-static in practice, so cache its JSON array and reuse it. - trust: trustPairSet gains a revision counter, bumped only on real add/remove; trust.Store.Revision() exposes it. - server: trustPairsEncodeCache memoizes the JSON array keyed by revision, with a 10-minute TTL safety net so a missed invalidation can't wedge a stale list. Empty list still returns nil so omitempty drops the field. - Snapshot.TrustPairs becomes json.RawMessage (emitted verbatim); load and replication decode it back to []string. Output is byte-identical to encoding the slice directly (tested), so the snapshot checksum and recovery semantics are unchanged. No behavioural change: the 5s save cadence and full-state snapshot are untouched. Tests: byte-equality vs encoding/json, revision invalidation, TTL expiry, empty->nil, and pairset revision semantics. Full module suite green.
200k nodes + 5M trust pairs: ~315ms -> ~203ms per snapshot encode (~35% faster), with ~344MB less allocated per save.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
added 3 commits
October 2, 2026 00:55
…tical) flushSave encoded the snapshot with encoding/json, which reflects over the whole 200k-node graph every save and allocates heavily (the dominant cost plus a GC storm at fleet scale). wal.AppendSnapshot is a hand-written top-level encoder: - Nodes and PubKeyIdx are encoded directly (map keys sorted, field order and omitempty mirrored) with a fast string escaper (falling back to encoding/json only for non-ASCII/control/quote/backslash strings). - TrustPairs is emitted verbatim from its cached RawMessage. - Small/rare fields delegate to encoding/json per value so their bytes are exactly right. flushSave now encodes into a reused buffer via AppendSnapshot, hashes, and writes body + checksum trailer atomically. Output is byte-identical to encoding/json (differential test + native fuzzing), so the snapshot checksum and load() recovery are unchanged. Bench (200k nodes): ~113ms -> ~64ms, 325MB -> 13MB allocated. Full cache-heavy case ~2.8x. Fuzzed: FuzzAppendJSONString, FuzzSnapshotNodes_MarshalParity, FuzzAppendSnapshot_MarshalParity (~31M execs, zero mismatches).
Chunking the node-map encode across goroutines measured no faster than serial (the encoder is memory-bandwidth-bound) while allocating ~18x more (172MB vs 9.4MB per 200k-node encode). Keep the serial hand-written encoder: 2.25x faster than encoding/json including the key sort, with ~6x less allocation.
Add an explicit differential test stuffing every string field (and map keys) with adversarial values — quotes, backslashes, HTML, control chars, U+2028/2029, unicode, emoji, invalid UTF-8, very long, whitespace — plus fuzz seeds for the same. Byte-parity with encoding/json holds; output is always valid JSON.
TeoSlayer
changed the title
TeoSlayer
enabled auto-merge (squash)
October 1, 2026 22:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
flushSavere-encodes the entire registry snapshot every ~5s. At fleet scale (~218k nodes, 8.3M trust pairs, ~191 MB JSON) the encode was the dominant cost:encoding/jsonreflects over the whole graph and allocates heavily (profiling showedappendString7.7s +mallocgc11s +findObject6s per window — i.e. a GC storm), all single-threaded.This PR makes the encode much cheaper while producing byte-identical output — the snapshot checksum and
load()recovery are unchanged.Changes
Hand-written top-level encoder —
wal.AppendSnapshot(withSnapshotNodes):NodesandPubKeyIdxare encoded directly (sorted keys, exact field order/omitempty) with a fast string escaper (falls back toencoding/jsononly for non-ASCII/control/quote/backslash strings).TrustPairsis emitted verbatim from a cachedjson.RawMessage.encoding/jsonper value so their bytes are exactly right.flushSaveencodes into a reused buffer, hashes, and writes body + checksum trailer atomically.Trust-pairs encode cache — a revision on the trust-pair set (bumped only on real add/remove) plus a 10-minute TTL safety net; empty →
nilsoomitemptystill drops the field.Funky-string hardening — explicit differential test stuffing every string field and map key with quotes, backslashes, HTML, control chars, U+2028/2029, unicode, emoji, invalid UTF-8, very long and whitespace values; plus fuzz seeds for the same.
Parity & tests
json.Encoder(SetEscapeHTML(false))— asserted by differential tests and three native fuzz targets (FuzzAppendJSONString,FuzzSnapshotNodes_MarshalParity,FuzzAppendSnapshot_MarshalParity; ~44M executions, zero mismatches).Measured (on the actual registry box under real load, Xeon 2.2 GHz)
Parallel chunking was tried and reverted — memory-bandwidth-bound, no faster, ~18× more allocation.
Notes
pd-standard→ 1 TBpd-ssd(fsync 3.7 MB/s → ~250–350 MB/s).