Skip to content

[PR] perf(affinity): construct only sampled SimHash shingles - #6249

Closed
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-11_sample-simhash-shingles
Closed

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-11_sample-simhash-shingles

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Fixes #6248

Problem

A necessary cold SimHash evaluation currently allocates every consecutive word pair, then consumes only the uniformly sampled subset (at most 256 features). A million-word input creates 999,999 bigram strings even though almost all are discarded unused. This adds substantial allocation and garbage collection for long messages.

Change

Compute the feature count and stride directly from token count and construct a bigram only when it is sampled. The sample positions, Unicode tokenization, unigram/raw-input fallbacks and 16-bit vote counters stay unchanged. Token storage remains linear; this removes the redundant full-size shingle array.

Golden sketches captured from the old implementation cover distinct tokens around stride boundaries, 100,000 tokens, Unicode and degenerate inputs. The manual benchmark runs the production function and reports the sketch so before/after compatibility is visible:

node --max-old-space-size=512 --expose-gc --import tsx scripts/bench-simhash-sampling.mjs 1000000

Validation

  • Node 24.22.1 / Linux x64, 7,889,651 bytes, one warm-up and three measured calls: median 330.96 ms → 40.14 ms; process peak RSS 366,812 KiB → 169,004 KiB. Measurements are machine/load dependent and include tokenization.
  • Before/after sketch is exactly 047251b26ebac679.
  • 108 tests passed across SimHash votes/sampling, prefix-affinity adoption, fold anchor doctrine and fold reconciliation.
  • The new compatibility test also passes on unmodified master; the benchmark is the performance counterexample, with no timing threshold in CI.
  • npm run typecheck, npm run build, and git diff --check: passed.

Based independently on current master dc041dd1ab1a014efa352ba47a6f31fc47f55284. Prior #6046 fixes vote overflow and #2828 skips unnecessary calls; this patch removes unused shingle allocation inside necessary evaluations.

@ranxianglei

Copy link
Copy Markdown
Owner

Landed via batch PR #6439 (merged) — closing as superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][affinity] SimHash allocates every bigram although it samples at most 256

2 participants