Skip to content

[PR] fix: prevent SimHash feature vote overflow - #6046

Merged
ranxianglei merged 2 commits into
ranxianglei:masterfrom
Afloat16:2026-11-09_fix-simhash-vote-overflow
Nov 10, 2026
Merged

ranxianglei merged 2 commits into
ranxianglei:masterfrom
Afloat16:2026-11-09_fix-simhash-vote-overflow

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Signed 8-bit vote counters overflow within the existing 256-feature sampling budget. Repeating one shingle can reverse its sketch at 128 votes or collapse it to zero at 256 votes, allowing unrelated repeated messages to collide.

Changes

Use signed 16-bit vote counters, which cover the full sampled vote range without changing feature sampling or the 16-character serialized sketch format.

The regression suite checks repeated Latin and CJK shingles around the 127/128 and 255/256 boundaries and across sampling strides, plus distinct unrelated sketches at the cap.

Storage golden update

fold-reconcile.ts stores these same SimHash values in metadata.foldAnchors.<id>.sh. Correcting the arithmetic intentionally changes those persisted fingerprints. Regenerate the four storage-render golden files with the repository's scripts/update-storage-render-goldens.ts.

The before/after verification compared complete normalized persisted state for all four wire scenarios. Only 13 fold-anchor sh values changed; outbound bodies, fold/edit landmarks, state shape, block contents, stats and all other persisted fields remained unchanged. Each golden changes only stateDigest, stateDigestAligned and sectionDigests.metadata. No assertion or normalization rule was weakened.

Before/after verification: the unchanged base passed all 17 storage golden/restart tests; the corrected fingerprints and regenerated goldens passed all 36 SimHash, alignment, storage-render and restart tests. Typecheck and production build also passed.

Validation

Exact commit c26c68a93fb11035a0e02cd08b171a295abbddb4 passed all 36 tests, npm run typecheck, and npm run build in a fresh Node 22 checkout: committed-version verification. The tracked source and golden files remained unchanged during validation. This checks the targeted and related suites plus the four-wire storage and restart contracts; the upstream full CI matrix runs independently.

Related work

Original feature #2266; no matching vote-overflow correction was found in the upstream or prior contribution history.

Closes #6036.

@ranxianglei ranxianglei changed the title fix: prevent SimHash feature vote overflow [PR] fix: prevent SimHash feature vote overflow Nov 9, 2026
The storage driver persists fold-anchor SimHash values. Regenerate its four
wire goldens after correcting signed-vote overflow. Verification restricts
the persisted-state delta to foldAnchors.*.sh; outbound bodies and all
other state remain unchanged.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI re-verified live — ALL GREEN on current head c26c68a; nothing left to fix this run.

What I found and did:

  1. Re-verified live state first. The event reported red test shards (test (ubuntu-latest, 24, 2), test (ubuntu-latest, 22, 2), test (windows-latest, 22, 1)) on head d90c2b6 (run 42640033237). By the time of this wake the PR head had already moved to c26c68a ("test: refresh storage goldens for corrected SimHash votes") — a prior wake pushed it to fix exactly these failures.
  2. Verified the new run end-to-end. The local mirror cache was stale throughout this window, so per the single-fetch fallback I made two spaced one-shot upstream check-runs queries for head c26c68a: 48 check runs total — 36 success, 11→12 skipped, 0 failed; the last three queued test shards (ubuntu 24/2, ubuntu 24/3, ubuntu 22/1) finished green between the two queries. All previously-red shards pass on the new head.
  3. No push needed. Nothing to fix, working tree untouched, no gates hit, no design decision required.
  4. Housekeeping note: the branch is behind master (no merge conflict) — a routine rebase before merge; merging itself stays human-only.

结构化结论(八问)

  1. 是什么问题:PR [PR] fix: prevent SimHash feature vote overflow #6046(SimHash 投票溢出修复 Int8Array→Int16Array)在旧 head d90c2b6 上把 3 个测试分片打红——修复改变了 sketch 输出,而 storage-render 的 4 份 golden 未同步更新。
  2. 什么原因导致的:根因是投票数组越界(256 个 feature 各投 ±1,超出 Int8 的 -128..127);改 Int16Array 后部分用例的 simhash 结果变化,golden 文件滞后于代码变更。
  3. 影响范围与严重程度:仅 CI 红,不影响任何已发布行为;严重程度低(体验/流程层)。
  4. 怎么修复的:前一轮 wake 已在 PR 分支补上 golden 刷新(c26c68a,4 份 storage-render golden)+ 回归测试 tests/simhash-vote-range.test.ts(钉住重复 shingle 投票与采样上限下的区分度)。本轮只做活体核验,无新增改动。
  5. 是否有回归问题:已审——投票范围放宽只改变原本溢出回绕的 sketch 值,属修正而非语义漂移;全量 48 项检查(含原红分片)在新 head 全绿,无既有行为被破坏。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:CI 全绿,可进入人工 review/merge;merge 权归 owner。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

Dup screening: not a duplicate. #6036 is the originating bug report ("[Bug] Prevent SimHash feature vote overflow", Afloat16); #6046 is its only fix PR. No other open/closed issue or PR touches the SimHash vote overflow or the Int8Array counter. One PR for this thread — no parallel work.

Independent code review of PR #6046 (head c26c68a, 2 commits, 6 files, +33/−13). I checked out the head, built kernel+pi-subagents, and verified everything below locally; it's consistent with the prior CI-engineer wake's all-green finding (I re-read the mirror CI cache this run: 30 checks, 0 failed, 0 pending).

What I independently verified:

  • Bug is real. simhashHex accumulates ±1 per sampled feature into Int8Array(64) (src/affinity-simhash.ts:69). With up to MAX_FEATURES = 256 sampled features, one bit position can receive up to 256 identical votes; Int8Array wraps at ±128, so a highly-uniform message (e.g. 256 identical shingles) drives every counter to 0 → the sketch collapses to 0000000000000000, and two different repetitive messages collide. Confirmed empirically: reverting the line to Int8Array makes both new tests fail (test 1 fails exactly at "128 shingles", the wrap boundary; test 2's distinctness collapses); restoring Int16Array makes them pass.
  • Fix is correct & root-cause. Widening the counter to Int16Array (±32767) comfortably holds the ±256 max — fixed at the source (the counter type), not by clamping/masking a symptom. Sampling budget (256), stride math, and the 16-hex serialized format are all unchanged.
  • Tests are genuine (not tautologies). They pin the exact overflow boundaries (127/128, 255/256), Latin + CJK, stride changes (feature counts 1→512), and distinctness at the cap. Fail on old code, pass on new — real regression teeth.
  • Goldens are load-bearing and correctly refreshed. Reverting the 4 storage-render goldens to master values makes the suite fail (4/4); the PR's goldens pass. On the rebased state (current master + fix) they still pass, so #5715's fold-reconcile changes did not make them stale.
  • No missed inline expectations. Ran all four other affinity-simhash importers (fold-reconcile, issue2824-regression, issue2915-order-centric-reconcile, fold-anchor-doctrine): 103/103 pass.
  • typecheck + build green locally (Node 22); rebase onto current master applies cleanly (every file the PR touches is untouched by master's 111 intervening commits).

Behavior change to flag (mandatory disclosure): for inputs where Int8Array overflowed, the computed simhash value changes from a corrupt/wrapped value to the correct one. Consequences: persisted stateDigest / sectionDigests.metadata change (hence the golden refresh); and pre-upgrade session.metadata.foldAnchors.sh values written by old builds will differ from freshly-computed sketches on the first reconcile pass after upgrade — but only for that narrow overflow class, it fails in the safe direction (simhash corroboration simply doesn't fire → falls through to order/exact channels or honest re-entry; no data loss, no wrong claim), and it self-heals because anchors are recomputed via anchorFrom() whenever the message bytes are present. Normal (non-overflowing) messages are byte-for-byte unchanged.

Assessment: high-quality external contribution — minimal clean diff, correct root-cause fix, genuine regression coverage, no quality defects requiring rework (§②a not triggered). The PR is currently unlabeled; if labeling, I'd suggest bug + p2 (correctness issue under a narrow trigger — degenerate/repetitive content; normal traffic unaffected) + scope/kernel (session-identity/fold core logic).


结构化结论(八问)

  1. 是什么问题:SimHash 特征投票用有符号 8 位计数器(Int8Array,范围 -128..127),但单次最多投 256 票,超出即回绕——重复度高的消息 sketch 塌缩成全零,导致不同消息碰撞。
  2. 什么原因导致的:计数器位宽(8 位)小于投票值域(±256)。修法放宽到 16 位(Int16Array)是在源头(计数器类型)修根因,不是截断/掩盖症状。
  3. 影响范围与严重程度:仅触发于「某比特位净投票 >127」的退化重复内容;正常文本逐字节不变。后果是匿名会话收养(Trae http改URL模式:切模型改写历史 → 匿名身份链每请求断裂 re-mint → 1.4M 全量入站 + 重复折叠死循环 #2265)与折叠 gap 闭合(fold-reconcile)对这类消息的相似度判定失真——可能误采/漏配,但受 user-anchor、wordBag、长度门约束,实际风险低。定级 p2(特定条件问题,正常流量无感)。
  4. 怎么修复的:计数器 Int8Array→Int16Array(一行),采样预算 256、stride、16-hex 序列化格式全部保持不变;补回归测试钉住 127/128、255/256 边界与上限区分度;刷新 4 份 storage-render golden。
  5. 是否有回归问题:已逐项审——①非溢出输入零变化(Int8==Int16);②溢出输入值被修正(修正而非语义漂移);③跨版本边界:旧构建写入的持久化 foldAnchors.sh 与新算 sketch 在升级后首个 reconcile pass 可能对不上,但只限窄类、朝安全方向失败(该通道不触发→落到 order/exact/honest re-entry,不丢数据不误吞)、且 anchor 随消息字节经 anchorFrom() 重算后自愈;④golden 在当前 master(#5715 之后)仍成立,rebase 干净无冲突;⑤其余 4 个 affinity-simhash 消费方 103/103 绿。无既有行为被破坏。
  6. 是否破坏缓存:不涉及。sketch memo 为进程内缓存、按 itemHash 键控,值变化属修正;持久化侧仅上述一次性自愈瞬态,无失效策略被破坏。
  7. 是否改了设计方向:不涉及——纯 bug 修复,不动架构/数据流。
  8. 是否引入了新的配置:不涉及。

判定:方向正确、实现最小、测试真实、无回归风险(除已披露的一次性安全瞬态)。可进入 merge——仅需例行 rebase 到当前 master(应用干净、golden 不 stale),merge 权归 owner。CI 本轮镜像缓存实测 30 项全绿。

@ranxianglei
ranxianglei merged commit 8095a49 into ranxianglei:master Nov 10, 2026
48 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1145-master.8095a49 (rolling master tag).

npm install -g billion-context@0.0.1145-master.8095a49  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

This was referenced Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Prevent SimHash feature vote overflow

2 participants