Skip to content

Validation: prevent SimHash feature vote overflow - #1

Closed
Afloat16 wants to merge 2 commits into
masterfrom
2026-11-09_fix-simhash-vote-overflow
Closed

Afloat16 wants to merge 2 commits into
masterfrom
2026-11-09_fix-simhash-vote-overflow

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Problem

Signed 8-bit counters overflow within the existing 256-feature sampling budget, collapsing repeated-message sketches and causing unrelated messages to collide.

Changes

Use signed 16-bit vote counters while preserving sampling and the serialized sketch format.

Regression coverage

The new suite checks repeated Latin and CJK shingles across 127/128 and 255/256 votes and across stride changes, plus distinct unrelated sketches at the 256-feature cap.

Validation

Passed against exact fork commit d90c2b6320e58f4b9b7c42d4b22a77d503aa511f in an isolated GitHub Actions checkout: validation job. The targeted regression and related suites, npm run typecheck, and npm run build all passed on Node 22. This is targeted verification, not the full upstream CI matrix.

node --import ./tests/preload-storage-baseline.mjs --import ./tests/preload-test-env.mjs --import tsx --test tests/simhash-vote-range.test.ts tests/prefix-affinity-simhash.test.ts
npm run typecheck
npm run build

Related work

Original feature ranxianglei#2266; no matching overflow correction found.

Purpose of this fork PR

Internal verification only. The upstream PR has not been opened; this fork PR is excluded from contribution and merged counts.

The storage driver persists fold-anchor SimHash values. Regenerate its four
wire goldens after correcting signed-vote overflow. Verification restricts
the persisted-state delta to foldAnchors.*.sh; outbound bodies and all
other state remain unchanged.
@Afloat16 Afloat16 closed this Nov 9, 2026
ranxianglei pushed a commit that referenced this pull request Nov 14, 2026
…n ref-ledger reset

detectGenerationBoundary() already proves that the persisted ref ledger belongs
to a previous generation: >=2 distinct above-frontier citations at request #1
(ranxianglei#4305). It only recorded a standing note, so the compression ledger was never
repaired. Observed on 0.1.212 with a ZCode client (ranxianglei#6061, ranxianglei#5589): acp_status
advertised the current generation (highest allocated ref m00314) while compress
validated against the stale map (highest allocated ref m159141, 586 blocks) and
rejected every live ref with "every ref is unknown to this session
[cause: stale-ref]" — 20 consecutive failures, circuit breaker armed, and the
nudge kept offering the same unreachable ranges.

The detector's verdict is exactly the evidence the native-compaction rebase has
always required, so arm it there: markNativeCompactionBoundary() sets
pendingRebase, and the next request runs resetSessionCompression() +
reserveRefsThrough(), re-anchoring the map onto the generation the client is
actually minting.

Also honour a pending boundary at the head of the anthropic lane. Its rebase
call sits behind the dsh/codex client-side-compaction signature, which a plain
model or upstream switch never produces, so an armed boundary would otherwise
wait forever on that lane (openai/google/responses reach reconcile through their
own branches; responses already reconciles unconditionally).

Verified: npm run typecheck clean; tests/generation-boundary.test.ts 12/12.
(cherry picked from commit 7a1acae)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant