Skip to content

fix: surrogate-pair-safe text excerpts — #1615 fingerprint line + family ratchet - #1626

Merged
ranxianglei merged 1 commit into
masterfrom
2026-09-29_surrogate-safe-fingerprint
Sep 29, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-09-29_surrogate-safe-fingerprint

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes #1615

What

summaryFingerprintLine (src/stream.ts) cut head/tail excerpts on raw UTF-16 code units. A cut landing inside a surrogate pair leaves a lone half; JSON.stringify emits it as an unpaired escape and strict upstream JSON parsers reject the entire request body. Because the receipt rides history every turn, this upgraded #816/#828's deterministic failure into permanent session death.

Root cause of the family (3rd recurrence)

#828 fixed the class by adding safePrefix — but only swept slice(0,N)+ellipsis sites. This site evaded the audit: no ellipsis, negative index slice(-100), and introduced 2026-09-25 by #1298 — after the hardening. New code kept hand-slicing instead of reusing the clamp. The durable fix is a shared module + a CI ratchet, not another one-site patch.

Changes

File Change
src/text-safe.ts (new) safePrefix / safeSuffix / scrubLoneSurrogates — single home for code-unit-safe excerpting
src/decompress-shared.ts local safePrefix deleted, imports the shared one (identical semantics, #828 behavior preserved)
src/stream.ts fingerprint head/tail → clamp + scrub
src/compress-loop-responses.ts, src/loop/core.ts log head/tail excerpts of tool arguments → safe helpers (same exposure class)
src/server.ts secret-mask head/tail → safe helpers (ASCII-identical output)
tests/text-safe.test.ts repro A (head cut on high surrogate) + repro B (tail cut on low surrogate) from the issue, JSON round-trip assertions, ASCII byte-stability of the fingerprint format, intact-pair passthrough, and the family gate: raw .slice(-N) on strings is ratcheted out of src/ (allowlist: array receivers + text-safe.ts itself)

Gates

  • tests/text-safe.test.ts 10/10
  • neighborhood (receipt/stream/decompress/conflict-watch) 30/30
  • full suite 3133 pass / 0 fail (16 e2e-gated skips)
  • npm run typecheck clean, npm run build OK

Notes

  • ASCII summaries produce byte-identical fingerprints as before — no wire-visible change for the common case.
  • The gate deliberately does not police positive slice(0,N) (too many benign array/log receivers); negative slices were the audit-evading shape and are now structurally banned.

…ting UTF-16 pairs (#1615)

summaryFingerprintLine sliced on raw code units; a cut landing inside a
surrogate pair left a lone half that JSON.stringify emits as an unpaired
escape, so strict upstreams reject the whole request body. The receipt
rides every turn once in history, upgrading the deterministic failure of
#816/#828 into permanent session death.

- src/text-safe.ts: shared safePrefix/safeSuffix/scrubLoneSurrogates
  (decompress-shared.ts now reuses it instead of its local copy)
- stream.ts fingerprint + compress-loop-responses/loop-core log head/tail
  + server.ts secret mask: all string excerpts go through the clamp
- family gate test: raw .slice(-N) on strings is ratcheted out of src/
  (allowlist: array receivers + text-safe itself)

Fixes #1615
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-29_surrogate-safe-fingerprint (d15ecc7)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1569-pr.1626.bf3be41

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1626.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei
ranxianglei merged commit 95530d9 into master Sep 29, 2026
11 of 13 checks passed
antclt pushed a commit to antclt/billion-context that referenced this pull request Oct 1, 2026
…anxianglei#1629)

path.relative yields backslash-separated keys on Windows while the
allowlist uses forward slashes — every allowlisted entry missed and the
gate reported already-approved array-receiver slices as offenders,
turning the Windows CI legs red repo-wide since ranxianglei#1626.

Fixes ranxianglei#1629
@ranxianglei
ranxianglei deleted the 2026-09-29_surrogate-safe-fingerprint branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant