Repository navigation
fix: compute body-stability LCP over real UTF-8 bytes (#2868) - #2870
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.3642-pr.2870.fe18dd8This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr2870.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 PR #2870 review(fixes #2868)— 结论:可合 单模块、外科级、测试充分的字节单位修正。方向对、根因准、无回归、diff 干净。以下为结构化八问。 ① 是什么问题 ② 什么原因导致的 ③ 影响范围与严重程度 ④ 怎么修复的 ⑤ 是否有回归问题(逐项过既有行为)
⑥ 是否破坏缓存 ⑦ 是否改了设计方向 ⑧ 是否引入新的配置 行为变更清单(均诊断字段,已披露)
验证证据
判定:可合。 按 §7.4 auto-merge 门衡量 1–7 条干净通过,第 8 条不触 load-bearing infra。唯一需 owner 知悉:大 CJK 正文字节超 8 MiB 后从「保留全文」变为「只留 digest」——有意、已披露的内存 bound 收紧,属运行时行为变化但不阻塞合并,特此点名。 |
d777243 to
5b2a92d
Compare
|
解决冲突 |
[bot] 🏷 上游合并冲突已解决并推送,PR #2870 现已恢复可合(API 核实 是什么问题:PR 分支与目标分支 master 出现合并冲突。先用 GitHub API 独立核实为真( 什么原因导致的:master 在分支基线(merge-base 影响范围与严重程度:纯诊断/测试夹具层。golden 用于钉住持久化+渲染行为,若手拼错值会让 golden 测试假红;无用户面影响。 怎么修复的:用 merge(而非 rebase+force-push)把 origin/master 并入分支——该分支由另一 agent 持有且可能仍有并发推送,普通 push 天然不会覆盖他人提交,更安全(规范允许 rebase 或 merge)。4 个冲突 golden 不手改哈希,而是跑项目自带的 是否有回归问题:无。PR 相对 master 的净 diff 仍是原样的 7 个文件( 是否破坏缓存:不涉及。未触碰任何缓存/重试/游标/去重逻辑;#2868 本身只是诊断字段字节化(ASCII 逐字节不变)。 是否改了设计方向:不涉及。仅为解冲突加了一个 merge 节点,架构/边界/数据流无转向。 是否引入新配置:不涉及。无新增 env/config/CLI。 处置说明:本地 typecheck + 定向族全绿后,先复核远端 head 仍为父提交 判定:冲突已清除,PR 恢复可合,等待人工合并。 |
[bot] 🏷 已核实:当前没有需要解决的冲突,PR #2870 处于可合状态。 上一轮我已把 origin/master 并入分支并推送(
结论:冲突在上一轮已清除,现在无需再做任何冲突处理。若你界面仍显示"有冲突",应为缓存未刷新——以 API 的 |
f39aa67 to
b60d468
Compare
|
冲突 rebase master |
The retention assertions now read the UTF-8 encoding back out of the Buffer slot (byte-equal for valid UTF-8 bodies), and a new test pins that a lone surrogate is retained as U+FFFD — it has no UTF-8 form. Heap-probe thresholds unchanged.
b60d468 to
5188102
Compare
(ai: vllm-qwen/qwen3.8-27b) 已按要求把分支 rebase 到最新 master(线性历史,去掉之前那个非线性的 merge 节点),验证通过、可合。 做了什么
验证
诚实提醒:master 这几分钟一直在动(push 时基线 八问
判定:可合,等人工合并(我不自行 merge PR)。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.416-master.8d54a3e # this exact build
npm install -g billion-context@master # always the newest merged build
|
What
byteLcp/seamLcpinsrc/cache-ledger.tscompared UTF-16 code units (charCodeAt) while every consumer treated the result as UTF-8 bytes:LedgerLine.bl("first diverging byte offset"),SeamEvent.lcpBytes, the 1024-byteheadthreshold, the ±64-byte append tolerance, the 512 KiB seam-head cap and the 8 MiB full-retention cap. For non-ASCII (CJK / astral) bodies the offsets were wrong by up to 3× and divergence classes could flip (head↔mid). The full SHA-256 equality proof (be/bd) was already byte-correct and stays untouched.Fix
Encode each forwarded body to UTF-8 once at capture (
noteForwardedBody) and reuse that buffer for the digest, the capped seam head and the retained full body — no comparison path re-encodes.byteLcp/seamLcpnow walk real bytes; caps, LCP, append tolerance, head threshold and displays are all byte-unified. JSON parsing reads the (possibly capped) buffer viatoString("utf8"); a mid-character cut yields the same parse-fail outcome as before. Slicing the string at the old LCP and re-encoding was explicitly rejected: distinct codepoints can share leading UTF-8 bytes (issue cases 3 & 5), so the comparison must happen on byte arrays.Behavior changes (diagnostic-only fields; no billing/closure math touched)
bl,lcpBytesvaluesbsclass for non-ASCII pairshead)mid)SEAM_BODY_CAP(512 KiB) enforcementBODY_FULL_CAP(8 MiB) retentionbs=unknown)str.length >= CAP(an exactly-at-cap body read as clipped)bytes > CAP(clipped only)Old persisted ledgers keep their historical
blvalues (display-only sparse field; no persistence format or version change).Config surface
None — no config fields, env vars or CLI flags added or changed.
Tests
New
tests/issue2868-byte-lcp-utf8.test.tspins the byte contract end-to-end against an independent Buffer-walking reference: the issue's five synthetic pairs (expected offsets 3/3/2/1200/2), the 1200-byte-LCPmidclass pin (the regression case), byte-exact head offset with CJK inside the first KiB, JSON requests with Chinese content, append/trim with CJK, both cap boundaries (over the byte cap yet under the old code-unit cap → clipped seam head with retained full body; over 8 MiB bytes yet under 8 MiB code units → digest-only), identical/different digests, a 16 MiB performance guard, and seam forensicslcpBytes. Existing #2131 / capped-attribution / seam-detector / fuzz / probe / ledger suites pass unchanged (their ASCII expectations are byte-invariant).Local: typecheck clean, targeted cache-ledger/seam family (21 files: 150 passed, 1 pre-existing skip) green, build OK. Full suite on CI.
Closes #2868
Storage-render goldens — regeneration justification (required by tests/golden-storage-render.test.ts)
The four
tests/golden/storage-render/*.jsonsnapshots pin the persisted session's metadata section digest, which includes the cache-ledgerblvalues. The golden scenario's pre-fold history is NOT pure ASCII: the judge upstream's generated assistant lines carry•bullets ([assistant • shell]) plus other multi-byte glyphs, so every outbound body shares a common prefix containing extra UTF-8 bytes beyond its code-unit count. With this fix, all 14 affected ledger lines reportblexactly +107 higher (the constant extra-byte count in that shared prefix) — verified by dumping the full normalized persisted state under base code vs fixed code and diffing: the ONLY differences are those 14blnumbers; wire-body digests, state, blockContents and stats sections are byte-identical. The old goldens pinned the buggy code-unit values; these regenerate them to true bytes. No render contract or storage format change.Rebase history: first onto
60c47ba0f(picks up #2867's persisted ledger/mode fields + golden regen41cede6d6), then onto current masterbfbdb23db. The second rebase had ONE conflict —src/cache-ledger.ts, rewritten by #2923 (3823e827e, seam slots moved from module WeakMaps tosession.transient.sentBody/settledBody). Resolved as the union of intent: #2923's session-lifecycle placement + this PR's byte-unit Buffer slot shape (head/bytes/fullBuffers instead of their UTF-16 copies). Their release invariant is preserved and strictly cheaper — one fresh UTF-8 allocation instead of two utf16le round-trip copies, and nothing retains the original request string.tests/session-transient-memory.test.tsported accordingly (retention asserted by reading the UTF-8 encoding back out of the slot; new lone-surrogate→U+FFFD pin; heap-probe thresholds unchanged, all pass). Golden regen stays valid: no master commit since touchedtests/golden-storage-render/; diff vs master's goldens remains exactly the two metadata-affecting digest lines per file.Third rebase onto
6580da585(owner-reported conflict): master's #2929 series (6ef17e8de,aba017fe5,6f4cd2380) extended the same seam logic this PR rewrote — the provider-side arms gained an idle-gap TTL/side split (bookProvider()) — and two of its commits refreshed the storage-render goldens again.src/cache-ledger.tsauto-merged as the union and was verified arm-by-arm (#2929's TTL/side split now wraps this PR's byte-unit guards; all four seam arms keephead.byteLength/bytes > CAPsemantics). The golden commit conflicted and was resolved by regenerating from the rebased tree: diff vs master's goldens remains exactly two digest lines per file (stateDigest + metadata sectionDigest), golden suite 5/5. Local: typecheck clean, targeted cache-ledger/seam/transient/golden family 110/110 pass including #2929's new fold-claim-window suite.Fourth rebase onto
c800ed1bb(owner-reported conflict): master's #2982 (c7b851eb2, body-stabilitytailclass viamessagesArrayCloseOffset) and #3023 (20c425823, client-side insert attribution) both landed in the same seam code this PR rewrote while still written against the old string slot. Textual auto-merge was clean but left one SEMANTIC conflict — #3023's client-arm forensics line calledseamLcp(prev.str, cur.str)and #2982's scan tookstring; both now operate on the byte-unit Buffer (messagesArrayCloseOffsetis a byte-level scan: every delimiter it checks is ASCII and UTF-8 continuation/lead bytes can never equal an ASCII code point, so the scan is equivalent on multi-byte content and returns a byte offset comparable tobyteLcp). Also found (not fixed here): at the intermediate basee74ed8990master itself was missing thepersonaNamespaceimport insrc/server/handle.ts(#3060) — already restored upstream bya9378038abefore this rebase, so no action needed. Golden regen + targeted family green again (delta vs master goldens: exactly two digest lines per file).