Skip to content

fix: compute body-stability LCP over real UTF-8 bytes (#2868) - #2870

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-12_byte-lcp-utf8
Oct 14, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-12_byte-lcp-utf8

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

What

byteLcp/seamLcp in src/cache-ledger.ts compared UTF-16 code units (charCodeAt) while every consumer treated the result as UTF-8 bytes: LedgerLine.bl ("first diverging byte offset"), SeamEvent.lcpBytes, the 1024-byte head threshold, the ±64-byte append tolerance, the 512 KiB seam-head cap and the 8 MiB full-retention cap. For non-ASCII (CJK / astral) bodies the offsets were wrong by up to 3× and divergence classes could flip (head ↔ mid). The full SHA-256 equality proof (be/bd) was already byte-correct and stays untouched.

Fix

Encode each forwarded body to UTF-8 once at capture (noteForwardedBody) and reuse that buffer for the digest, the capped seam head and the retained full body — no comparison path re-encodes. byteLcp/seamLcp now walk real bytes; caps, LCP, append tolerance, head threshold and displays are all byte-unified. JSON parsing reads the (possibly capped) buffer via toString("utf8"); a mid-character cut yields the same parse-fail outcome as before. Slicing the string at the old LCP and re-encoding was explicitly rejected: distinct codepoints can share leading UTF-8 bytes (issue cases 3 & 5), so the comparison must happen on byte arrays.

Behavior changes (diagnostic-only fields; no billing/closure math touched)

Field Old New Why
bl, lcpBytes values UTF-16 code-unit counts true UTF-8 byte offsets the declared semantics was always "bytes" (KiB/MiB displays, 1024-byte threshold); ASCII sessions are byte-for-byte unchanged
bs class for non-ASCII pairs computed from code units (e.g. the 中×400 pair → head) computed from bytes (same pair → mid) same root cause
SEAM_BODY_CAP (512 KiB) enforcement code units (up to ~1.5 MiB retained for CJK) bytes the memory bound is about bytes
BODY_FULL_CAP (8 MiB) retention code units (up to ~24 MiB retained for CJK) bytes — bodies between 8 MiB and ~24 MiB bytes are now digest-only (bs=unknown) same; also fixes the worst-case per-slot retention
capped detection str.length >= CAP (an exactly-at-cap body read as clipped) bytes > CAP (clipped only) measure-zero correction, strictly more correct

Old persisted ledgers keep their historical bl values (display-only sparse field; no persistence format or version change).

Config surface

None — no config fields, env vars or CLI flags added or changed.

Tests

New tests/issue2868-byte-lcp-utf8.test.ts pins the byte contract end-to-end against an independent Buffer-walking reference: the issue's five synthetic pairs (expected offsets 3/3/2/1200/2), the 1200-byte-LCP mid class pin (the regression case), byte-exact head offset with CJK inside the first KiB, JSON requests with Chinese content, append/trim with CJK, both cap boundaries (over the byte cap yet under the old code-unit cap → clipped seam head with retained full body; over 8 MiB bytes yet under 8 MiB code units → digest-only), identical/different digests, a 16 MiB performance guard, and seam forensics lcpBytes. Existing #2131 / capped-attribution / seam-detector / fuzz / probe / ledger suites pass unchanged (their ASCII expectations are byte-invariant).

Local: typecheck clean, targeted cache-ledger/seam family (21 files: 150 passed, 1 pre-existing skip) green, build OK. Full suite on CI.

Closes #2868

Storage-render goldens — regeneration justification (required by tests/golden-storage-render.test.ts)

The four tests/golden/storage-render/*.json snapshots pin the persisted session's metadata section digest, which includes the cache-ledger bl values. The golden scenario's pre-fold history is NOT pure ASCII: the judge upstream's generated assistant lines carry • bullets ([assistant • shell]) plus other multi-byte glyphs, so every outbound body shares a common prefix containing extra UTF-8 bytes beyond its code-unit count. With this fix, all 14 affected ledger lines report bl exactly +107 higher (the constant extra-byte count in that shared prefix) — verified by dumping the full normalized persisted state under base code vs fixed code and diffing: the ONLY differences are those 14 bl numbers; wire-body digests, state, blockContents and stats sections are byte-identical. The old goldens pinned the buggy code-unit values; these regenerate them to true bytes. No render contract or storage format change.

Rebase history: first onto 60c47ba0f (picks up #2867's persisted ledger/mode fields + golden regen 41cede6d6), then onto current master bfbdb23db. The second rebase had ONE conflict — src/cache-ledger.ts, rewritten by #2923 (3823e827e, seam slots moved from module WeakMaps to session.transient.sentBody/settledBody). Resolved as the union of intent: #2923's session-lifecycle placement + this PR's byte-unit Buffer slot shape (head/bytes/full Buffers instead of their UTF-16 copies). Their release invariant is preserved and strictly cheaper — one fresh UTF-8 allocation instead of two utf16le round-trip copies, and nothing retains the original request string. tests/session-transient-memory.test.ts ported accordingly (retention asserted by reading the UTF-8 encoding back out of the slot; new lone-surrogate→U+FFFD pin; heap-probe thresholds unchanged, all pass). Golden regen stays valid: no master commit since touched tests/golden-storage-render/; diff vs master's goldens remains exactly the two metadata-affecting digest lines per file.
Third rebase onto 6580da585 (owner-reported conflict): master's #2929 series (6ef17e8de, aba017fe5, 6f4cd2380) extended the same seam logic this PR rewrote — the provider-side arms gained an idle-gap TTL/side split (bookProvider()) — and two of its commits refreshed the storage-render goldens again. src/cache-ledger.ts auto-merged as the union and was verified arm-by-arm (#2929's TTL/side split now wraps this PR's byte-unit guards; all four seam arms keep head.byteLength/bytes > CAP semantics). The golden commit conflicted and was resolved by regenerating from the rebased tree: diff vs master's goldens remains exactly two digest lines per file (stateDigest + metadata sectionDigest), golden suite 5/5. Local: typecheck clean, targeted cache-ledger/seam/transient/golden family 110/110 pass including #2929's new fold-claim-window suite.

Fourth rebase onto c800ed1bb (owner-reported conflict): master's #2982 (c7b851eb2, body-stability tail class via messagesArrayCloseOffset) and #3023 (20c425823, client-side insert attribution) both landed in the same seam code this PR rewrote while still written against the old string slot. Textual auto-merge was clean but left one SEMANTIC conflict — #3023's client-arm forensics line called seamLcp(prev.str, cur.str) and #2982's scan took string; both now operate on the byte-unit Buffer (messagesArrayCloseOffset is a byte-level scan: every delimiter it checks is ASCII and UTF-8 continuation/lead bytes can never equal an ASCII code point, so the scan is equivalent on multi-byte content and returns a byte offset comparable to byteLcp). Also found (not fixed here): at the intermediate base e74ed8990 master itself was missing the personaNamespace import in src/server/handle.ts (#3060) — already restored upstream by a9378038a before this rebase, so no action needed. Golden regen + targeted family green again (delta vs master goldens: exactly two digest lines per file).

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-12_byte-lcp-utf8 (ca34fe8)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3642-pr.2870.fe18dd8

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2870.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #2870 review(fixes #2868)— 结论:可合

单模块、外科级、测试充分的字节单位修正。方向对、根因准、无回归、diff 干净。以下为结构化八问。

① 是什么问题
cache-ledger 的正文稳定性 LCP(byteLcp/seamLcp)用 UTF-16 code unit(charCodeAt)比较,但所有消费方——bl「首个分叉字节偏移」、SeamEvent.lcpBytes、1024 字节 head 阈值、±64 字节 append 容差、512 KiB/8 MiB cap——都把它当 UTF-8 字节用。非 ASCII(CJK/astral)正文偏移最多差 3×,divergence 分类还会翻转(head↔mid)。SHA-256 等值证明(be/bd)本就按字节算,未受影响。

② 什么原因导致的
单位不匹配(根因层):比较侧产出 code unit,消费侧语义是 byte。ASCII 下两者相等所以长期无症状;一旦含 CJK(1 cu=3 B)或 astral(2 cu=4 B),LCP 数值与所有按字节解释的阈值/cap/display 全部错位。表面现象(偏移偏小、分类翻转、内存超 bound)都是这单一根因的不同投影。修法必须在字节数组上比较——不能「按旧 LCP 切字符串再重编码」:不同码点可共享前导 UTF-8 字节(中=E4 B8 AD vs 六=E4 B8 AE 前 2 字节相同;U+20000/F0 80 80 80 vs U+20400/F0 80 90 80 前 2 字节相同),切字符串会丢真实公共前缀。

③ 影响范围与严重程度
仅诊断面:/acp-cache 报告 divergence class 计数、seam 事件 lcpBytes、web UI 偏移展示。不触及计费/闭包/nudge/压缩决策——我 grep 确认 bl/bs/bd/be/lcpBytes 的全部读取点都是聚合计数(cache-ledger.ts:1331-1336)、格式化(web/client.ts fmtB(ev.lcpBytes))、日志。触发条件:非 ASCII 正文 + 相邻两次请求正文分叉。严重度=「特定条件下有问题」,非功能故障;附带修正了 CJK 大正文内存保留超上限(旧 8 MiB code-unit cap 实际可留到 ~24 MiB 字节)。

④ 怎么修复的
捕获时(noteForwardedBody)把正文一次性编码为 UTF-8 Buffer,digest/capped head/retained full 复用同一 buffer,任何比较路径不再重编码;byteLcp/seamLcp 改走真实字节,cap/LCP/append 容差/head 阈值/display 全统一到字节。capped head 用 Buffer.from(buf.subarray(0, CAP)) 拷贝以真正切断对多 MiB 父 buffer 的引用(subarray 视图会钉住整个父 buffer,不拷贝则内存 bound 失效)。JSON 解析读可能被切的 buffer 经 toString("utf8")——切在字符中间产生替换字符并让截断 JSON 解析失败,与旧行为同果。

⑤ 是否有回归问题(逐项过既有行为)

  • be/bd 等值证明:不变,digest 仍由相同 UTF-8 字节算出 ✅
  • ASCII 会话:cu==byte,逐字节不变 ✅
  • bl/bs/bd/be/lcpBytes 消费方:全 display/diagnostic,无计费/闭包/nudge 依赖 ✅
  • 持久化格式/版本:未变——SeamSlot 只在 WeakMap(纯内存,从不落盘);LedgerLine 字段类型不变(bl:number/bs:string/bd:string/be:number),旧行保留历史 display 值 ✅
  • cap 语义:常量值未变,只是从 code-unit 改为按字节量度(对齐注释一直声明的「512 KiB/8 MiB」意图);capped 判定 bytes > CAP 替代 str.length >= CAP,正好 at-cap 不再误判 clipped——measure-zero 且更正确 ✅
  • settle 路径 line 1141 Math.min(prevSlot.full.length, …):.full 现为 Buffer,Buffer.length 即字节数,无需改动即保持字节一致(已核 PR head 该行)✅
  • seamLcp 的 JSON parse 成本:仅在 detectSeam 实质 miss 分支触发(tr>8192 且 >30% input),每调用稳定性检查仍走无解析的 byteLcp,性能画像不变;16 MiB perf guard <5000ms ✅
    高危模式(语义漂移 / fail-fast 变吞错 / 阈值时序输出格式漂移):无意外项,唯一行为变化即下面披露的诊断口径修正。

⑥ 是否破坏缓存
不碰任何缓存/重试/增量游标/去重键。改的是 cache-ledger 的诊断观测,非 provider prefix cache 本身;转发出去的字节一字未变(digest 同源)。provider 侧缓存行为零影响。

⑦ 是否改了设计方向
无架构转向。仍是同一套 seam-slot 机制,只把内部表示从 string 换成 Buffer 承载真实字节——SeamSlot 为模块私有接口(未导出),数据流不变。

⑧ 是否引入新的配置
无。不加 env/config/CLI flag,不改默认值。(PR 正文 Config surface: None,与核对一致。)

行为变更清单(均诊断字段,已披露)

字段 旧 新
bl / lcpBytes UTF-16 code-unit 计数 真实 UTF-8 字节偏移(ASCII 不变)
bs(非 ASCII 对) 按 cu 判定(中×400 对 → head) 按字节判定(同一对 → mid)
SEAM_BODY_CAP(512 KiB) cu(CJK 最多留 ~1.5 MiB) 字节
BODY_FULL_CAP(8 MiB) cu(CJK 最多留 ~24 MiB) 字节——8~24 MiB 字节的正文现只留 digest(bs=unknown)
capped 判定 str.length >= CAP(at-cap 误读 clipped) bytes > CAP(仅真 clipped)

验证证据

  • 新测试对旧代码跑:10 条中 8 条失败——回归用例 expected 'mid' actual 'head';cap 交叉 expected 786486 actual 262196;8 MiB 边界 expected 'unknown' actual 'append' → 证明有牙且 bug 真实存在。
  • 对 PR head:typecheck 干净、build(kernel+pi-subagents+tsup bundle)通过、cache-ledger/seam/fuzz/probe/attribution 定向族 117 passed / 0 failed(含新文件 10 条)。全量交 CI。
  • diff 恰好 2 文件(src/cache-ledger.ts + 新增测试),无 lockfile/version 变动,单个 fix: commit,Closes #2868。
  • 全仓查同机制其余 charCodeAt/codePointAt 点(server asciiCI、chain-checkpoint、affinity-simhash、text-safe、preflight)均为文本通道合法字符操作,无遗漏的同源 bug。

判定:可合。 按 §7.4 auto-merge 门衡量 1–7 条干净通过,第 8 条不触 load-bearing infra。唯一需 owner 知悉:大 CJK 正文字节超 8 MiB 后从「保留全文」变为「只留 digest」——有意、已披露的内存 bound 收紧,属运行时行为变化但不阻塞合并,特此点名。

@ranxianglei

Copy link
Copy Markdown
Owner Author

解决冲突

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 上游合并冲突已解决并推送,PR #2870 现已恢复可合(API 核实 mergeable: true)。

是什么问题:PR 分支与目标分支 master 出现合并冲突。先用 GitHub API 独立核实为真(mergeable: false),不是误报也不是注入。

什么原因导致的:master 在分支基线(merge-base 6580da585)之后前进了 49 个提交。逐文件核对后,真正发生文本冲突的只有 4 个生成物——storage-render golden 摘要文件(tests/golden/storage-render/{anthropic,chat,google,responses}.json):两侧各自重新生成过它们(#2868 因 byte-unit bl 改了 metadata/stateDigest;master 侧这期间的改动让每个 turn 的 sha/bytes 全部变化,如 #2929 的 per-turn 归属字段)。核心代码 src/cache-ledger.ts master 完全没动,零冲突、干净合并。golden 是渲染产物指纹,不能手工拼合,只能对合并后源码重新生成。

影响范围与严重程度:纯诊断/测试夹具层。golden 用于钉住持久化+渲染行为,若手拼错值会让 golden 测试假红;无用户面影响。

怎么修复的:用 merge(而非 rebase+force-push)把 origin/master 并入分支——该分支由另一 agent 持有且可能仍有并发推送,普通 push 天然不会覆盖他人提交,更安全(规范允许 rebase 或 merge)。4 个冲突 golden 不手改哈希,而是跑项目自带的 scripts/update-storage-render-goldens.ts 从合并后源码重新生成;新的 stateDigest 与两个父提交都不同,证明它同时反映了 #2868 的 byte-unit bl 与 master 的 per-turn 字段。

是否有回归问题:无。PR 相对 master 的净 diff 仍是原样的 7 个文件(src/cache-ledger.ts + 新测试 tests/issue2868-byte-lcp-utf8.test.ts + tests/session-transient-memory.test.ts 适配 + 4 个 golden),未把 master 的无关改动混进贡献面。typecheck 干净;定向回归族(golden-storage-render + issue2868 + session-transient-memory + cache-ledger/seam 全家)122/122 通过,其中 golden-storage-render 专门证明重生成的摘要与合并后的实时渲染一致。

是否破坏缓存:不涉及。未触碰任何缓存/重试/游标/去重逻辑;#2868 本身只是诊断字段字节化(ASCII 逐字节不变)。

是否改了设计方向:不涉及。仅为解冲突加了一个 merge 节点,架构/边界/数据流无转向。

是否引入新配置:不涉及。无新增 env/config/CLI。

处置说明:本地 typecheck + 定向族全绿后,先复核远端 head 仍为父提交 5b2a92d0(防覆盖并发推送),再普通 push(fast-forward 5b2a92d0..f39aa6766)。未 force-push、未碰 master、未合并 PR(合并仍归人)。当前 API:head=f39aa6766 base=1f936a3c mergeable=true。

判定:冲突已清除,PR 恢复可合,等待人工合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 已核实:当前没有需要解决的冲突,PR #2870 处于可合状态。

上一轮我已把 origin/master 并入分支并推送(5b2a92d0..f39aa6766,fast-forward),当时即确认 mergeable: true。刚才对最新 master 尖再次独立验证:

  • 本地 git merge-tree 非破坏性预览:HEAD(f39aa6766)× origin/master(9fb79ddd)合并干净、零冲突(产出树 d2b20395)。
  • GitHub API:mergeable: true,head=f39aa6766。
  • master 自上次基线后新增的 3 个提交(#2979 release-gate-local:package.json / reference/release.md / scripts/release-gate-local.mjs)都不触碰本 PR 的文件(src/cache-ledger.ts、4 个 storage-render golden、两个测试),因此不可能重新引入冲突。

结论:冲突在上一轮已清除,现在无需再做任何冲突处理。若你界面仍显示"有冲突",应为缓存未刷新——以 API 的 mergeable: true 为准。PR 就绪,等待人工合并(我不代合)。

@ranxianglei
ranxianglei force-pushed the 2026-10-12_byte-lcp-utf8 branch from f39aa67 to b60d468 Compare October 13, 2026 16:23
@ranxianglei

Copy link
Copy Markdown
Owner Author

冲突 rebase master

ework-agent added 4 commits October 14, 2026 17:55
The retention assertions now read the UTF-8 encoding back out of the
Buffer slot (byte-equal for valid UTF-8 bodies), and a new test pins
that a lone surrogate is retained as U+FFFD — it has no UTF-8 form.
Heap-probe thresholds unchanged.
@ranxianglei
ranxianglei force-pushed the 2026-10-12_byte-lcp-utf8 branch from b60d468 to 5188102 Compare October 14, 2026 10:07
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

已按要求把分支 rebase 到最新 master(线性历史,去掉之前那个非线性的 merge 节点),验证通过、可合。

做了什么

验证

诚实提醒:master 这几分钟一直在动(push 时基线 51aa9f1fd,随后又进 9db9cb1cd=#3117/#3100 dsh-stale-proxy-guard)。那条新提交只动 dsh-native-client/admin/settings,不碰本 PR 任何文件,所以分支此刻仍干净可合。若合入前 master 再落一个 golden 重生成提交,这 4 个夹具可能需例行重生成一次——但现在是 mergeable 的。

八问

  1. 什么问题:PR 分支落后于快速推进的 master(behind/冲突态),你要求 rebase 成干净的线性栈。
  2. 什么原因:master 高频合并(含多次 storage-render golden 重生成 + kernel 0.0.122→0.0.123),分支基线停在旧点,双方都在重生成同一批 digest 夹具 → 冲突被限制在那 4 个生成文件上。
  3. 影响范围/严重度:仅本分支可合并性(CI/门禁),对用户零行为影响(byte-LCP 修复本身未变,早前已 review 判定)。合并前是阻塞项,现已解除。
  4. 怎么修:线性 rebase(非 merge)到最新 master,保留另一 agent 更完整的提交集;唯一冲突用「重建最终树后重新生成」解决;守卫式 force-with-lease push。
  5. 回归:无。typecheck 干净 + 125/125 定向测试绿;除已 review 的诊断字段外无行为变化。
  6. 缓存:不涉及(纯 rebase + 夹具重生成,未触 cache/retry/cursor/dedup)。
  7. 设计方向:不涉及(纯 rebase,架构不变)。
  8. 新配置:不涉及。

判定:可合,等人工合并(我不自行 merge PR)。

@ranxianglei
ranxianglei merged commit 8d54a3e into master Oct 14, 2026
24 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.416-master.8d54a3e (rolling master tag).

npm install -g billion-context@0.0.416-master.8d54a3e  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] BODY STABILITY 的 byteLcp 实际统计 UTF-16,导致非 ASCII 字节偏移和 head/mid 分类错误

1 participant