Repository navigation
[PR] fix(#4797): root-cause fix for superseded live-key collision — restamp pruning + semantics-preserving guard translation | fix(#4797): superseded 活键碰撞根治——restamp 剪枝 + 守卫保义翻译 + 一次性标志修复 (kernel 0.0.166) - #4807
Conversation
…mp map, identity-preserving guard translation, flag-safe clones Root cause (phase-C audit, reference/ref-keyed-surfaces.md): the one-time ledger-alignment restamp vacates old ref numbers and hands OTHER messages seq+1 targets in the same sweep — in a dense space a superseded key is routinely ALSO the live ref of a different message (exactly the numbers acp_status advertises). The compress-input guard consulted superseded before the live map, so it refused every advertised range and pointed the caller back at acp_status — a structural dead loop; the proxy's own preflight rescue was refused by the same gate. Additionally the kernel's syncBlocks/cloneState dropped refLedgerAligned every turn, re-arming the restamp and keeping the stale superseded map alive for the process lifetime. Kernel (0.0.165 -> 0.0.166): - restampRefsToLedger: post-pass prune of superseded keys that are live in the final map; block startRef/endRef and deadRefs now ride the same migration (identity-preserving translateRef via the pre-pass map) - syncBlocks clone + cloneState carry refLedgerAligned (one-time stays one-time) Host: - supersededRefRejection reworked to a resolution ladder per ENDPOINT (the interior span-scan only manufactured false hits): live-first keep; vacated -> translate oldRef -> rawId -> same message's current ref; refuse only dead-end cites. applyRanges proceeds with translated ranges (logged) - maybeRestampToLedger clears session.restampSuperseded on an empty outcome (a converged pass must not leave the previous map in place) Tests: kernel ref-restamp-collision (3, RED on master) + seq-restamp S13a/S13b re-pinned to the pruned-map contract + host issue4797 T1-T4 (T1/T2/T4 RED on master in an isolated master worktree); kernel suite 1483/1483; typecheck 0.
…y residual) Review of #4807 found the deadRefs migration in restampRefsToLedger is a no-op for organic tombstones: a tombstone's number is absent from the PRE-pass map by definition, so translateRef leaves it verbatim — even when this very pass hands that number to another message as a seq+1 target. The stale tombstone then sits on a LIVE number and excludeDeadRefRanges / markNoMergeAcrossDeadHoles over-filter ranges around it (same collision band as superseded; safe direction, wrong behavior — exactly what the registry doc flagged for this surface). Drop tombstones the final map vends as live; empty ⇒ undefined per the #3302 discipline. Re-pins the migration test (its old expectation encoded the stale-tombstone end state) and adds organic pins: re-granted dropped / genuinely-free kept / all-re-granted ⇒ undefined. Rides the unreleased 0.0.166 bump already on this branch.
…ed in this PR The audit table listed blocks[].startRef/endRef and state.deadRefs as NOT yet migrated, but this PR's restamp pass translates both (kernel/src/refs.ts translateRef pre-pass map, pinned by kernel/tests/ref-restamp-collision.test.ts). Move them to the migrated table so the registry matches shipped code; the remaining rows (hiddenOrphanRefs, latches, messages.ref_num) stay open.
(ai: vllm-qwen/qwen3.8-27b) PR review 结论(独立审核,代码级逐段验证 + 本地定向测试) 查重结论见首评(与 #4799 同族非纯重复,机制覆盖不同,处置待 owner 拍板)。本轮完成:kernel/host diff 逐段通读、根因手工推演复现、既有行为清单回归审查、定向测试、一处文档小修。 ① 是什么问题:代理重启/LRU 驱逐触发 ledger-alignment restamp 后,压缩输入守卫把「活编号误判为已重编号」——acp_status 广告的区间被 compress 拒绝,会话永久压不了缩(p1 死锁,#4797)。 ② 什么原因导致的(逐条在代码里验证成立):
③ 影响范围与严重程度:p1。任何跨过触发边界(代理重启、unified 冷载、strictly-ahead 恢复、会话中途 LRU 驱逐)且旧编号被重排的会话,压缩能力永久失效——包括代理自己的 preflight 自救也被拒;未跨边界的会话不受影响。 ④ 怎么修复的(机制层面):
⑤ 是否有回归问题(既有行为清单逐项过,无无意回归):
⑥ 是否破坏缓存:不涉及。前缀缓存策略不动——出站 tags 仍由 live messageRefs 经 render 管道派生(已核 render-refs/byRaw 路径);restamp 仍是一次性事件,出站字节不变。 ⑦ 是否改了设计方向:无架构转向。这是既有 ledger-alignment 机制内的根治(剪枝 + 保义翻译 + 标志传播)+ 防再犯登记文档 reference/ref-keyed-surfaces.md(kernel Boundary 合规:PR body 有 kernel-change 声明 ✓、版本 bump 0.0.165→0.0.166 ✓)。因触碰 kernel/,按 §7.4 必须人工合并。 ⑧ 是否引入新配置:无。无新增 env/config/CLI flag,wire 形状不变。 验证证据(本地只做定向,全量交给 CI):
顺手小修(独立标注提交 a783834,docs only):reference/ref-keyed-surfaces.md 把 blocks[].startRef/endRef 与 state.deadRefs 两行仍列在「NOT migrated」表,与本 PR 实际迁移两者(b72c291 也只改了状态列没挪行)不符——两行移入 migrated 表,deadRefs 行采用作者精确措辞(含 live-drop + #3302),删旧行。无代码改动。 与 #4799 的取舍对比(供 owner 拍板):验收面上 #4807 严格占优——真旧引用 #4807 翻译成同一条消息的新号、折对消息,#4799 直接拒;活号引用两者行为相同(都放行到当前 owner,也都自认同样的 replay 窗口暴露)。#4807 另有 kernel 剪枝(未来碰撞不再产生)、block/deadRefs 迁移、标志携带、clear-on-empty;#4799 独有 facet(restamp 时把腾出未重授号登记进 deadRefs)在 #4807 阶梯下不必要(stage 3 让未知号落到 kernel boundary 处理)。若 #4807 合入,建议 #4799 关为 duplicate(owner 决定,我不关)。 判定:方向对、实现验证通过、测试有牙(正负例齐全、master 红证据成立)、diff 干净(11 文件 +503/−58,每行与目的相关)。可合——但触 kernel/,需人工合并。 |
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.6229-pr.4807.c6000cfThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4807.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…ract Regenerated via scripts/update-storage-render-goldens.ts after a clean build. Wire bodies byte-identical (the restart-invisibility contract holds); only persisted-state digests moved — the superseded map is now liveness-pruned and block spans/deadRefs ride the migration (kernel 0.0.166), so the aligned state's digest changes by design. Same justification class as the #4599 wire-parity re-pins.
(ai: vllm-qwen/qwen3.8-27b) 结论:无需我动手——这个 lock-sync 红是旧 head 的,已被 PR 后续提交修掉,当前 head 已绿。 报过来的失败是在 head 验证证据
八问
处置判定:无需开新 issue(非继承性 master 红、无设计决策、问题已自解),也不做无意义的空提交去刷 CI。等当前 head 剩余 test 分片收尾转绿即可正常推进合并流程。 |
…— prune/migration deferred to the storage shadow layer Bisection result (see PR discussion): the kernel-side superseded liveness prune + block/deadRefs migration broke the storage-engine restart-invisibility contract — the shadow digest comparison flags the migrated state as 'unified replay != legacy load view [diverged: state]', drops the ingest watermark, and the full-mode leg falls back to the pre-alignment legacy envelope, drifting the wire (storage-engine-proofs 4x red, golden wire parity). With master kernel + this branch's host guard the same suites are 12/12 and 17/17 green. The deadlock itself is fully resolved host-side: the guard's per-endpoint ladder consults the LIVE map first, so a superseded key that is also a live ref never blocks a status-advertised range (issue4797 T1-T4 green, T1/T2/T4 verified RED on isolated master). The kernel sweep (prune + block/deadRefs migration) stays registered in reference/ref-keyed-surfaces.md as deferred — it needs the storage shadow layer to hash the aligned state shape first. Kept kernel changes (0.0.166): syncBlocks/cloneState carry refLedgerAligned (the one-time alignment stays one-time across turns) — golden storage-render google.json re-pinned for that state-shape change (wire bodies byte-identical). seq-ledger-pr3 S14 re-pinned to the resolution-ladder contract (translate / dead-end / live-first); S13/S15 revert to master expectations.
…erdict; #847/#1492B contracts reconciled to the #4288 world Both files were red since the #4725 merge (verified at 931c66a in an isolated worktree): the re-landed #4288 exit fired at round top, but the final verdict block still stamped an exhausted failure whenever currentTokens — floored by the same stale baseline the exit exists to bypass — read over-limit. A fit-forward walk then reported 'no viable range could be compressed' while the caller forwarded the fitting payload anyway: the marathon was prevented, but the reported verdict lied. - preflight.ts: fitExitForward flag — a round-top fit exit suppresses the final failure stamp (aborts still win). - #1492 B: contract updated to the #4288 doctrine — a usage-grade baseline above the window on a FITTING payload forwards; a usage baseline above the limit cannot have been measured under this window regime, and a real overflow arms the one-shot rescue (#1110/#987) on rejection. - #847: the honest-exhaustion pin now covers its actual post-#4288 meaning — the fitting sub-gate payload exits fit-forward with no verdict; the exhaustion path stays for payloads that genuinely exceed the target. issue847 4/4, issue1492 3/3, issue4288 3/3, issue1195 4/4, preflight family 21/21, 4797 family 4/4, typecheck clean.
|
(ai: zhipuai-lb/glm-5.3) CI 状态说明(head 9384074)本 PR 引入过的失败已全部转绿:unused-exports、lock-sync、storage-engine 线缆对等 ×4(kernel 剪枝/迁移撤回后 12/12)、golden 重钉、#847/#1492B(根因:#4288 出口生效但收尾判决被陈旧托底 currentTokens 覆盖—— 剩余 2 个红分片不是本分支引入:issue4125-giveup-reason(deterministic,5ms)与 dsh-native #1365(15s 超时)在 master 单跑同样红,是 CI 分片重排后被暴露的既有红。已开 #4824(证据:归因到 bc50f1a 的尾部 kernel 部分本来就是人工评审层级——是否在 #4824 修复落地前合并本 PR(或先修 #4824 再重跑),请 owner 判断。本地:kernel 1480/1480、storage-proofs 12/12、golden 5/5、issue847 4/4、issue1492 3/3、issue4288 3/3、issue1195 4/4、issue4797 4/4、typecheck 0。 |
… branch reason must survive the callback bc50f1a (#4516) appended state.lastUnavailableReason = "dead" right after onGiveUp fired with "send-failed" — the dsh give-up client-event line reads that slot AFTER the callback to name the branch that fired, so every send-failure give-up reported 'dead'. The steady-state degraded-mode visibility belongs to #4516's dispatch-mode tracker (noteDispatchMode already logs the degrade), not to overwriting the branch reason. Both send-failure sites dedented; issue4125-giveup-reason 5/5 green (was deterministic red on master and on this branch's CI shards). #1365 (dsh-native attach family) verified NOT a master regression: master's latest CI run (cdac1c0) ran the suite green — the family hangs only on this dev box (node 25.9; zero fetch calls reach the mock — registration never issues its manifest probe) and had one razor-edge flake on a CI shard (waitFor's 15s internal deadline == attach health deadline). Filed as environment-artifact evidence on #4824.
…ize refusal wording The #4125 stomp removal exposed a load-bearing consumer: #4516's fallbackOversizeRefusal picked the 'died and no replacement came up' wording only for reason "dead" — a send-failure give-up (proxy WAS live, died mid-session) read as 'did not come up'. Both contracts now hold: the reason slot keeps the branch name for the fire-time consumer (#4125), and the steady-state wording maps send-failed onto the died branch (it did die). issue4516 11/11 + issue4125 5/5 green; typecheck clean.
…the Windows AV-lock race at file level Two consecutive CI runs (windows-22) show the SUBTEST green (3.4s) with the FILE marked red: the after() hook's bare rmSync threw on the transient EBUSY/EPERM that tests/tmp-rm.ts exists to absorb (#1646/#1910/#2908 ladder). Same class as the plugin-fork bounded-wait fixes in the #4599 sweep.
…d index.db handle was the real Windows EBUSY The rmrf swap was necessary but not sufficient: the harness closes proxy and mock upstream but never store.closeQuietly(), so sessions/index.db stayed open and the after() rm retried ~9.6s into a permanent EBUSY (hookFailed, subtest green). Same handle-discipline as the plugin-fork/#4599 sweep.
(ai: vllm-qwen/qwen3.8-27b) 补充一条时效更新(上面那条报告基于 head
合并时机与 kernel 的人工评审归 owner 决定,我不代判。就 CI 而言:deterministic 阻塞项(#4125/#4516)已随分支修好,残留只有 #1365 的偶发抖动(必要时重跑一次);是先等 #4824 加固落地再合、还是本 PR 先合,请 owner 拍板。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.852-master.e290b4f # this exact build
npm install -g billion-context@master # always the newest merged build
|
Labeled follow-up fixes for comments that contradicted code facts after #4807's kernel liveness prune was reverted: - reference/ref-keyed-surfaces.md: the contract section listed the reverted prune as enforced, and the 暂缓 section claimed host live-first '已消除其全部用户可见危害' — disproven by #5199 (isolated fixture reproduced the silent wrong-target application). Both now say what is actually true and point at #5199/#5204. - tests/issue4797-superseded-deadlock.test.ts: header comment described the post-prune superseded shape ({m3→b, m7→f}); master produces {m3→b, m6→e, m7→f} where m6 is the dual-identity case now pinned in tests/issue5199-dual-identity-log.test.ts.
…o messages (#5196) Cold-load ref reconciliation (maybeRestampToLedger, #4316 PR3/3) aligns refs to ledger row order seq+1. When mover A stamped its seat in one pass, the slot A vacated became immediately visible as free to a later mover B in the same pass — B took over A's pre-restart number while superseded[thatRef] still named A. The live-first resolver (src/stream.ts supersededRefRejection) then re-targeted every stale citation of that number at B: a compress() citing A's pre-restart ref folded B's contents behind A's summary text, HTTP 200, block created (0.1.209 field fixture). restampRefsToLedger now retires every seat a mover vacates into its own seq+1; any mover whose target is a retired seat parks on a fresh number at/ after the cursor via the existing displaced/rehomed path instead. A retired slot can never be targeted again within the pass (targets are unique per row), and park numbers always start above every used index, so no freshly stamped or parked ref can carry two identities — live-first becomes safe by construction on kernel-produced data, with zero new rejection surface. Pre-#5196 ambiguous data keeps the documented #4807 live-first trade. - kernel 0.0.173 → 0.0.174 - tests: seq-restamp S13b rewritten to parking semantics (a swap's second side parks, superseded carries both old mappings), 3 new regressions: the exact issue fixture shape, a vacate chain proving untouched seats stay reusable, and cold-load persistence of the parked layout - src/stream.ts: refresh the stale 'kernel prune' doc paragraph left over from the rolled-back liveness prune (#5199 / af1f92a) Verified: kernel seq-restamp/ccr-restamp/dead-refs/dead-ref-split/ issue3939 23/23, host seq-ledger×2/issue4316/issue4797 13/13, e2e-trae-model-restamp 1/1, kernel+root tsc clean, unused-exports gate clean.
(ai: zhipuai-lb/glm-5.3)
Fixes #4797
kernel-change: #4797 根治——ledger-alignment restamp 的 superseded 活键碰撞(owner 在 #4797 讨论中指定的 C→A→B 推土机方案,kernel 侧为其中 A/B 的必要部分)
一句话
代理重启(或 LRU 驱逐)触发一次性编号对齐后,压缩输入守卫把「活编号误判为已重编号」,会话永久压不了缩;根子是 superseded 表与活编号稠密碰撞 + 一次性标志每轮被克隆丢掉。
根因(phase-C 全量审计,两路并行)
restampRefsToLedger腾出的旧号同轮被重用为别家消息的 seq+1 新号 → 稠密空间里几乎每个活编号同时是 superseded 键;守卫查表先于查活表,span 扫描命中即拒 → acp_status 吐同样的活号 → 结构性死循环(连代理自己的 preflight 自救也被拒)。syncBlocks/cloneState克隆丢refLedgerAligned→ 每轮重排;host 只在 superseded 非空时覆盖 → 陈旧表进程内永生(inline 宿主一活几天)。getSession内存 miss →loadSync→ afterLoad 同样触发重排,「重启边界没有在途调用」的假设不成立。reference/ref-keyed-surfaces.md(所有按 ref 键控的面 + 迁移状态 + 防再犯清单)。修法(C→A→B)
refLedgerAlignedsupersededRefRejection改为按端点解析的三级阶梯——活表优先保留 → 真空置号保义翻译(旧号→rawId→同一条消息的新号)→ 翻译死路才拒绝;applyRanges用翻译后的区间继续并留日志;收敛后的空结果清除陈旧 map验证
seq-restampS13a/S13b 按剪枝后契约重钉(S13a 原断言恰恰钉住了碰撞本身——审计确认它是 After model switch / client-side history rewrite, all message identities (content hashes) are replaced: ACP numbers re-m | 切模型/客户端重写历史后,消息身份(内容哈希)全部更换:acp 编号重铸、折叠覆盖大规模重挂、死 ref、churn 区临时膨胀重折 #4316 时代就存在的结构性盲区)issue4797T1–T4:T1(acp_status 广告的活区间被接受并折叠,即死锁本体)/T2(真旧号保义翻译)/T4(死路诚实拒绝)在隔离的 master worktree(独立 node_modules + kernel build)上全红issue3157-persist-durability在 master 上同样红(预存在,与本分支无关,见 CI)边界