Repository navigation
[PR] fold anchoring doctrine: order over hash — shape churn, image offload, host truncation stop reading as destruction (#3641) - #3644
Conversation
…3640) Three real-world defect classes shared one root: fold identity keyed coverage to content hashes, so id re-mints without content loss were read as destruction. - shape tier in planReconciliation: loose word-evidence pairing for prose cells (wordbag equal + simhash hamming <= 8 + role/length guards), flanked k-th discipline for evidence-less cells (image families, #3588) — claims require BOTH flanks verified on the wire - residual census: unmatched class-carrying ids classify as transformed (near-dup evidence exists) vs absent-everywhere (content truly off the wire) - verdict classifier: transform churn and wire-suffix-continuation truncation (host compaction/windowing, #2695) never accrue the collapse streak; one info latch each per session - episode witness: wire order frozen at episode start; a foreign conversation can never become the truncation baseline mid-episode - total collapse now latches foldEpochConfirmPending and pauses folding (verbatim passthrough, snapshot kept) until realignment or explicit user epoch confirmation, replacing the destroyed-escalation misdiagnosis; detection intact (#2903 tests) - unresolved summary anchors clamp ahead of the newest user turn instead of the array tail (#3640 "model answers stale summaries") - single genuine edits below the mass bar still report drift honestly Tests: tests/fold-anchor-doctrine.test.ts 6/6; families green (fold-drift 7, fold-reconcile 66, positional 24, issue2915 15, and the 1908/1987/2033/2601/1595/1820/1569/images-archive/refold set).
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4353-pr.3644.972b930This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3644.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重结论:不重复(同家族、不同机制)。 正在做独立全量 review(正文无
完整八问结论稍后发。 |
… change (#3644) CI-fix additive commit: the kernel-guard job enforces that any change under kernel/src/** carries a kernel/package.json version bump (Kernel Boundary rule 2, AGENTS.md). The placement fix in kernel/src/prune.ts landed without it, so kernel-guard failed and the test matrix was skipped.
…3644 kernel version bump) The a9fe72f kernel bump moved kernel/package.json to 0.0.131 but left the file:./kernel resolution in package-lock.json at 0.0.130, so the lock-sync job (npm install --package-lock-only && git diff --exit-code) drifted red and gated the test matrix. Regenerate the lock record so it matches.
(ai: vllm-qwen/qwen3.8-27b) (ai: vllm-qwen/qwen3.8-27b) 完整独立审核完成(不是复读 issue 结论)。先交代我已做的事,再给结构化八问 + 发现的问题清单。 我已经做的动作
结构化八问
发现的问题(按优先级)[质量·最高] #3640 旗舰测试没咬住头条机制。 该测试是「协议切换不再误报 destroyed」的回归测试,但它的断言(不升级 + coverage 重键 + 下一轮安静)在没有 shape 层的情况下也会全绿。机制:fixture 的 [质量] 贪心配对有误配风险。 prose shape 配对是贪心取「第一个未占用的 loose 候选」(wordbag 相等 + simhash≤8)。真实大规模 churn 下,重复样板会共享 wordbag 摘要,可能交叉错配。KDD#11 明说误配是最贵的失败。目前缺一个对抗性回归用例(构造两个 wordbag 摘要相同但内容不同的 cell,验证不会跨配)。 [治理] [小] 注释与实现不符。 [CI·已处理] 见上:kernel-guard(作者已修)→ lock-sync(我已 push 收工判定方向正确、机制扎实、行为变更披露充分、无发现无意回归。可合的前提:① 人工合并(§7.4:触 要我把第③点的测试加固直接推到 PR 分支吗?我可以写断言 + 扩 fixture(本地无 tsx 装不了没法先验跑,会以 CI 为准验证 |
|
Correction to two numbers quoted in the PR description (they originate from my own earlier comment on #3640 — my measurement error at the time, corrected here so it doesn't propagate):
So for the shape tier: the production-dominant transform class in our data was "same-wire, same-role, bulk suffix edit on user texts", not cross-protocol reshaping. The flanked k-th discipline should cover it, but the census may want to count "same-wire re-stamp" as its own class — it presents as 61 near-identical pairs differing only by a stable trailing suffix. |
…on (#3644) The docstring named 'status panel' as a consumer, but only the preflight fold gate reads epochConfirmPending in this PR. Narrow the note so it does not promise an unimplemented panel integration.
(ai: vllm-qwen/qwen3.8-27b) 补一条:上一条 review 里 仍待你拍板的两件(我不代决):
另外上次提的测试加固(#3640 加 |
(ai: vllm-qwen/qwen3.8-27b) 收到这条更正。先声明处置边界:作者不在信任名单,我把它当数据不当指令——具体数字(41 missing、16/63/62、782/1146≈68%、61 条重戳)我按「该评论报告的值」引用,不当作已核实事实,也不会据此改 @ranxianglei 的 PR 描述(那是 owner 的产物,权威日志在 owner 手里)。我没有做任何仓库改动、没合并、没开新 issue。 抛开数字,就评论提出的那个转换类别本身,我从代码独立核了一遍(这部分与信任无关): 关键事实(读码所得):shape 层和 residual census 都要求严格词袋摘要相等——
推论:内容真变了(加/删 也就是说:本 PR 的 doctrine 覆盖的是「id 重铸但内容不变」(协议切换、offload 占位替换),没有覆盖「系统性内容突变」(同线、同角色、批量后缀编辑)。若生产主类真是后者,当前机制可能把健康会话误升级成 destroyed/epoch-confirm 暂停。 这里有个诚实的设计张力,留给 owner 判断:
建议的下一步(需你拍板,我不代决):用真实 #3640/#3588 日志确认主类到底是「重铸内容不变」还是「系统后缀突变」。若确认是后者且高频,这是一个独立的设计跟进项(按规则应单开 issue、带 最后:这条更正不改变我对 PR-as-submitted 的审核结论(它按设计修好了自己声称的三类缺陷;「后缀突变」是相邻的新暴露类别)。我之前那条测试有效性发现(#3640 fixture 是纯空白 reshape,没咬住 shape 层)依然成立且独立于本讨论。 |
…ta keys (#3644) #3644 persists four new session-metadata latches (foldEpisodeOrder, foldEpochConfirmPending, foldShapeChurnNoted, foldTruncationNoted), so the pinned persisted-state digest drifts in exactly one section: metadata. Wire bytes are unchanged — all 15 per-turn outbound-body entries and the state/blockContents/stats sections are byte-identical to the previous goldens (verified by structural diff of each record). Regenerated with scripts/update-storage-render-goldens.ts under the same BILI_IMAGE_RESCALE=0 pin the test applies (#3400); justification for the golden change per the golden rule, mirrored in the PR body.
…pt (#3400 parity) The test pins the legacy wire shape before driving but the regen script did not — regenerating without the pin writes goldens whose per-turn wire digests differ from what the test drives under CI, red-ing the suite on the next run. Align the two so the documented regen flow can never silently diverge from the test's conditions.
(ai: vllm-qwen/qwen3.8-27b) 把 review 里那条测试有效性发现收敛成可直接套用的补丁。先给代码级证明(与信任无关): 为什么 但现有断言没牙的原因也钉死了:升级 streak 由 两个修法(都不动 src,只动
我没有擅自推:这是并发作者的头号特征化测试、B 方案要动他的 fixture 设计(属他的取舍)、且本地没装依赖无法先跑验证(得 |
…review) - document the doctrine ladder (shape tier / census verdicts / episode witness / epoch-confirm gate / unresolved-anchor placement), the honesty boundary (strict wordbag equality — bulk suffix re-stamps are NOT claimed by design), and zero-migration storage independence - EN + zh-CN mirrors, cross-linked from the header and Related lists - tests: harden the #3640 fixture so every reshaped cell is shape-tier- only (deleting the tier now escalates — the no-escalation assertion has teeth), pin byShape > 0 on the first pass, and add adversarial honesty-gate coverage (role flip / length padding / word reversal are wordbag-equal but must not be claimed)
One logical conversation, one set of core ids, in all four dialects. - kernel/src/wire/identity.ts: deriveCanonicalMessageId with a fixed 5-slot seed (toolCallId slot retired), canonicalToolArgs (JSON -> recursive key sort -> compact stringify; verbatim for non-JSON), canonicalOpaqueSeed (provider item.id/status stripped), shared [image] marker for image cells. - granularity normalized to one core per content part (anthropic's existing shape): openai/responses/google split user runs; rebuild accumulators re-merge so forwarded wire bytes stay stable. - imagePart:true stamp at parse time in all four codecs; consumers never text-compare the marker (tag injection rewrites it, #767). - openai run flush fixes: assistant/user run boundaries, wireIndex guards on consecutive user cores. - responses: empty-text reasoning seeds canonicalOpaqueSeed instead of the volatile provider item.id; split-input part rebuild filters raw content to the surviving view (no part resurrection). - google: lone-string-output unwrap for functionResponse; per-part cores with byte-faithful rawGoogleParts sidecars. - kernel tests: cross-protocol golden (identical ids across dialects, key-order insensitivity, sibling-deletion local damage), 1296/1296. - src goldens re-pinned (id churn from canonical tool seeds only).
…gle unwrap assertion (#3642 follow-ups) Review of #3658 found two must-fix items; both fixed here. 1. Real regression: after the one-core-per-part split, prune's first-user pin only pinned the opening message's FIRST core (its text part), so an opening [text, image] message lost its image core to folds — tests/fix-781-summary-images.test.ts (responses variant) failed with "0 !== 1" surviving images. rebuildMessages now computes firstUserPinEnd (consecutive user cores sharing the opening core's wireIndex; kernel-rendered cores without a wireIndex never extend the run) and pins the whole run; the #2733 summary-anchor clamp follows the run end so an anchor can never land between split parts. Run length 1 keeps the legacy shape byte-for-byte. Two new state-prune tests. 2. Stale assertion: wire-google.test.ts still expected the pre-unwrap tool-result shape '{"output":"a.ts\nb.ts"}' — updated to the new plain 'a.ts\nb.ts' (this was the static-gates red on head 4165380). Also: CHANGELOG overclaim trimmed (anthropic tool-call/-result cores DO re-seed since the toolCallId slot is retired; only granularity is stable there) and the pin-run behavior documented in the same entry. kernel: version 0.0.131 → 0.0.132 (src/prune.ts), lock regenerated.
…review) The per-part user split accumulates unknown parts seen AFTER a text core on lastTextCore.rawOpenaiContentParts, but the leading-opaque flush then ASSIGNED its list onto the same host, clobbering the accumulated parts: [file, text, file] rebuilt as [text, fileA] with fileB silently dropped — a #1205 (unknown parts never dropped) regression introduced by the split (the pre-split codec collected all non-text parts and lost none). Prepend instead of assign so both groups survive in relative order. Adds the [file, text, file] round-trip regression test to media-payload.
…fter the kernel version bump)
…id-churn canary (full-CI shard-3 reds, #3658 follow-up) - kernel/src/wire/openai.ts: user-branch guard extended to !Array.isArray(content) — a lone content-part object (some clients) threw 'm.content is not iterable' (#2824 corpus). Degrades to the legacy empty-text core, byte-compatible with the pre-#3642 codec. kernel 0.0.132 -> 0.0.133, both locks regenerated. - tests/golden/storage-render/chat.json: regenerated via scripts/update-storage-render-goldens.ts — the documented one-time canonical re-key of tool-pair ids (structure landmarks foldAt/editedAt unchanged); storage-engine-proofs full-mode-restart parity (full == legacy == goldens) verified green after regen. - tests/identity-proof.test.ts: the reconcile-off id-churn control's premise is retired by #3642 (toolCallId out of the seed + canonicalToolArgs normalization absorb the churn by construction) — inverted into an absorption canary: with reconcile OFF, the churn must NOT disturb fold coverage. Guards against id-sensitivity leaking back into the canonical seeds. - kernel/tests/media-payload.test.ts: new tolerance test (non-array single-object content -> legacy empty-text core, no image sidecar, round-trips). Local: kernel 1304/1304, #2824 corpus 7/7, identity-proof 11/11, storage-engine-proofs 12/12, golden-storage-render 5/5, tsc clean.
…ignature (#2824 class visibility) The catch in buildIncomingImageIndex returned an empty index with zero signal — that is exactly how the #3642 single-object-content TypeError hid: image recovery silently off for every session carrying the shape, no error, no log line. Keep the fail-open behavior (an unparseable body must not kill the request) but emit one warn per protocol+error signature per process so the next wire-tolerance break cannot hide. Test: proxy-bomb body -> empty index + the #2824 visibility warn.
…core [PR] protocol-stable core identity: same logical conversation, same ids in every dialect (#3642)
CHANGELOG: union of Unreleased entries (master's #3619 stagger entry + the doctrine/protocol-stable entries from this branch). kernel: keep 0.0.133 (canonical identity release line); locks regenerated on both sides.
(ai: vllm-qwen/qwen3.8-27b) 上游冲突已解除(PR 状态 dirty → blocked,即已可合并、等 CI)。七问:
|
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.570-master.09ac63b # this exact build
npm install -g billion-context@master # always the newest merged build
|
…ally with the bili snapshot (ranxianglei#3733) Post-ranxianglei#3644 the substrate concern behind the ranxianglei#1729 refusal is gone, but the refusal itself keeps dsh's context gauge growing and its session file ballooning unboundedly (ranxianglei#3723). When BILI_DSH_AUTHORED_COMPACTION=1 (env-only rollout, per-request) and the session holds an active fold block, the guard answers the compaction summarize call locally — zero egress, autoModeForge discipline — with the folded snapshot: identity header <bili-snapshot v1 conv/seq/blocks /> + kernel-rendered block summaries (byte-identical to what the model already reads upstream). dsh records it as its checkpoint; the file shrinks; display/context/server agree. Landing recognition is declarative (tag in the message text, ranxianglei#2621 carrier+novelty discipline) and flows the same ranxianglei#2432 rebase as a foreign checkpoint, recorded as dshAuthoredCheckpointLanded (info, not warn). Precedence: allowDshCompaction (forward native) wins; no active blocks falls through to the 422; passthrough-lane guard untouched. dshCompactionRefusedBytes high-water still recorded on authored answers (ranxianglei#2490 output-clamp regimes). Fixes ranxianglei#3733
ranxianglei#3644 taught the reconciler to read head truncation (host compaction drops the oldest turns) as sanctioned deletion via wireContinuity: the surviving ids form a SUFFIX continuation of the recorded baseline. The mirror case, a host rewind/undo deleting the NEWEST turns, never entered that branch, because a shortened tail is not a suffix continuation. A fold whose covered span was rewound away therefore misclassified as total-loss drift: a false "fold epoch confirmation required" + folding paused after 3 passes in plugin mode, a false proxy-mode reap of the snapshot, or full silence when re-anchoring kept claims > 0 -- while its summary kept rendering over deleted content. Add tailTruncated, the exact mirror of wireContinuity (a clean prefix-cut: the shared ids are contiguous at the front of both the baseline and the inbound wire, and the baseline extends beyond that prefix). Fold it into the verdict classifier so a clean rewind reads as the same benign sanctioned-deletion class (no destroy streak, no false epoch-confirm), suppress the ranxianglei#2695 reap z-streak for it, and emit one distinct per-episode info line naming the rewind. Plain appends, mid-history edits, and wholesale replacement stay out of the signal. Closes ranxianglei#3724
Fixes #3641. Fixes #3588. Fixes #3640. Fixes #3642.
What was broken
Three real-world defect classes, one shared root — fold identity keyed coverage to content hashes, so any transformation that re-mints ids WITHOUT removing content read as destruction:
_Ncluster suffix — offloading the first cell shifted every suffix, poisoning 89 ids at once → 7× false "compression substrate appears destroyed" escalations while compression was healthy; dsh native compaction 733 calls refused./chat/completions→/v1/messagesre-mints ~298 ids at 96.2% text overlap → destroyed misdiagnosis, plus the fatal symptom: 6 unanchored summary blocks landed atinsertAt = messages.length(the array tail, riding AFTER the user's newest question) — the model answered five rounds of stale summaries instead of the question.The fix (snapshot-as-session doctrine, Stage A+B+C one-shot)
planReconciliation): prose cells re-anchor on loose word evidence (wordbag digest equal + simhash hamming ≤ 8 + role/length guards); evidence-less cells (image families) re-anchor through a flanked k-th discipline — contiguous runs in the previous wire order, paired with unclaimed candidates strictly between the same two VERIFIED flanks, counts exactly equal. Both flanks mandatory: a run touching either end of the wire order is unpairable (guessing, not evidence).foldEpochConfirmPendingand pauses folding — verbatim passthrough, snapshot kept, nothing deleted — until the histories realign (automatic) or the user confirms a new epoch. The old "compression substrate appears destroyed" error becomes "fold epoch confirmation required: …". Detection itself stays intact ([Bug] 旧版账本里的遗留锚被当作漂移证据,长会话被永久判为「折叠基底打穿」(0.1.194 / 0.1.195) #2903 honesty tests unchanged).prune.ts): unresolved summary anchors clamp ahead of the newest user turn instead of the array tail — the [Bug] User questions are silently ignored — unanchored fold blocks land after the latest user turn (0.1.197 / 0.1.199) #3640 "stale summary after the question" shape dies at the source.What deliberately did NOT change
dshLedgerFloorTokensstays identity-blind ([dsh] 原生压缩拦截门的 SHAPE 信号(≤4 条消息)在 dsh-compaction-basic 0.2.0-rc.2 上恒假 → 压缩静默落地 → 1510 条 fold 锚点永久失配,连刷 166 条 no-anchor-match 且面板仍报 netSaved 355.9M #2193 design).Verification
tests/fold-anchor-doctrine.test.ts6/6 — each defect class as a pinned behavior (protocol switch, offload family, truncation, epoch-confirm + auto-recovery on realignment, honesty gate, preflight gate).kernel-change: unresolved summary anchors clamp ahead of the newest user turn instead of the array tail (#3640) — kernel/src/prune.ts placement only, no state-shape or protocol change; kernel/tests/state-prune.test.ts +2 tests, 22/22.
Scope note (post-#3658 merge)
PR #3658 (protocol-stable core identity, #3642) was merged INTO this branch (its base), so this PR now carries BOTH bodies of work: the fold anchoring doctrine AND the canonical cross-protocol identity — plus the four review follow-ups pushed directly to the branches (opening-run multimodal pin, leading/trailing opaque parts, non-array content tolerance, warn-once image-index visibility). Merging this single PR lands everything; full-matrix CI ran green on the combined head (23 pass / 0 fail, run 39115262095) and the master (v0.1.200) merge conflict was resolved and re-verified locally (tsc 0, 130 tests across the touched families).