Skip to content

[PR] kernel: consolidate all mid-flight rewrites into fold-boundary transactions (#3939 ①–⑤, fixes #3003/#3691/#3769/#3805) | kernel: 中途重写全部收敛到折叠边界事务(#3939 ①–⑤, 修 #3003/#3691/#3769/#3805) - #3956

Merged
ranxianglei merged 28 commits into
masterfrom
2026-10-20_3939-prb-fold-transactions
Oct 20, 2026

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

这是什么

内核侧「折叠边界事务化」:五类会在安静轮次悄悄改写旧消息字节的行为,全部改为只在折叠边界一次性生效。每一类都是前缀缓存断裂源(#3805 家族),本 PR 让它们字节稳定。

kernel-change: 折叠边界事务化(#3938 / #3939 ①–⑤)——owner 已在 #3939 批准,打桩前置 PR #3941 已绿。

五个子项

项 机制 吸收/修复
① 失败 compress 记录轮换 折叠门控清扫:orphans 折叠间全可见,成功折叠后下一轮只清扫 refNum≥foldMinRef 的 orphans;swept 集持久单调,重启不复活;inactive 块仍无条件隐藏(幂等) 吸收 #3784 (18 commits), Fixes #3769
② BLOCKED→数字 ref 迁移 迁移即时铸号但载体 标签推迟到下一折叠边界(避免安静轮改写旧载体) Fixes #3805
③ 紧急截断 水位决策只做一次:clip 字节 latch 并每轮 replay;edited 消息(新 id)重新判定;折叠重置 Fixes #3691
④ 多 range 压缩 两层全有或全无:summary 长度预检(吸收 #3670)+ 应用级 clone-and-simulate(致命错误整批拒绝,zero 逻辑复制) Fixes #3003
⑤ opt-in 重写面 结构性 view latch:absorb-hide/crush/filter-keepLastOnly 的输出按 ref latch 并逐轮 replay;刷新只骑折叠边界、absorb 动作边界(保即时回收)、config 编辑边界(digest 放行)。ccr 无需 latch(到达即替换,设计上字节稳定 #1460) #3939 ⑤

打桩先行(#3941)

12 个行为 pin 先行固定旧契约(fold 边界 + wire 未知部分),其中 4 个随新契约更新语义(pin 名同步改),其余保持旧断言。

语义不变承诺(只动何时,不动改什么)

版本

kernel 0.0.137 → 0.0.141(每次内核源改动对应 bump)。

合并顺序

先合 #3941(打桩),再合本 PR —— 本分支包含打桩 commits,合并 #3941 后 diff 自动收缩为纯增量。

验证

  • kernel 套件 1356/1356 绿(新增 12 pins + 3 latch pins + 2 迁移 pins + 3 view-latch pins)
  • bili 定向面全绿:truncation/fork/persist/storage-render/absorb/loop/basic/cache-friendly/plugin ~300 tests
  • 顺手修:scripts/update-storage-render-goldens.ts 补 BILI_SNAPSHOT_SHADOW=0 奇偶校验(再生成的 golden 会在下一次 CI 红转,阻塞本 PR 的 golden 再生成)

Part of #3938(总账,勿随本 PR 关闭)。#3939 ⑥(wire 保真)属 PR-C,故本 PR 不关闭 #3939。

ework-agent and others added 21 commits October 20, 2026 17:42
The kernel kept only the newest TWO unmatched compress call+result pairs and hid older ones on EVERY turn, decoupled from real folds. On a third failed compress this deleted the oldest pair mid-history, rewriting an earlier cached prefix at full cost with no fold behind the break.

Replace the keep-two rotation with fold-gated, position-aware, persistent sweeping: orphans stay visible between folds (append-only, cache-friendly); a committed fold stamps pendingOrphanSweepMinRef (earliest new-block startRef) and the next pass hides only orphans at/after that ref, so the sweep rides the fold's own rewrite (cache-free) while earlier orphans (still-cached prefix) survive; swept ids persist in sweptOrphanCallIds (monotonic) so a resumed session never resurrects them.

Bump acp-kernel 0.0.136 -> 0.0.137.
…3769)

The #3769 seam landed in the 'no fold/switch/restart attribution' bucket because the break came from bili's own cleanup of failed-compress records, not a fold. Add an orphan-sweep attribution channel to the cache ledger: detect growth in the session's sweptOrphanCallIds set per sample and, when a mid-history break has no other claimant, label it prov='orphan-sweep' (count + missed tokens) instead of leaving it unexplained. A real fold claims its own break first, so there is no double-counting. New counters normalize through getCacheLedger for persistence compatibility.
types.ts and two kernel test files sit on a prettier-clean master baseline; the #3769 additions landed over-indented by one space. Normalized to the surrounding 2-space style so the diff stays confined to the feature. No behavior change; typecheck + targeted kernel/host suites re-run green.
…eview fix)

Review found a defect in the fold-marker computation: block-boundary specs (tier distillation compresses bN..bM) store startRef="bN" verbatim on the new block, so Number(startRef.replace(/\D/g,"")) parsed a BLOCK number, not the earliest covered MESSAGE ref. In long sessions that marks the sweep far too early: the next hide pass deletes failed pairs sitting in the still-cached prefix before the fold's own divergence point — re-introducing exactly the #3769 mid-history cache break on every tier-distillation fold.

Compute foldStartRef as the min refNum over each new block's effectiveMessageIds via state.messageRefs.byRaw instead; ids missing from the map (pruned) contribute nothing, which only under-sweeps (safe direction). refNum is exported from hide-consumed.ts so both consumers share one "unknown" definition.

Regression test 6 in issue3769-orphan-sweep.test.ts: m-boundary fold then block-boundary fold asserts pendingOrphanSweepMinRef === first covered ref (not the block number) and that the earlier pair stays visible. Verified to fail against the pre-fix code.
Review found the two new kernel state fields were dropped on every session load: buildRecord persists session.state wholesale, but the load path rebuilds it through mergeState's explicit field whitelist, which predated sweptOrphanCallIds / pendingOrphanSweepMinRef. Losing sweptOrphanCallIds resurrects already-cleaned failed pairs after a restart — a mid-history rewrite (cache break) that violates the PR's own persistence acceptance case. Losing the pending marker only defers a sweep by one fold (conservative).

Extend the mergeState whitelist with both fields, degrading to undefined for old persisted states exactly like every other optional field.
…-gated sweeps (#3769)

Three stale pins broke under the approved keep-two -> fold-gated behavior change:

- stub-resubmission-guard (#2749): the fixture applied folds without a call id, unlike production (src/loop/core.ts passes callId), so the resent call classified as orphan and the new sweep hid it. Pass "call-b1" at each first fold; all summary-integrity assertions unchanged. 5/5 green.
- cache-friendly-plugin: the two anthropic geometry subtests pinned the old KEEP_LAST_ORPHANED=2 rotation (oldest pair pruned on later turns). Rewritten to pin the new contract: failed pairs stay visible between folds, every anchor stays stripped while its own pair rides, quiet turns append-stable. A live-server probe plus the passing drop-pair/main-matrix subtests confirm the geometry expected values. Header docs updated to match the new mechanism. 11/11 green.
- storage-render goldens: regenerated via scripts/update-storage-render-goldens.ts. Diff is additive only — the persisted envelope gained lastSweptCount (cache ledger) and the two new kernel state keys (empty digests); state/blockContents/stats section digests unchanged.
Pre-#3769 states persisted the keep-two rotation's hidden set as message refs (hiddenOrphanRefs); the new hide pass ignored it, so upgrading would resurrect rotation-hidden failed pairs mid-history and break the cached prefix again. Honor the legacy set on upgrade; rememberHiddenRef re-records those refs every pass, so the clause self-perpetuates without any state migration.
syncBlocks deep-clones CompressionState with an explicit field list that missed hiddenOrphanRefs / sweptOrphanCallIds / pendingOrphanSweepMinRef. It runs BEFORE the hide-compress-calls node in every processTurn, so all three ledgers were wiped each turn: the fold-gated sweep never fired end-to-end and the legacy upgrade-compat clause was unreachable. Carry them exactly like cloneState does (lastPassIds stays excluded: its only reader runs pre-sync and the field is re-stamped at processTurn exit).
…P0)

Two processTurn-level pins: an upgraded pre-fold-gate state keeps its rotation-hidden orphan hidden across quiet turns (legacy hiddenOrphanRefs clause), and a fold marker set by applyCompression survives the full pipeline to the hide node (sync-blocks carry) so the fold-gated sweep actually fires end-to-end.
applyForkAdoption rebuilds session state from the parent on resume/fork replay; without carrying hiddenOrphanRefs and sweptOrphanCallIds, failed compress pairs already hidden pre-upgrade or already swept would resurface on replay, rewriting mid-history at full cache cost. Both ledgers are monotonic sets re-recorded every pass, so the carry self-perpetuates.
Real-server e2e: drive a session with a failed compress pair, persist it, patch the record back to an old-shape state (hiddenOrphanRefs set, no #3769 fields), reload under a fresh proxy, and assert the previously-hidden pair stays out of the outbound history while new turns append stably.
With the sync-blocks carry in place the sweep fires end-to-end: a failed compress pair appended at a fold step is swept by that step's own model request, so it never rides inbound and its anchor carries the summary for good (zero-carrier fail-safe, #1567). Re-pin the main matrix (all four wires), the swallow/back-to-back geometries (wide fold absorbs the alpha carrier; FOLD_E tail hits the protected zone), the drop-pair subtest (client-side drop is now a wire no-op) and the 3-concurrent-folds pin (carriers stack one per fold, stable thereafter), from observed live-server bodies. Google pair-absence proxy switches from the topic (which the carrier embeds) to the fold-spec refs that only the re-sent functionCall args carry.
…3769/#1567)

The fold-gated sweep hid EVERY orphan-fate compress pair at/after the fold's
start ref — including successful echoes whose call id never matched the
server-minted block id. A re-sent pair quoting an ACTIVE block's range is the
plugin lane's summary carrier (KDD#7), so hiding it strands the summary: the
#1999 double-carrier test, the plugin session-lock e2e, #1426 mode-upgrade
proof and #1638 notification pin all went red.

resolveFate's 'orphan' verdict is id-based and cannot see success; the sweep
branch now additionally requires the call to quote NO active block range
(quotesActiveBlock: both arg shapes — content[] entries and flat top-level
startId/endId — mirroring the host stripKernelSummaries carrier recognition).
Genuinely failed records and dead-block pairs (swallowed by a wider fold)
still sweep at the fold boundary, per the approved #3769 design.
…carrier regime and #3818 full-mode default (#3769 follow-through)

cache-friendly-plugin: the previous rewrite pinned the pre-floor-20 sweep
regime (successful echo pairs hidden at their fold step, anchors carrying).
That contradicts KDD#7 (plugin lane's wire carrier IS the tool call; the
agent view never renders the kernel acp_summary), the approved #3769 scope
(failed-record cleanup only), and the suite's own P6 dsh-rebase subtest,
which already pinned pair-rides/anchor-stripped. Re-pin to the verified
carrier regime across all four wires: pairs ride from their fold turn on
(summary twice per riding pair: args + receipt), SUM_MARKER zero while any
pair rides, drop-pair surfaces exactly its anchor once then stays
append-stable (#1567 fail-safe), dead-block pairs still sweep at the
swallow boundary (#3769), google lane expectations follow the #3159
canon-shift.

issue3769-upgrade-compat: since #3818 flipped the storage default to full,
the unified index row is the load path — patching only the JSON envelope
left a stale row the loader served instead, so the simulated old-kernel
record now goes through the store (loadSync + flushSync dual-writes both
copies). Teardown now closes both proxies even when an assert throws
(previously a mid-test failure hung the runner).
…#3691)

The watermark decision is made once: replaying the latched clip on every
later turn keeps outbound bytes stable when the token estimate fluctuates
across the threshold. Refs are content-bound so edited messages re-judge;
entries retire when their message leaves the view; a successful fold
resets the latch (the clipped originals ride the fold away). Carry
surfaces: sync node, persist mergeState, fork-adoption, plugin fork
prefix-filter, storage ingest.
…dary (#3939 ②, #3805)

The migration (#2607) mints the fresh numeric ref immediately (citable for
decompress bookkeeping) but the carrier message keeps its pre-migration,
UNTAGGED bytes until the next successful fold — stamping the <acp> tag on
the migration turn rewrote an old message mid-history on an otherwise
quiet turn (the #3805 durable-carrier cache breaker). Fold success
releases the deferral; carried across sync/persist/fork/plugin/ingest.

Also fixes the golden regen script missing the BILI_SNAPSHOT_SHADOW=0
parity pin (regenerated goldens red-flipped on the next CI run without
it); storage-render goldens now pin the two new undefined-valued state
keys, wire bodies unchanged.
…ch (#3939 ⑤)

absorb-hide, crush and filter keepLastOnly re-derive their output every
pass from state/context (absorb records growing, crush responding to
token pressure, keep-last-only demotions as new occurrences arrive) —
each change flipped OLD message bytes on an otherwise quiet turn (the
#3805 prefix-cache breaker class). One structural latch now covers all
of them: post-transform bytes are latched per padded ref at first
application and replay verbatim; refreshes ride fold boundaries
(applyCompression resets the map), the absorb action itself (instant-
reclaim contract kept), and config edits (digest release — enable/
disable applies on the edit turn, never silently later). ccr needs no
latch (replace-once-at-arrival, byte-stable by design #1460).

Placed after hide-compress-calls (the #3784 sweep keeps its own gate)
and before recommend/nudge/truncate/render so downstream judges see the
view the model actually sees. Carried across sync/persist/fork/plugin/
ingest; storage-render goldens pin the two new state keys (undefined
valued), wire bodies unchanged.
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_3939-prb-fold-transactions (961469e)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4746-pr.3956.6fbeb44

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3956.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title kernel: 中途重写全部收敛到折叠边界事务(#3939 ①–⑤, 修 #3003/#3691/#3769/#3805) [PR] kernel: consolidate all mid-flight rewrites into fold-boundary transactions (#3939 ①–⑤, fixes #3003/#3691/#3769/#3805) | kernel: 中途重写全部收敛到折叠边界事务(#3939 ①–⑤, 修 #3003/#3691/#3769/#3805) Oct 20, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

CI 三处失败已全部定位并修复(head 06b7d236, 3 个新 commit):

1. tests/compress-observability.test.ts #189 pin — 旧 fixture 的 range 落在保护区(last-3)内, 在新的原子批契约下会整批拒绝, 但旧测试实际 pin 的是「部分落地」旧语义。fixture 移到保护区外, 测试意图(参数顺序无关)保留。

2. tests/issue3787-overflow-refold.test.ts numberless 救援 — 根因两层:

  • 宿主: numberless 400 时 overflowRefold(overflowInfo.window) 传 undefined → refold 按声明窗判「装得下」→ 折不动 → 400 原样透传。修复: relay.ts numberless 时 armedWindow = floor(arm × 0.9)(arm 按构造=被拒 payload 自身大小, 10% 余量给折叠真实空间), refold 真正按 armed 窗折。
  • preflight: ranges 耗尽分支直接判失败, 不给 tier 蒸馏机会(futility 分支有 defer)。修复: exhausted + eligibleForTierRecovery() → defer 到 [bug][preflight] Raw-range futility gate may strand tier-enabled sessions with distillable T1 blocks (502 deadlock) #2952 funnel。
  • fixture 重写: 旧声明窗 10k<payload, 校准触发器早已在首发前折干一切, 历史「救援成功」是截断不对称的假阳性(③ latch 消灭了它)。新声明窗 100k, raw body 真出站 → 400 → arm → in-request 救援, 5/5 绿。
  • 顺带把 tests/e2e-overflow-selfheal.test.ts T2 从 context window problem continues #987 旧契约(400 透传)升级到 Context Overflow | 上下文溢出 #3787 新契约(in-request 救援), T1 不动。

3. tests/ccr-plugin.test.ts route-scoped CCR — kernel ⑤ 的真 bug, 感谢这个测试逮住: viewLatchConfigDigest 只含 {absorb, crush, filters}, 没有 CCR。前一个测试的会话(同 conv id)在 armed CCR 下把占位符 latch 进 state; 换 rig 后 CCR 已 disarm 但 digest 不变 → latch 不释放 → 占位符照投, 而 acp_retrieve 通道已不存在 = 悬空占位符, 静默丢失, 正是 #1273 治理要防的。修复: viewLatchConfigDigest 加入 resolveCcrConfig(ctx.config) — CCR 开关翻转 = 用户动作边界, 释放 latch 重判。kernel 新 pin「CCR disarm releases the latch」(kernel/tests/issue3939-view-latch.test.ts), kernel 0.0.141→0.0.142, 本地 kernel 1357/1357 + ccr-plugin 14/14 + 相关 70+ 全绿。

遗留: build-artifact 在 merge ref 上仍红 — 那是 master #3946(mysql 后端)自己把 mysql2 漏出 package.json 造成的(详见 #3964), 不是本 PR 的回归; master 修好后 merge ref 重跑即绿。

ework-agent and others added 2 commits October 20, 2026 20:39
…eline the #3769 proof to the fold-gated contract (#3939)

- truncate-tools: build truncatedIds in the loop instead of spreading
  Map.keys() — the iterator spread degrades the symbolic translation of the
  3691 harness (verif refuses to certify on degradation).
- hide-consumed: hoist state.hiddenOrphanRefs ?? [] — .includes on a
  possibly-undefined receiver degrades the same way.
- verif/symbolic: fold Number(constString) to its numeric constant; the old
  {kind:any} made Number.isFinite collapse to constant false, silently
  deleting the refNum-gated fold-sweep path from every model (real code
  fires it; the model could not — engine gap, not kernel bug).
- verif-bug3769 harness re-baselined onto the #3784 fold gate: Q1 (append-only
  growth hides r1) is now UNSAT — the fix holds; Q1b proves the sweep is
  still derivable THROUGH the fold marker (judge not vacuous); replay pins
  hidden=0 without a pending fold and newlySwept=[c1,c2,c3] with one.
- kernel 0.0.142 -> 0.0.143 (both locks synced).

verif: 15/15 green with cvc5 (subset-guard, symbolic-smoke, 3691, 3769,
3805, 3676, generic, induction). kernel suite 1357/1357.
…-fold-transactions

# Conflicts:
#	tests/verif-bug3676-wire-reorder.test.ts
#	tests/verif-bug3691-oscillation.test.ts
#	tests/verif-bug3769-orphan-window.test.ts
#	tests/verif-induction-append-stability.test.ts
#	verif/symbolic.ts
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.677-master.562468a (rolling master tag).

npm install -g billion-context@0.0.677-master.562468a  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment