Skip to content

[PR] fix: recover non-retained parent fork snapshot from intact unified raw tail after cold restart (#5140) - #5144

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-29_cold-raw-tail-fork
Oct 29, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-29_cold-raw-tail-fork

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #5140

What & why

A non-retained plugin parent intentionally does not persist its raw pluginSnapshot — the retention gate (src/persist.ts) only writes it after a fork receipt / retained-snapshot contract. So after a cold restart, GET /__bili/plugin/snapshot failed 409 SNAPSHOT_UNAVAILABLE for such a parent even though index.db still held the independently recorded, unfolded canonical raw tail. Retrying the same cold source could never manufacture the missing resident copy, and child adoption lost access to already-stored compression work. This is item ④ of #4266.

Fix (src/plugin.ts)

forkSnapshot(session, source?) now falls back to a new pure helper reconstructRawTailForFork(session) when the resident snapshot is absent:

  • reads the unified view through its logical projection (unifiedTailSeqs + unifiedTailViewRows), never append-log order or the folded envelope;
  • serves it only after proving it is the session's complete ordered history: zero gaps, length + per-cell id match against state.lastPassIds, UTF-8 byte cap;
  • then runs through every existing bidirectional-ref / CCR witness / opaque-thinking gate unchanged;
  • fails closed on missing/stale/partial/capped data; legacy mode (no unified source) is untouched.

GET establishes no permanent retention; only a successful fork materializes the verified source onto the parent (retention-on-success), so the existing flushSync persists it for continuity.

Intended behavior change (disclosed)

case before after
GET /__bili/plugin/snapshot for a cold-restored non-retained parent whose verified raw tail survives in the unified store 409 SNAPSHOT_UNAVAILABLE until the next model request re-established the resident view 200 exact immediately (no model call needed)

Why: recovering already-stored source material must not require a fresh parent model turn (#5140's core ask). It fails closed everywhere the source cannot be verified, so no partial/mis-attributed snapshot is ever served.

Consequently tests/plugin-fork.test.ts's "self-heals on the next model request" test is updated to the new contract (its pre-model-request assertion flips 409→200); its post-model-request resident-view check is preserved.

Tests

New tests/issue5140-cold-raw-tail-fork.test.ts — synthetic cold close/reopen HTTP inheritance:

  • positive (shadow): build a non-retained parent, close+reopen the store on the same data root, GET snapshot → 200 full count, assert GET did not retain, POST fork → 201, assert retention-on-success;
  • fail-closed over-cap: BILI_PUBLIC_SNAPSHOT_CAP_BYTES=1 in phase 2 → 409;
  • fail-closed legacy: no unified source → 409.

Targeted suites green locally: issue5140 (3/3), dsh-fork-advantage + plugin-fork (54/54), fork-replica-convergence, dsh/pi-omp/opencode fork-adopt + issue3573-cold-load (47/47). Full-suite verdict = CI.

Scope / review notes

  • No storage schema/version, no config surface, no kernel, no native-adopter retry policy, no warm-tail selection, no HTTP200 overflow behavior, no release change.
  • Touches fork identity/session-binding logic → human review required (§7.4); not auto-mergeable.

ework-agent and others added 4 commits October 29, 2026 14:46
…ession persistence (#4599)

The four native-lane e2e suites SIGTERM the hermetic proxy and then wait a
fixed 500ms before reading persisted sessions from disk. That races the
product's own shutdown order: SIGTERM -> server.close() drains keep-alive
connections first (SHUTDOWN_DRAIN_GRACE_MS = 10s backstop) and only then
flushAllSessions() runs, so on a slow socket teardown the 500ms read lands
pre-flush and the suite flakes ("proxy must persist the session"). Seen in
the e2e-native-opencode scheduled lane at 5594c13 (run 40751086427 red,
same-sha rerun green).

Process exit implies flushAllSessions completed, so waiting for the recorded
instance pids to actually exit makes the disk assertions deterministic:

- e2e-native-opencode / e2e-subagent-sessions: reuse the file's existing
  awaitProxyExit() bounded poll (20s budget > 10s drain grace).
- e2e-native-pi / e2e-native-dsh: extract liveInstancePids() and poll it
  until empty (same 20s budget).

e2e-pi-codex-ws's fire-and-forget variant is untouched: it only runs in
teardown hooks and gates no disk assertion.

Verified locally: tsc --noEmit clean; the previously flaking subtest passes
4/4 consecutive hermetic runs (zero tokens). Full-suite verdict: remote CI.
#4599)

Master red at 1fad21d (run 40852517737, 6 shards): two mechanisms, both in
the #5107/#5115/#5120 hermetic data-root cluster.

1. preload-test-env.mjs pinned BILI_CONFIG_FILE to an absent file.
   configFile() (src/paths.ts) consults that env BEFORE the XDG default, so
   the pin shadowed every fixture that re-points XDG_CONFIG_HOME at its own
   dir and writes the default config path: the claude-native attach-gate /
   lane-priority / policy / url file-layer assertions failed on all shards,
   and the #3436 compat checks resolved mitm.domains=[] -> spawn path -> 60s
   fake-child timeout. The XDG_CONFIG_HOME pin already keeps the default
   config path inside the throwaway root; DELETE the env instead of pinning
   so fixtures are un-shadowed AND a developer's ambient pin is dropped. The
   isolation probe now asserts configFile() resolves UNDER the sandboxed
   XDG_CONFIG_HOME (same property, sibling-check style) instead of equality
   with the removed pin.

2. test-suite-lock.mjs snapshotSessionsTree keyed entries by raw
   path.relative(): Windows yields 'anthropic\\s.json' while the diff, the
   banner and the tests all compare '/'-separated paths (4 failures on the
   windows s2 shards). Normalize keys to '/' at the source.

Verified locally with the CI preload chain (targeted): claude-native +
#3436 + test-suite-lock suites 96/96 pass, 0 fail. Windows separator
behavior holds by construction (split(path.sep).join('/')); full verdict =
remote CI.
…w tail after cold restart (#5140)

A non-retained plugin parent intentionally does not persist its raw
pluginSnapshot (the retention gate only writes it after a fork receipt), so a
cold restart left GET /__bili/plugin/snapshot failing 409 SNAPSHOT_UNAVAILABLE
even though index.db still held the independently recorded unfolded canonical
raw tail. Retrying the same cold source could never manufacture the missing
resident copy.

forkSnapshot now falls back to reconstructing that tail through its LOGICAL
projection (unifiedTailSeqs + unifiedTailViewRows) when the resident snapshot
is absent, serving it only after proving it is the session's complete ordered
history (zero gaps, length + per-cell id match against lastPassIds, UTF-8 byte
cap) AND it passes every existing bidirectional-ref / CCR / opaque-thinking
gate. Missing/stale/partial/capped data fails closed; legacy mode (no unified
source) is unaffected.

GET establishes no permanent retention; only a successful fork materializes the
verified source onto the parent (retention-on-success), so the existing
flushSync persists it for continuity.

Intended behavior change: GET /__bili/plugin/snapshot now returns 200 exact for
a cold-restored non-retained parent whose verified raw tail survives in the
unified store, instead of 409 until the next model request re-established the
resident view. The plugin-fork self-heal test is updated to the new contract.

Adds synthetic cold close/reopen HTTP inheritance tests (positive shadow
reconstruction + over-cap fail-closed + legacy fail-closed).
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-29_cold-raw-tail-fork (7e4e017)

Latest build — install this exact version

npm install -g billion-context@0.0.6779-pr.5144.42a6c83

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
7e4e017 billion-context@0.0.6779-pr.5144.42a6c83 build
ebefa8b billion-context@0.0.6775-pr.5144.e3a8386 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5144.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: recover non-retained parent fork snapshot from intact unified raw tail after cold restart (#5140) [PR] fix: recover non-retained parent fork snapshot from intact unified raw tail after cold restart (#5140) Oct 29, 2026
ranxianglei pushed a commit that referenced this pull request Oct 29, 2026
…ments cite #5139, not #5141

Owner correction on #5144: the Codex separator-endpoint bug report is GitHub issue #5139 (kuviki), not #5141 (an unrelated in-flight external PR). Renames tests/issue5141-turn-separator-ranges.test.ts -> tests/issue5139-turn-separator-ranges.test.ts and re-points every in-code citation. No behavior change: comments, test names/session-id prefix only.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:重复——与并行 PR #5141 重复(同 issue #5140、同症状、同机制、同根因)。#5141(abworks-dev,12:05Z 先提交)与本 PR(#5144,12:39Z)都修「冷重启后非保留父会话的 fork 快照恢复」,方案同为「从 unified 存储重建 raw tail」。按先来后到默认保留更早的 #5141、本 PR 标 duplicate;两个都不关闭,owner 可凭下方质量对比改判(若留本 PR 则关 #5141)。

标签: bug / dsh / p1 / scope/storage / duplicate(会话恢复类缺陷按 owner 修订归 p1;fork 读路径落存储域;dsh 为主要宿主车道;duplicate 见上)。


独立审核结论(非复述 PR 描述)

机制核实(逐条对代码验证过):

  • 门槛前提成立:lastPassIds 是内核定义的「最近一次入站 pass 的消息 id 序列」(kernel/src/compress.ts:868 inboundIds = input.messages.map(m => m.id),:876 写入 state;kernel/tests/issue1476-reid.test.ts:232 钉住语义)。重建结果与它做长度+逐格 id 比对,能证明的就是「这次 pass 完整重发了该序列」——比对了什么、没证明什么,见下面披露的局限。
  • shadow/full 两种模式可读:unifiedTailSeqs/unifiedTailViewRows 走 sidecar 同步库(src/storage/shadow.ts:746/:772 → sync.tailSeqs/sync.getMessages),只以 sidecar 存在性为门,不受 readsUnified(full-only)限制,所以 replicasAvailable() 门在 shadow 下有效——PR 的 positive 测试正是在 shadow 模式跑通的,与代码一致。
  • GET 路径确实吃到回退链:admin.ts:473 → handlePluginSnapshot → forkSnapshot(session);本地实测冷重启后 GET 直接 200(改名后的既有测试断言已翻转为 200 并通过)。
  • source! 非空断言安全:handler 在 withSessionLock 内算一次 source,forkSnapshot 内部再解析是同一锁下确定性重算,forkSnapshot 成功 ⇒ source 必非空;不加死分支 guard(避免死代码)。

测试审计(新文件 228 行,真 e2e 有牙): 真 HTTP + 真 close/reopen(阶段 2 全新 store+proxy 开同一数据根);断言钉住两条契约——GET 不建立保留(pluginSnapshot === undefined 且 publicSnapshotRetained === undefined)、仅成功 fork 才保留(fork 201 后 publicSnapshotRetained === true);cap 超限(phase 2 单独设 BILI_PUBLIC_SNAPSHOT_CAP_BYTES=1)与 legacy 无源两条 fail-closed 都在。覆盖缺口:没有压缩过再冷重启的用例(issue 原 repro 带 mock compression block),也没有显式的 gap/stale 行 fail-closed 用例——前者受限于下面的覆盖局限,后者建议补。

回归清点(逐项): ① 常驻快照优先级不变(source ?? pluginSnapshot ?? reconstruct,warm 路径字节级不变);② GET 前后均无保留副作用(测试钉住);③ child 切片来源从 parent.pluginSnapshot! 改为 source,常驻场景两者恒等,行为不变;④ legacy(无 unified 源)维持 409;⑤ wire/schema/config/kernel/release 零变更。唯一有意行为变更即 PR 正文披露表(冷非保留父 GET 409→200),披露合规。

CI: 首跑红 = 继承红(master@1fad21de8 自身 6 分片红,#4599 在档 fixture-shadowing 家族)。ubu24/2 日志逐字命中签名(resolveNativeAttach{External,LanePriority,Policy,Url}×4 + #3436 mitm.domains×2);win22/4 + ubu24/3 待重跑确认。已按协议 merge ci-fixes@cf99d935f 进分支并推送,新 head sha=7e4e0172e,等全矩阵重跑(已在 #4599 留言)。本地:typecheck 干净 + 定向 57/57 绿(issue5140 3 + plugin-fork 54)。

结构化八问

  1. 是什么问题:冷重启后,从未被 fork 过的「非保留」父会话(驻留 raw 快照按设计从不落盘)即使 index.db 里还躺着完整的未折叠 raw tail,GET /__bili/plugin/snapshot 也永远 409,子会话收养拿不到已存下的压缩成果,只能等下一次模型回合自愈。
  2. 什么原因导致的:写侧保留门只在 fork receipt/显式保留后才把 pluginSnapshot 写盘(设计如此),而读侧 forkSnapshot 硬依赖这个内存字段、对真正持久化了数据的 unified 存储零回退——数据在,读路径不看它。
  3. 影响范围与严重程度:所有插件车道(dsh/opencode/pi/omp…)的非保留父会话,经历压缩后冷重启,到下次模型回合前 fork/snapshot 面不可用;不崩溃不丢数据,但属 Zero-Inheritance Session Cloning: Unified Fix for Eight Independent Defects in the Fork Adoption Chain (Next Release Tar | 会话克隆零继承:fork 收养链上八类独立缺陷统一修复(下版目标) #4266 item ④ 的核心能力窗口期失灵,按 owner 修订(会话恢复类)定 p1。
  4. 怎么修复的:加纯函数回退 reconstructRawTailForFork——沿 unified 视图的逻辑投影(tailSeqs 指针序 + 逐行内容)重建,只有能证明「完整有序历史」(零缺口、长度+逐格 id 对齐 lastPassIds、UTF-8 字节上限)才放行,随后原样穿过既有全部双向 ref/CCR witness 门;缺失/陈旧/部分/超限一律 fail closed;GET 不建立永久保留,仅成功 fork 把验证过的源固化到父会话(retention-on-success)。
  5. 是否有回归问题:审了上面回归清点的 5 项,无意回归未发现;高危模式逐项过——无语义漂移(常驻路径恒等)、无吞错(fail-closed 不变)、无阈值/时序/输出格式变化、无缓存策略变化。
  6. 是否破坏缓存:不涉及(不动前缀缓存接缝、消息身份/reconcile、重试与增量游标、去重键)。
  7. 是否改了设计方向:无——按工单 bounded fix 执行,只给 fork 源选择点加读侧回退,写侧保留契约未动。
  8. 是否引入新配置:无(复用既有 BILI_PUBLIC_SNAPSHOT_CAP_BYTES)。

必须披露的覆盖局限

回退只在该会话「最近一次 pass 完整重发了整条 raw tail」时命中(lastPassIds 语义决定);最近一次 pass 重发的是折叠视图(短于 raw tail)的会话,仍保持旧行为 409 直到下次模型回合。工单本身指定 lastPassIds 匹配为验证机制,这是字面实现,但实际覆盖面比标题暗示的窄——请人工审核确认这个取舍可接受;若要求更宽覆盖,验证基准要换(如账本全集),那是超出本工单的设计决策。

两 PR 对比(供裁决)

维度 #5141(先到) #5144(本 PR)
验证强度 更强:lastPassIds 之外再做 stored/live state、blockContents、CCR contentStore 三重深等值(工单验收字面)+ 显式拒 folded tail(lastMessagesFolded/tailGaps) 较窄:lastPassIds 长度+逐格 id+UTF-8 cap,state 一致性靠下游 forkSnapshot witness 门兜底
改动面 persist.ts(+21)+plugin.ts(+25)+测试;额外把 forkParentRevisionLegacy 改走同源选择(行为面更宽) 只动 plugin.ts(+51)+独立测试文件(228 行)+既有测试 6 行;legacy revision 路径不碰
读成本 每次冷 GET 走 loadUnified 全视图 tailSeqs 指针+行内容(同量级,纯函数)
测试 +169 行进 plugin-fork.test.ts(未全量审计) 3 个真 close/reopen HTTP e2e(正例+cap fail-closed+legacy fail-closed),本地全绿

判定:代码审核通过(带上述局限披露);§7.4 fork 身份/会话绑定 → 必须人工审核,不可自动合并。 待办:CI 新 head 7e4e017 重跑转绿 + owner 在两 PR 间裁决(默认留 #5141;若留本 PR,建议吸收 #5141 的状态深等值门作为强化,即其自己警告过的 union-of-intent 合流)。我不合并、不关闭任何条目。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.915-master.54aec3d (rolling master tag).

npm install -g billion-context@0.0.915-master.54aec3d  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei added a commit that referenced this pull request Oct 30, 2026
…d sidecar; recover an unlucky first open from the save path (#5188)

Two independent one-way degradations made the unified store vanish under CI
load and turned #5144's fork raw-tail rebuild into hard 409s:

1. The 3-strike disable latch counted every ingest error equally. Under
   machine load sqlite surfaces resource-class errors (locks held past the
   5s busy_timeout, momentary disk-full, fd spikes) that settle on their own;
   three ordinary hits latched the sidecar legacy-only for the process
   lifetime. noteIngestFailure now classifies via isTransientContentionError
   and only structural/operator-evidence errors (corruption, readonly, NOTADB,
   internal invariants) count toward the latch. lastError stays fresh either
   way; an already-latched sidecar stops re-warning on every save.

2. A FIRST synchronous open that failed transiently had no retry path at all:
   maybeOpen warns once and the process runs legacy-only forever. The save
   path is now the 'still wanted' signal: SessionStore.tryRecoverSidecar
   retries maybeOpen at most once per 30 s cooldown window (fixed constant,
   no config surface); on success the unified surfaces resume WITHOUT a
   restart (status wire clears via recordSidecarInstance inside maybeOpen).

In-process probe evidence from the failing CI shard shape:
'[recon-debug] ... gateB-replicasUnavailable sid=fork-parent-* disj=disabled:
lastError=SQLITE_BUSY: database is busy' (loop-A-r2/r3, tmp/ci-repro/).
ranxianglei pushed a commit that referenced this pull request Oct 30, 2026
…names its refusal branch

The bare status assert swallowed the SNAPSHOT_UNAVAILABLE body for two weeks,
so every CI hit of the #5144 cold raw-tail flake showed only '409 !== 200'
with no way to tell which forkSnapshot branch refused. The body carries the
exact error string (reconstructRawTailForFork null vs CCR invariant vs cap) —
assert on status with the body as the failure message. No behavior change.
Louie0806 pushed a commit to Louie0806/billion-context that referenced this pull request Oct 30, 2026
ranxianglei#5155/ranxianglei#5144: incremental lane — serialize the wire prefix once per canonical
entry and splice only the delta per turn (laneBodyBuffer sentinels with
full-restring fallback); recordCanonicalPointers walks only rows appended
after the last recorded pointer (windowed rawMessages(fromSeqExclusive))
with cluster-suffix (<base>_<n>) tolerant matching so pointers no longer
silently freeze; wrap-up save debounces 500ms via scheduleSave instead of a
synchronous writeNow+fsync on every turn.

Conflict resolutions (both sides' features unioned):
- src/storage/shadow.ts: keep ours' activeForeignWriter() (ranxianglei#4612) AND take
  p5156's optional fromSeqExclusive param on rawMessages() — the two edits
  sit on adjacent methods; final file carries both.
- src/storage/store.ts: getMessages() opts unioned to { includeRetired?,
  seqs?, fromSeqExclusive? } — ranxianglei#4765's pointer-scoped seqs fetch + read
  instrumentation coexist with ranxianglei#5155's windowed scan (seqs takes priority
  when both are present; params assembled positionally). Async backend
  paths untouched by p5156, so no interface change required.
Louie0806 added a commit to Louie0806/billion-context that referenced this pull request Oct 30, 2026
…f inheriting it

The cold-restore case reconstructs the parent's raw tail from the unified
sidecar, so it only means anything in `full` mode. It never set the knob: it
inherited `full` from whichever earlier case in the file left it behind.

That dependency is invisible in a standalone run — and the CI harness makes it
fatal, because `tests/preload-storage-baseline.mjs` pins
`BILI_STORAGE_UNIFIED ??= "legacy"` (ranxianglei#2815) for the whole corpus. Result: the
shard where this file lands has been red on every run since ranxianglei#5144 merged, with
`409 SNAPSHOT_UNAVAILABLE` and an 18ms duration (the store never had a sidecar
to read).

Reproduced deterministically:
  node --import ./tests/preload-storage-baseline.mjs --import tsx \
    --test --test-name-pattern=cold-restored tests/plugin-fork.test.ts
  => not ok 1 (before) / ok 1 (after)

Pin the mode in the case and restore the prior value in finally, so the case
states its own precondition and cannot be re-broken by an unrelated suite's
cleanup order.
ranxianglei added a commit to Augenblick0v0/billion-context that referenced this pull request Oct 30, 2026
…nstead of failing closed silently

The ranxianglei#5144 cold-restored snapshot path fails closed with a bare
SNAPSHOT_UNAVAILABLE when any of its five reconstruction gates misses.
On master CI the ranxianglei#5140 regression subtest has been red shard-after-shard
with zero signal about WHICH gate fired — the five null returns were
indistinguishable from the server log and from the client-visible error.

Every miss now records its gate (empty-lastPassIds / replicasUnavailable /
empty-tailSeqs / null-tailViewRows / gap-tailRow / unparseable-row /
bad-row-shape / length-mismatch / id-mismatch / cap-exceeded), logs one
warn line per session+gate per 10s, and appends [reconstruction: <gate>]
to the thrown SNAPSHOT_UNAVAILABLE message so a 409 body says exactly why
the tail could not be rebuilt. Behavior is unchanged — same statuses, same
codes, same base messages; pure observability ahead of the root fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh duplicate This issue or pull request already exists p1 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4)

1 participant