Repository navigation
[PR] fix: recover non-retained parent fork snapshot from intact unified raw tail after cold restart (#5140) - #5144
Conversation
…ession persistence (#4599) The four native-lane e2e suites SIGTERM the hermetic proxy and then wait a fixed 500ms before reading persisted sessions from disk. That races the product's own shutdown order: SIGTERM -> server.close() drains keep-alive connections first (SHUTDOWN_DRAIN_GRACE_MS = 10s backstop) and only then flushAllSessions() runs, so on a slow socket teardown the 500ms read lands pre-flush and the suite flakes ("proxy must persist the session"). Seen in the e2e-native-opencode scheduled lane at 5594c13 (run 40751086427 red, same-sha rerun green). Process exit implies flushAllSessions completed, so waiting for the recorded instance pids to actually exit makes the disk assertions deterministic: - e2e-native-opencode / e2e-subagent-sessions: reuse the file's existing awaitProxyExit() bounded poll (20s budget > 10s drain grace). - e2e-native-pi / e2e-native-dsh: extract liveInstancePids() and poll it until empty (same 20s budget). e2e-pi-codex-ws's fire-and-forget variant is untouched: it only runs in teardown hooks and gates no disk assertion. Verified locally: tsc --noEmit clean; the previously flaking subtest passes 4/4 consecutive hermetic runs (zero tokens). Full-suite verdict: remote CI.
#4599) Master red at 1fad21d (run 40852517737, 6 shards): two mechanisms, both in the #5107/#5115/#5120 hermetic data-root cluster. 1. preload-test-env.mjs pinned BILI_CONFIG_FILE to an absent file. configFile() (src/paths.ts) consults that env BEFORE the XDG default, so the pin shadowed every fixture that re-points XDG_CONFIG_HOME at its own dir and writes the default config path: the claude-native attach-gate / lane-priority / policy / url file-layer assertions failed on all shards, and the #3436 compat checks resolved mitm.domains=[] -> spawn path -> 60s fake-child timeout. The XDG_CONFIG_HOME pin already keeps the default config path inside the throwaway root; DELETE the env instead of pinning so fixtures are un-shadowed AND a developer's ambient pin is dropped. The isolation probe now asserts configFile() resolves UNDER the sandboxed XDG_CONFIG_HOME (same property, sibling-check style) instead of equality with the removed pin. 2. test-suite-lock.mjs snapshotSessionsTree keyed entries by raw path.relative(): Windows yields 'anthropic\\s.json' while the diff, the banner and the tests all compare '/'-separated paths (4 failures on the windows s2 shards). Normalize keys to '/' at the source. Verified locally with the CI preload chain (targeted): claude-native + #3436 + test-suite-lock suites 96/96 pass, 0 fail. Windows separator behavior holds by construction (split(path.sep).join('/')); full verdict = remote CI.
…w tail after cold restart (#5140) A non-retained plugin parent intentionally does not persist its raw pluginSnapshot (the retention gate only writes it after a fork receipt), so a cold restart left GET /__bili/plugin/snapshot failing 409 SNAPSHOT_UNAVAILABLE even though index.db still held the independently recorded unfolded canonical raw tail. Retrying the same cold source could never manufacture the missing resident copy. forkSnapshot now falls back to reconstructing that tail through its LOGICAL projection (unifiedTailSeqs + unifiedTailViewRows) when the resident snapshot is absent, serving it only after proving it is the session's complete ordered history (zero gaps, length + per-cell id match against lastPassIds, UTF-8 byte cap) AND it passes every existing bidirectional-ref / CCR / opaque-thinking gate. Missing/stale/partial/capped data fails closed; legacy mode (no unified source) is unaffected. GET establishes no permanent retention; only a successful fork materializes the verified source onto the parent (retention-on-success), so the existing flushSync persists it for continuity. Intended behavior change: GET /__bili/plugin/snapshot now returns 200 exact for a cold-restored non-retained parent whose verified raw tail survives in the unified store, instead of 409 until the next model request re-established the resident view. The plugin-fork self-heal test is updated to the new contract. Adds synthetic cold close/reopen HTTP inheritance tests (positive shadow reconstruction + over-cap fail-closed + legacy fail-closed).
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.6779-pr.5144.42a6c83This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5144.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…ments cite #5139, not #5141 Owner correction on #5144: the Codex separator-endpoint bug report is GitHub issue #5139 (kuviki), not #5141 (an unrelated in-flight external PR). Renames tests/issue5141-turn-separator-ranges.test.ts -> tests/issue5139-turn-separator-ranges.test.ts and re-points every in-code citation. No behavior change: comments, test names/session-id prefix only.
(ai: vllm-qwen/qwen3.8-27b) 查重:重复——与并行 PR #5141 重复(同 issue #5140、同症状、同机制、同根因)。#5141(abworks-dev,12:05Z 先提交)与本 PR(#5144,12:39Z)都修「冷重启后非保留父会话的 fork 快照恢复」,方案同为「从 unified 存储重建 raw tail」。按先来后到默认保留更早的 #5141、本 PR 标 标签: bug / dsh / p1 / scope/storage / duplicate(会话恢复类缺陷按 owner 修订归 p1;fork 读路径落存储域;dsh 为主要宿主车道;duplicate 见上)。 独立审核结论(非复述 PR 描述)机制核实(逐条对代码验证过):
测试审计(新文件 228 行,真 e2e 有牙): 真 HTTP + 真 close/reopen(阶段 2 全新 store+proxy 开同一数据根);断言钉住两条契约——GET 不建立保留( 回归清点(逐项): ① 常驻快照优先级不变( CI: 首跑红 = 继承红(master@1fad21de8 自身 6 分片红,#4599 在档 fixture-shadowing 家族)。ubu24/2 日志逐字命中签名( 结构化八问
必须披露的覆盖局限回退只在该会话「最近一次 pass 完整重发了整条 raw tail」时命中(lastPassIds 语义决定);最近一次 pass 重发的是折叠视图(短于 raw tail)的会话,仍保持旧行为 409 直到下次模型回合。工单本身指定 lastPassIds 匹配为验证机制,这是字面实现,但实际覆盖面比标题暗示的窄——请人工审核确认这个取舍可接受;若要求更宽覆盖,验证基准要换(如账本全集),那是超出本工单的设计决策。 两 PR 对比(供裁决)
判定:代码审核通过(带上述局限披露);§7.4 fork 身份/会话绑定 → 必须人工审核,不可自动合并。 待办:CI 新 head 7e4e017 重跑转绿 + owner 在两 PR 间裁决(默认留 #5141;若留本 PR,建议吸收 #5141 的状态深等值门作为强化,即其自己警告过的 union-of-intent 合流)。我不合并、不关闭任何条目。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.915-master.54aec3d # this exact build
npm install -g billion-context@master # always the newest merged build
|
…d sidecar; recover an unlucky first open from the save path (#5188) Two independent one-way degradations made the unified store vanish under CI load and turned #5144's fork raw-tail rebuild into hard 409s: 1. The 3-strike disable latch counted every ingest error equally. Under machine load sqlite surfaces resource-class errors (locks held past the 5s busy_timeout, momentary disk-full, fd spikes) that settle on their own; three ordinary hits latched the sidecar legacy-only for the process lifetime. noteIngestFailure now classifies via isTransientContentionError and only structural/operator-evidence errors (corruption, readonly, NOTADB, internal invariants) count toward the latch. lastError stays fresh either way; an already-latched sidecar stops re-warning on every save. 2. A FIRST synchronous open that failed transiently had no retry path at all: maybeOpen warns once and the process runs legacy-only forever. The save path is now the 'still wanted' signal: SessionStore.tryRecoverSidecar retries maybeOpen at most once per 30 s cooldown window (fixed constant, no config surface); on success the unified surfaces resume WITHOUT a restart (status wire clears via recordSidecarInstance inside maybeOpen). In-process probe evidence from the failing CI shard shape: '[recon-debug] ... gateB-replicasUnavailable sid=fork-parent-* disj=disabled: lastError=SQLITE_BUSY: database is busy' (loop-A-r2/r3, tmp/ci-repro/).
…names its refusal branch The bare status assert swallowed the SNAPSHOT_UNAVAILABLE body for two weeks, so every CI hit of the #5144 cold raw-tail flake showed only '409 !== 200' with no way to tell which forkSnapshot branch refused. The body carries the exact error string (reconstructRawTailForFork null vs CCR invariant vs cap) — assert on status with the body as the failure message. No behavior change.
ranxianglei#5155/ranxianglei#5144: incremental lane — serialize the wire prefix once per canonical entry and splice only the delta per turn (laneBodyBuffer sentinels with full-restring fallback); recordCanonicalPointers walks only rows appended after the last recorded pointer (windowed rawMessages(fromSeqExclusive)) with cluster-suffix (<base>_<n>) tolerant matching so pointers no longer silently freeze; wrap-up save debounces 500ms via scheduleSave instead of a synchronous writeNow+fsync on every turn. Conflict resolutions (both sides' features unioned): - src/storage/shadow.ts: keep ours' activeForeignWriter() (ranxianglei#4612) AND take p5156's optional fromSeqExclusive param on rawMessages() — the two edits sit on adjacent methods; final file carries both. - src/storage/store.ts: getMessages() opts unioned to { includeRetired?, seqs?, fromSeqExclusive? } — ranxianglei#4765's pointer-scoped seqs fetch + read instrumentation coexist with ranxianglei#5155's windowed scan (seqs takes priority when both are present; params assembled positionally). Async backend paths untouched by p5156, so no interface change required.
…f inheriting it The cold-restore case reconstructs the parent's raw tail from the unified sidecar, so it only means anything in `full` mode. It never set the knob: it inherited `full` from whichever earlier case in the file left it behind. That dependency is invisible in a standalone run — and the CI harness makes it fatal, because `tests/preload-storage-baseline.mjs` pins `BILI_STORAGE_UNIFIED ??= "legacy"` (ranxianglei#2815) for the whole corpus. Result: the shard where this file lands has been red on every run since ranxianglei#5144 merged, with `409 SNAPSHOT_UNAVAILABLE` and an 18ms duration (the store never had a sidecar to read). Reproduced deterministically: node --import ./tests/preload-storage-baseline.mjs --import tsx \ --test --test-name-pattern=cold-restored tests/plugin-fork.test.ts => not ok 1 (before) / ok 1 (after) Pin the mode in the case and restore the prior value in finally, so the case states its own precondition and cannot be re-broken by an unrelated suite's cleanup order.
…nstead of failing closed silently The ranxianglei#5144 cold-restored snapshot path fails closed with a bare SNAPSHOT_UNAVAILABLE when any of its five reconstruction gates misses. On master CI the ranxianglei#5140 regression subtest has been red shard-after-shard with zero signal about WHICH gate fired — the five null returns were indistinguishable from the server log and from the client-visible error. Every miss now records its gate (empty-lastPassIds / replicasUnavailable / empty-tailSeqs / null-tailViewRows / gap-tailRow / unparseable-row / bad-row-shape / length-mismatch / id-mismatch / cap-exceeded), logs one warn line per session+gate per 10s, and appends [reconstruction: <gate>] to the thrown SNAPSHOT_UNAVAILABLE message so a 409 body says exactly why the tail could not be rebuilt. Behavior is unchanged — same statuses, same codes, same base messages; pure observability ahead of the root fix.
Closes #5140
What & why
A non-retained plugin parent intentionally does not persist its raw
pluginSnapshot— the retention gate (src/persist.ts) only writes it after a fork receipt / retained-snapshot contract. So after a cold restart,GET /__bili/plugin/snapshotfailed 409 SNAPSHOT_UNAVAILABLE for such a parent even thoughindex.dbstill held the independently recorded, unfolded canonical raw tail. Retrying the same cold source could never manufacture the missing resident copy, and child adoption lost access to already-stored compression work. This is item ④ of #4266.Fix (
src/plugin.ts)forkSnapshot(session, source?)now falls back to a new pure helperreconstructRawTailForFork(session)when the resident snapshot is absent:unifiedTailSeqs+unifiedTailViewRows), never append-log order or the folded envelope;idmatch againststate.lastPassIds, UTF-8 byte cap;GET establishes no permanent retention; only a successful fork materializes the verified source onto the parent (retention-on-success), so the existing
flushSyncpersists it for continuity.Intended behavior change (disclosed)
GET /__bili/plugin/snapshotfor a cold-restored non-retained parent whose verified raw tail survives in the unified storeWhy: recovering already-stored source material must not require a fresh parent model turn (#5140's core ask). It fails closed everywhere the source cannot be verified, so no partial/mis-attributed snapshot is ever served.
Consequently
tests/plugin-fork.test.ts's "self-heals on the next model request" test is updated to the new contract (its pre-model-request assertion flips 409→200); its post-model-request resident-view check is preserved.Tests
New
tests/issue5140-cold-raw-tail-fork.test.ts— synthetic cold close/reopen HTTP inheritance:BILI_PUBLIC_SNAPSHOT_CAP_BYTES=1in phase 2 → 409;Targeted suites green locally: issue5140 (3/3), dsh-fork-advantage + plugin-fork (54/54), fork-replica-convergence, dsh/pi-omp/opencode fork-adopt + issue3573-cold-load (47/47). Full-suite verdict = CI.
Scope / review notes