Repository navigation
[PR] fix: bound ordinary session loading and avoid full reloads for lazy fields - #4765
Conversation
…ment #4316 PR3/3 (restampRefsToLedger) re-homes citation numbers once on every full-mode cold load; goldens pinning exact ref numbers are born-red. Re-pin the 7 storage-lane tests (engine-proofs x6 + orphan-recovery x1) modulo that renumbering — zero production code touched. - golden-storage-drive.ts: per-turn citation-normalized digest (shaCiteNorm) and state digest modulo stripOneTimeAlignment (stateDigestAligned); the deep-sort helper is hoisted into golden-canonical.ts for sharing. - corrupt-index-child.ts: new stripOneTimeAlignment normalizer — strips the refLedgerAligned stamp and re-homes refs onto canonical raw-id order so a clean full-mode read and the legacy fallback load compare equal. - storage-engine-proofs.test.ts: wire legs assert shaCiteNorm equality vs the legacy restart leg and the committed goldens; state legs assert stateDigestAligned; rollback leg now spells BILI_STORAGE_UNIFIED=legacy explicitly (#3818 flip: absent resolves to full, deleting the env would vacate the legacy path). - storage-orphan-recovery.test.ts: pins the content-addressable byRaw->byRef round-trip instead of a counter-era m00003 literal. - golden/storage-render/*.json regenerated with shaCiteNorm/stateDigestAligned. Converges the master-red storage-lane family (#4599 plan A, owner directive).
…dger parity masks The #4316 one-time renumbering restamped messageRefs but left state.tokenSnapshot keyed by the OLD refs, so post-restart bodies carried stale per-message token counts (1.3K vs 1.4K) and the full-mode-restart wire drifted off the legacy baseline (#4599). persist.ts: two-pass remap (stage moves, delete old keys, then write new keys) so entries whose new ref collides with another move's old ref survive; fires only when superseded refs exist. Aligned normalizer (corrupt-index-child.ts): the cacheLedger's segment bookkeeping (be/bl/bs, unfinalized tail line), the derived agg block, and per-line miss-attribution (prov/seam) are restart experience, not content — no-restart and restart legs legitimately differ there. Masked in the aligned view only; raw digests keep full accounting. Goldens regenerated (stateDigest + aligned). storage-engine-proofs 12/12, golden-storage-render 5/5 locally.
…persisted state (#4599) The fork-survival test asserted immediately after the compress tool response, but the fold lands in the live session state asynchronously (#3573 replay semantics). Slow CI runners persisted the pre-fold state ('0 block(s) (compressions=0)' cold load) and the deepEqual diverged. Bounded 5s wait on the in-memory fold; fast machines exit on the first iteration. Product semantics unchanged.
The #4316 cold-load alignment reads the unified sidecar; its ingest of just-flushed records completes asynchronously. Slow runners compared the pre-ingest state and saw refLedgerAligned missing while the fold itself had landed. Poll (bounded 5s) until the loaded state carries the flag; fast machines exit on the first probe.
Git-for-Windows autocrlf checks the bundled .md posts out with CRLF; the exact-match '---' fences then failed to parse and every post lost its title/excerpt/date frontmatter (Windows CI: 'bundled meet-bili post found' assertion). Normalize CRLF to LF inside parseFrontmatter when the fence is CRLF-shaped; LF content is byte-identical as before. Regression test with a CRLF fixture.
… storage mode (#4599) The #4316 cold-load alignment sets refLedgerAligned only when the unified store is available; config-clean machines (CI runners) run legacy mode where it is an honest skip. The test asserted the flag unconditionally, so every shard carrying this file was born-red on CI while unified-config machines passed. Expectation now mirrors store.replicasAvailable(); verified 49/49 in BOTH modes locally.
… reads Ordinary full-mode loads materialized every message row (row + content-CAS read + media hydration) although the view resolves only the tail/pluginSnapshot pointer seqs; dropped metadata and blockContents each rebuilt the whole session through loadSync after every idle save. On a 5,043 message / 236 block session one isolated read measured 15.1 -> 230.1 MiB heap growth (#4739). - UnifiedStore.sessionViewRows: the viewFor bundle from ONE read transaction; messages fetched only at pointer seqs (getMessagesAt, chunked IN, same hydration/retired/order semantics as getMessages). - StorageBackend.sessionViewRows (mysql lane): same pointer-scoped fetch. - UnifiedStore.getSessionMetadata / getBlockContents: single-column / blocks+orphans-only reads mirroring loadLegacyView's replay leg. - SessionStore.lazySource + narrowMetadata/narrowBlockContents: one build-scoped source decision per saved record (index vs envelope), bound-source-first with the other copy as fallback; never imports, never schedules saves, refuse-empty throw semantics preserved. Served bytes unchanged: lastMessages/pluginSnapshot resolution, tailGaps counting, orphan journal merge, media hydration, drift-latch and index/envelope freshness rules all preserved. No schema/config/kernel changes.
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 — 独立检索 open+closed issue 无同症状/同机制条目;属统一存储读路径内存域,是 #3050(resident-memory fold)在读侧的收尾;与 #4529/#4538(流式 digest)、#4540(采样验证)机制不同。 标签: bug + p1 + scope/storage(另保留 pr)— 缺陷类、存储域;加载路径整 log 物化、内存只增不减,按 p1 口径。 审核结论先行:可合(CI 转绿后) — 独立通读全部 diff、重跑定向验证,served bytes 字节等价、测试有牙;review 中补了一处 mysql 车道契约测试缺口,已作为独立 commit 八问
测试有效性(质量关):新套件有牙——T1 的 messageRowReads delta 断言在回退为全量 fetch 时立即失败(40≠400),字节等价对并行参考解码断言,T3 跑真实 golden folded corpus。缺口:orphan-journal 合并路径在 corpus 里没有直接 case(结构镜像但未实测);drift latch 下 lazySource 绑定无测试。 review 中顺手修(已推送):新契约在 mysql 车道零覆盖(新套件仅 sqlite)→ 在 tests/storage-backend-contract.test.ts 补 sessionViewRows 指针范围契约 case(sqlite 恒跑 + mysql 车道 CI 带 DSN 跑),本地 7 pass/1 skip、typecheck clean。独立 commit 本地定向验证(head 上重跑):issue4739 套件 4/4;memory-{metadata,blockcontent,tail}-lazy 9/9;storage-unified 14 pass/1 skip;storage-backend-contract 7 pass/1 skip(含新 case);typecheck clean;unused-export gate clean。storage-orphan-recovery 10 pass/1 fail——纯 master bb7c249 本沙箱复现同样失败,系既有沙箱差异(PR 引用 #4599 floors 196-198 属实),与本 diff 无关;权威判定以 CI 为准。 CI 状态:检查时 7b1ecc2 的全部 check-run 仍在排队;新 head 41422f3 将触发重跑。若红,按 #4599 协调流程处理。 相邻发现: 判定:可合(待 CI 在 41422f3 上转绿),合并操作由人执行。 |
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.5950-pr.4765.24c468fThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4765.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…mspec wrap on win32 (#4599)
The ci-check lane's expectation change (c496cfb) exposed a real Windows bug: stripInheritedProxy builds inlineEnv as a spread copy of process.env, which keeps the original key casing (ComSpec) — Windows env vars are case-insensitive but plain objects are not. planClientSpawn's exact env.COMSPEC probe then missed it and silently degraded every wrapped spawn to the bare-name 'cmd.exe' fallback (verified by repro on a real win32 box: childEnv.COMSPEC=undefined while process.env.COMSPEC was set). Fix: probe COMSPEC/ComSpec/comspec in planClientSpawn and planDshSpawn. Verified on real Windows (Node 22): launcher-opencode-inline + fix-679 + dsh-refresh 37/37; host tsc clean, 35 pass / 2 win32-skips.
…OMSPEC case-probe c496cfb re-derived the win32 expectation from ambient process.env, which still mismatches production: Node's process.env member access is case-insensitive on Windows while the child env handed to the planner is a plain object whose key keeps the OS casing (ComSpec), so the exact env.COMSPEC probe fell through to the bare-name fallback. The assertion now derives from the child's OWN captured env/options and probes COMSPEC/ComSpec/comspec exactly like planClientSpawn does post-dfeea8a11 — self-contained under any concurrent-test env churn in single-process CI shards — and additionally pins windowsVerbatimArguments=true on the wrap.
ranxianglei#4538: sampled read-back verification in bounded batches of 32 (headers separately, blocks in persisted order, message hydration only at the verifier's pointer seqs); sync and async share one code path; SQLite and MySQL get bounded LIMIT/OFFSET + seq-IN reads. Conflict resolutions (p4540 base is pre-ranxianglei#5193/ranxianglei#4765/ranxianglei#4750; our side is the superset in every region): - src/storage/backend.ts: keep ours — p4540 added no content beyond ours (its sessionViewRows/readSnapshot/StorageReadView surface already present from ranxianglei#4765/ranxianglei#4750 merges); verified diff shows zero theirs-only lines. - src/storage/store.ts: union — p4540's getBlocks 'ORDER BY ord, block_id' total ordering (stable pagination across generations) onto our gc() batch-sweep constants, viewPointerSeqs, readSnapshot/readSnapshotAsync, activeForeignWriter, formatDbSize; comment kept ours' pluginSnapshotSeqs wording (documents the live field shape). - src/storage/mysql.ts: union — p4540's dedicated-pool reads with hydrateMediaAsync(conn) parameterization onto our v9 stamp + idx_content_refcount probe block + collectZeroRefIn lifecycle scoping; theirs-only lines audited line-by-line: all stale base versions of code our branch already carries (ranxianglei#5193 master state). - tests/plugin-fork.test.ts: keep ours' longer ranxianglei#4599 comment, replicasAvailable() gate (supersedes theirs' storageMode==="full"), and ours' ranxianglei#5140 raw-tail cold-restart test semantics (200, not 409); p4540's storageMode loop + env save/restore already on our side via ranxianglei#4765. - tests/storage-orphan-recovery.test.ts: take p4540's extra assert (alignedRef === "m00001") — pure addition, variable already defined.
ranxianglei#5155/ranxianglei#5144: incremental lane — serialize the wire prefix once per canonical entry and splice only the delta per turn (laneBodyBuffer sentinels with full-restring fallback); recordCanonicalPointers walks only rows appended after the last recorded pointer (windowed rawMessages(fromSeqExclusive)) with cluster-suffix (<base>_<n>) tolerant matching so pointers no longer silently freeze; wrap-up save debounces 500ms via scheduleSave instead of a synchronous writeNow+fsync on every turn. Conflict resolutions (both sides' features unioned): - src/storage/shadow.ts: keep ours' activeForeignWriter() (ranxianglei#4612) AND take p5156's optional fromSeqExclusive param on rawMessages() — the two edits sit on adjacent methods; final file carries both. - src/storage/store.ts: getMessages() opts unioned to { includeRetired?, seqs?, fromSeqExclusive? } — ranxianglei#4765's pointer-scoped seqs fetch + read instrumentation coexist with ranxianglei#5155's windowed scan (seqs takes priority when both are present; params assembled positionally). Async backend paths untouched by p5156, so no interface change required.
…anxianglei#4765/ranxianglei#4750 The two PRs added different getBlockContents call shapes (paged rows for bounded view decode vs full narrow contents map for lazy-field restore); merged as one union return type, which broke caller-side narrowing in the read-snapshot test. Declare the overload pair so paged callers get the row array and no-arg callers get the map. storage-bounded-load's read counter monkey-patch now casts to the implementation signature it already matches at runtime (every counted call passes opts.limit=32).
…l-log scan Found while working ranxianglei#4739 (reported in its thread and in PR ranxianglei#4765's review notes): UnifiedShadow.sessionTipSeq / sessionTipSeqAsync materialized the session's entire message log (every row + one content-CAS read + media hydration per row) just to reduce it to max(seq)+1 — the checkpoint insert's seq_at. O(N) row materialization per checkpoint on the write path. Rewire both legs onto ranxianglei#4730's existing lastLiveSeq (SELECT MAX(seq) over live rows; store/backend/mysql already carry it end-to-end): identical value (same live-row filter, empty log -> tip 0), fail-fast error propagation unchanged, zero new interface surface. tests/storage-tipseq-narrow-read.test.ts pins both halves: tip == max live seq + 1 AND zero message rows materialized (proven failing on the old scan).
(ai: vllm-qwen/qwen3.8-27b)
Closes #4739
What was wrong
On full-mode (unified storage) installations, every ordinary load of a persisted session materialized the ENTIRE message log — one row per message plus a content-CAS read and media hydration per row — although
loadLegacyViewconsumes only the tail/pluginSnapshot pointer rows (tail_json). Separately, after each idle save the two dropped lazy fields restored by callingSessionStore.loadSync— i.e. rebuilding the whole session TWICE per saved record. On a 5,043-message / 236-block session a single isolated read measured 15.1 → 230.1 MiB heap growth with 118.2 MiB retained after GC.How this fixes it
UnifiedStore.sessionViewRows: the exactviewForbundle (session + messages + blocks + deadRefs + refs) from ONE better-sqlite3 read transaction (WAL snapshot pinned across all five reads); messages fetched ONLY at thetail_jsonpointer seqs via newgetMessagesAt(chunked IN ≤500, same hydrated row shape / retired exclusion / seq ordering asgetMessages).StorageBackend.sessionViewRows(+ mysql lane): same pointer-scoped fetch; per-statement reads as today, safe because ingest writes each session row and its pointed-at message rows in one transaction.UnifiedStore.getSessionMetadata/getBlockContents: single-column metadata read, and a blocks+orphans-only read mirroringloadLegacyView's replay leg exactly (including the orphan-journal merge without resurrectingstate.blocks).SessionStore.lazySource+narrowMetadata/narrowBlockContents: ONE build-scoped source decision per saved record (index vs envelope, mirroringloadSync's routing incl. drift latches), bound-source-first with the other copy as fallback. The narrow path never triggersimportRecord, never schedules saves, never touches drift state; the refuse-empty throw semantics of both lazy proxies are preserved.Behavior changes (disclosed)
lastMessages/pluginSnapshotresolution,tailGapscounting, block/ref/deadRef contents, media hydration are byte-equal to the old full-log decode (asserted by test against a side-by-side reference decode).metadata/blockContentsrestore now reads one column / the block rows instead of rebuilding the session twice; values served are identical (index leg mirrors the replay leg; envelope leg parses the same payload fieldbuildSessionused).Test evidence
New hermetic suite
tests/issue4739-bounded-session-load.test.ts(4 tests): bounded-decode byte-equality vs full-log reference + exact row-read count viamessageRowReads; index-alone metadata restore after envelope deletion (no thinned erase on later save); narrow reads ==loadLegacyViewreplay leg on a real folded corpus;getMessagesAtsparse/chunked/missing-seq ordering. Targeted local runs green: this suite 4/4, memory-{metadata,blockcontent,tail}-lazy 9/9, storage-unified 14 pass/1 skip, backend-contract/pins/orphan-recovery/storage batches 19+35+38 pass, unused-export gate clean, tsc clean. Note:storage-orphan-recoveryshows 1/11 failing in THIS sandbox identically on pure master (documented sandbox artifact, master CI green at bb7c249 per #4599 floors 196-198) — not caused by this diff; authoritative verdict is CI on this head.Adjacent observation (not fixed here)
UnifiedShadow.sessionTipSeq(src/storage/shadow.ts:631) fetches the whole message log just to compute max(seq)+1 (single caller: checkpoint insert atsrc/persist.ts:2042). Same failure domain as #4739 but on the checkpoint write path, not ordinary loads — recommend tracking it as a batch-2 checklist item under #4739 rather than a separate issue.查重: 不重复 — follows #3050 (resident-memory fold, predecessor); distinct from #4529/#4538 (streaming digest / sampled verification) and #4540 (sampled verification lane).