Repository navigation
[PR] docs(storage): state planIngestDelta's strict-append precondition and pin the #4182 duplicate-identity corpus - #4399
Conversation
… pin the #4182 duplicate-identity corpus planIngestDelta's doc claimed 'resulting store state equals a full ingest's' unconditionally, but its raw plan diverges from planIngest when the new tail portion carries an already-stored identity (resent/ replayed message): full dedups via seqByIdentity, delta appends a fresh placeholder slot. The divergence is by design — the delta planner cannot see stored prefix identities — and is neutralized at the store layer by the strictly-append guard in UnifiedStore.ingestSession's delta branch (#3133, SQLite + MySQL backends), which refuses such plans in-transaction so the exact full path runs instead. State that contract explicitly in planIngestDelta/finishPlan docs, and pin it with two regression tests: a row-driven matrix over the #4182 five-shape corpus (rows/watermark/tailSeqs/served-view equal a pure full-ingest oracle; no-duplicate control still rides the delta) and a planner-level pin of the intentional [0,1,2] vs [0,1,0] divergence. No runtime behavior change: comment + test diff only. Closes #4182
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.5334-pr.4399.e06d60bThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4399.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 按 标签: pr(保留)+ scope/storage(存储层持久化路径)+ p3(纯注释+测试,无运行时行为变更,低严重度)。不打 bug——本 PR 不是缺陷报告。 独立审核进展(不复述 PR 描述,逐项自验):
进行中:merge master → 在注释里补上第二个 facet 的精确表述(纯注释修正,复用既有措辞)→ push 触发 CI core matrix → CI 结果出来后给八问结论。 |
…-append precondition (#4182)
(ai: vllm-qwen/qwen3.8-27b) 审核结论(最终)— 八问
独立验证证据(不复述 PR 描述):
审核中发现并已处置的两处:
行为变更 flag:无(与 PR 正文一致)。 判定:可合并——代码绿、文档准确、测试钉牢、无回归。另有一项待 owner 决策:intra-plan facet 是否根治(选项与推荐见 #4182 评论)。合并动作留给 owner。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.801-master.f7def41 # this exact build
npm install -g billion-context@master # always the newest merged build
|
Closes #4182
What
#4182 (verif BEX equivalence comparison) found that
planIngestDelta's raw plan diverges fromplanIngestwhen the NEW tail portion carries an already-stored identity (resent/replayed message — content-derived ids make a resend the same identity): full dedups viaseqByIdentity([A,B,A] → 2 rows, tailSeqs=[0,1,0]), delta appends unconditionally (tailSeqs=[0,1,2]).Triage on current master established two layers:
UnifiedStore.ingestSession's delta branch carries the strictly-append guard since 影子存储(默认开启)每次落盘把整个会话重新序列化+哈希,大会话明显变卡;jsonWriter=off 时连防抖合并都没了 #3133 (both SQLitestore.tsand MySQLmysql.tsbackends): inside its transaction it probes every new message's identity against the messages table and refuses the whole plan on any hit; the caller then runs the exact full path, whose dedup makes the committed state equal to a pure full ingest. Verified end-to-end throughUnifiedShadow.record()for all five shapes from the issue thread (adjacent / middle / double / interior-prefix duplicates + no-duplicate control): row count, watermark, tailSeqs and served view all equal the full-ingest oracle; cost is one refused delta attempt + one redundant full write per resend event, and the next ordinary save rides the delta again.Changes (no runtime behavior change — comment + test diff only)
src/storage/ingest.ts:planIngestDelta's doc now states the strict-append precondition explicitly (previously "resulting store state equals a full ingest's" was unconditional), names the store-layer guard that neutralizes violations, and states the violation cost.finishPlan's doc gets the same "given the same log" precision.tests/storage-ingest-watermark.test.ts: two new tests — (a) row-driven matrix over the issue's five-shape corpus asserting rows/watermark/tailSeqs/served-view equality against a pure-full-ingest oracle, plus the no-duplicate control still riding the delta (the guard must not over-refuse); (b) planner-level pin of the intentional divergence ([0,1,2] vs [0,1,0]) so a future change moves the guarantee loudly instead of silently.Behavior-change flag
None. No code path, default, threshold, timing, wire shape or persistence format changes; runtime behavior is byte-for-byte what it was (the guard already enforced the guarantee this PR documents and pins).
Verification
npm run typecheck: cleannode --import tsx --test tests/storage-ingest-watermark.test.ts: 16/16 pass (incl. the 2 new)node --import tsx --test tests/issue3919-shadow-drift-forensics.test.ts: 3/3 pass (otherplanIngestDeltaconsumer)Note: making the planners themselves equivalent (issue direction 1:
knownIdentitiesparameter + old-seq reuse + store tailSeqs/watermark reconstruction changes in both backends) is deliberately NOT done here — it touches the storage hot path across four files to save one bounded, self-healing full re-ingest per resend event. If that is wanted it should be its own PR with a separate decision.