Skip to content

[PR] docs(storage): state planIngestDelta's strict-append precondition and pin the #4182 duplicate-identity corpus - #4399

Merged
ranxianglei merged 3 commits into
masterfrom
2026-10-23_issue4182-delta-duplicate-contract
Oct 24, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
2026-10-23_issue4182-delta-duplicate-contract

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #4182

What

#4182 (verif BEX equivalence comparison) found that planIngestDelta's raw plan diverges from planIngest when the NEW tail portion carries an already-stored identity (resent/replayed message — content-derived ids make a resend the same identity): full dedups via seqByIdentity ([A,B,A] → 2 rows, tailSeqs=[0,1,0]), delta appends unconditionally (tailSeqs=[0,1,2]).

Triage on current master established two layers:

  1. Planner level: the divergence is real and by design — the delta planner cannot see stored prefix identities (that is the point of skipping them).
  2. Store level: the issue's claimed runtime consequence (extra rows / watermark drift) does NOT occur. UnifiedStore.ingestSession's delta branch carries the strictly-append guard since 影子存储(默认开启)每次落盘把整个会话重新序列化+哈希,大会话明显变卡;jsonWriter=off 时连防抖合并都没了 #3133 (both SQLite store.ts and MySQL mysql.ts backends): inside its transaction it probes every new message's identity against the messages table and refuses the whole plan on any hit; the caller then runs the exact full path, whose dedup makes the committed state equal to a pure full ingest. Verified end-to-end through UnifiedShadow.record() for all five shapes from the issue thread (adjacent / middle / double / interior-prefix duplicates + no-duplicate control): row count, watermark, tailSeqs and served view all equal the full-ingest oracle; cost is one refused delta attempt + one redundant full write per resend event, and the next ordinary save rides the delta again.

Changes (no runtime behavior change — comment + test diff only)

  • src/storage/ingest.ts: planIngestDelta's doc now states the strict-append precondition explicitly (previously "resulting store state equals a full ingest's" was unconditional), names the store-layer guard that neutralizes violations, and states the violation cost. finishPlan's doc gets the same "given the same log" precision.
  • tests/storage-ingest-watermark.test.ts: two new tests — (a) row-driven matrix over the issue's five-shape corpus asserting rows/watermark/tailSeqs/served-view equality against a pure-full-ingest oracle, plus the no-duplicate control still riding the delta (the guard must not over-refuse); (b) planner-level pin of the intentional divergence ([0,1,2] vs [0,1,0]) so a future change moves the guarantee loudly instead of silently.

Behavior-change flag

None. No code path, default, threshold, timing, wire shape or persistence format changes; runtime behavior is byte-for-byte what it was (the guard already enforced the guarantee this PR documents and pins).

Verification

  • npm run typecheck: clean
  • node --import tsx --test tests/storage-ingest-watermark.test.ts: 16/16 pass (incl. the 2 new)
  • node --import tsx --test tests/issue3919-shadow-drift-forensics.test.ts: 3/3 pass (other planIngestDelta consumer)
  • Full-suite verdict: remote CI on this head.

Note: making the planners themselves equivalent (issue direction 1: knownIdentities parameter + old-seq reuse + store tailSeqs/watermark reconstruction changes in both backends) is deliberately NOT done here — it touches the storage hot path across four files to save one bounded, self-healing full re-ingest per resend event. If that is wanted it should be its own PR with a separate decision.

… pin the #4182 duplicate-identity corpus

planIngestDelta's doc claimed 'resulting store state equals a full
ingest's' unconditionally, but its raw plan diverges from planIngest
when the new tail portion carries an already-stored identity (resent/
replayed message): full dedups via seqByIdentity, delta appends a fresh
placeholder slot. The divergence is by design — the delta planner cannot
see stored prefix identities — and is neutralized at the store layer by
the strictly-append guard in UnifiedStore.ingestSession's delta branch
(#3133, SQLite + MySQL backends), which refuses such plans in-transaction
so the exact full path runs instead.

State that contract explicitly in planIngestDelta/finishPlan docs, and
pin it with two regression tests: a row-driven matrix over the #4182
five-shape corpus (rows/watermark/tailSeqs/served-view equal a pure
full-ingest oracle; no-duplicate control still rides the delta) and a
planner-level pin of the intentional [0,1,2] vs [0,1,0] divergence.

No runtime behavior change: comment + test diff only.

Closes #4182
@ranxianglei ranxianglei changed the title docs(storage): state planIngestDelta's strict-append precondition and pin the #4182 duplicate-identity corpus [PR] docs(storage): state planIngestDelta's strict-append precondition and pin the #4182 duplicate-identity corpus Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-23_issue4182-delta-duplicate-contract (61bbcd8)

Latest build — install this exact version

npm install -g billion-context@0.0.5334-pr.4399.e06d60b

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
61bbcd8 billion-context@0.0.5334-pr.4399.e06d60b build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4399.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 按 planIngestDelta、duplicate identity、strictly-append 等关键词检索 open+closed issue 与 PR:与 #4182 相关的在途工作只有本 PR 与父 issue #4182 本身;关闭的 #3948/#3166/#3919/#3437 是不同机制(不同失败域),#3812(已关)属快照渲染签名问题,主题相近但机制不同。无并行修复 PR。

标签: pr(保留)+ scope/storage(存储层持久化路径)+ p3(纯注释+测试,无运行时行为变更,低严重度)。不打 bug——本 PR 不是缺陷报告。

独立审核进展(不复述 PR 描述,逐项自验):

  1. 核心主张已独立验证成立:store 层 strictly-append guard 在两个后端都存在且行为如文档所述(SQLite src/storage/store.ts:591-603 事务内逐条 identity 探测,命中即整体拒绝;MySQL src/storage/mysql.ts:324-348 同构,约束 uq_messages_identity (session_id, msg_id, content_hash));调用方 src/storage/shadow.ts:346 delta 被拒后走精确全量路径(planIngest + completeLog),五种重发形状确实收敛到纯全量状态。planner 层分歧真实存在且是设计使然(ingest.ts:310-331 无条件 seq = watermark+1+log.length vs planIngest 的 seqByIdentity 去重)。
  2. 本地定向测试 19/19 绿(storage-ingest-watermark 16 + issue3919-shadow-drift-forensics 3,含新增 2 个)。
  3. 发现两个问题,均不阻塞本 PR 方向,处置如下:

进行中:merge master → 在注释里补上第二个 facet 的精确表述(纯注释修正,复用既有措辞)→ push 触发 CI core matrix → CI 结果出来后给八问结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论(最终)— 八问

  1. 是什么问题:Storage fast path: resent duplicate messages bypass identity deduplication; rows written via the fast and slow paths are | 存储快路径:重发的重复消息绕过身份去重,快慢两条路径写入的行不一致 #4182(verif 求解器等价性验证)发现新 tail 含已存储身份(重发/重放消息)时,planIngestDelta 的 raw plan 与 planIngest 分歧。本 PR 把这个分歧作为有意设计写进文档并用测试钉住,不改运行时行为。
  2. 什么原因导致的:delta planner 按设计跳过已存储前缀(快路径的意义所在),看不到存储消息的身份,只能把重发追加为新槽位;全量路径用 seqByIdentity 去重。这是两个 planner 可见性不同的能力边界,不是编码错误。
  3. 影响范围与严重程度:无用户面影响(纯注释+测试)。底层分歧本身有界:每次重发事件一次被拒的 delta 尝试 + 一次冗余全量写,提交后状态与纯全量灌入一致。p3。
  4. 怎么修复的:planIngestDelta JSDoc 写明 strict-append 前置条件、中和违规的 store 层 guard、违规成本;新增两个测试——五形状语料矩阵(rows/watermark/tailSeqs/服务视图对纯全量 oracle,含无重复对照仍走 delta 的反向断言)+ planner 层分歧钉([0,1,0] vs [0,1,2])。
  5. 是否有回归问题:无。diff 仅触及 src/storage/ingest.ts 的 JSDoc 块与测试文件新增;planIngestDelta/finishPlan/planIngest 函数体、store guard、shadow 回退语义经 git diff 逐行确认未动。审过的既有行为:delta 拒绝条件集(null 条件 8 项)、事务内 identity 探测、boundary/head probe、双后端一致性——全部保持。
  6. 是否破坏缓存:不涉及(未改任何缓存/重试/游标/去重键策略,只是把既有的文档化)。
  7. 是否改了设计方向:没有。明确不实现 issue 方向 1(planner 自身等价化),维持「planner 有意分歧 + store 层 guard 中和」的现有架构。
  8. 是否引入新配置:没有。

独立验证证据(不复述 PR 描述):

审核中发现并已处置的两处:

  1. 原 head 的 CI 从未运行:c74a23c2a 基于 c90e6ea 之前的 master,其 ci.yml 带重复顶层 concurrency 键,GitHub 判工作流无效(push run 0 job 秒挂,该 commit 上也没有 ci.yml 的 pull_request suite),PR 里「Full-suite verdict: remote CI on this head」实际未达成。已把当前 master 合入分支(master 侧仅 ci.yml 差异,与本 PR 文件零重叠、无冲突),CI 现已正常跑完并全绿。
  2. 发现文档未覆盖的既有缺陷面(非本 PR 引入):同一次保存的 tail 里出现两条全新字节相同消息时,guard 探测两者都 miss(都未入库),第二条 insert 被 UNIQUE 静默跳过,tailSeqs 留悬空槽位,该轮回放视图少一条,下次保存自愈。已 probe 实证(rows=3/tailSeqs=[0,1,2,3] 第 4 槽悬空/视图缺第二条 C;下一轮恢复 [0,1,2,2,3])。处置:doc 注释补上该 facet 的精确表述(commit 61bbcd81e,标注为 review 顺手修正);根治(guard 扩到 intra-plan seen-set,每后端约 5 行)属回退行为变更,等 owner 拍板,已按同一机制家族列入 Storage fast path: resent duplicate messages bypass identity deduplication; rows written via the fast and slow paths are | 存储快路径:重发的重复消息绕过身份去重,快慢两条路径写入的行不一致 #4182 子缺陷清单(见 Storage fast path: resent duplicate messages bypass identity deduplication; rows written via the fast and slow paths are | 存储快路径:重发的重复消息绕过身份去重,快慢两条路径写入的行不一致 #4182 评论),不开新 issue。

行为变更 flag:无(与 PR 正文一致)。

判定:可合并——代码绿、文档准确、测试钉牢、无回归。另有一项待 owner 决策:intra-plan facet 是否根治(选项与推荐见 #4182 评论)。合并动作留给 owner。

@ranxianglei
ranxianglei merged commit f7def41 into master Oct 24, 2026
47 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.801-master.f7def41 (rolling master tag).

npm install -g billion-context@0.0.801-master.f7def41  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant