Skip to content

[PR] fix: same-tick ingest dedup — skip the redundant writer-off build after ingestRowsNow (#6782) - #6786

Open
ranxianglei wants to merge 1 commit into
masterfrom
2026-11-15_6782-same-tick-ingest-dedup
Open

ranxianglei wants to merge 1 commit into
masterfrom
2026-11-15_6782-same-tick-ingest-dedup

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

What

When ingestRowsNow already ingested the current content in the same tick (the prepare phase, before the forward), the markDirty-triggered writer-off coalesced build was running a second full delta-ingest transaction (3-table full read + tail_json rewrite + tx commit) for an empty delta — the watermark was already advanced by the first ingest.

This adds a same-tick dedup signal: ingestRowsNow records the revision and lastMessages reference it ingested; the writer-off build skips the redundant guardedBuild when the proof revision is exactly +1 (only the unconditional markDirty bumped) and the tail reference is unchanged.

Why

#6782: every save's ingest transaction unconditionally full-reads the session's 3 ledger tables (refs/blocks/ccr) for final-state reconciliation — O(session size) fixed cost. 0.1.214 amplified this: ingestRowsNow (PR #6326, fixes #3397) + the writer-off microtask path (commit 5a80e2b) made the heavy transaction run TWICE per round pre-forward, the second being an empty delta that pays the full fixed cost for nothing. In the #6780 A/B scenario, ~half the ~3.2s pre-forward cost was this no-op.

How

  • ingestRowsNow now calls noteIngested(session) after a successful shadow.record (both branches), recording session.revisionEpoch and session.lastMessages reference.
  • The writer-off scheduleSave run() computes skipRedundantBuild: true when ingestedRevision is defined, proofRevision === ingestedRevision + 1, and ingestedTailRef === session.lastMessages. When true, it skips guardedBuild (the rows are already durable) but still runs the full settle protocol (noteDurable, lastSaveLanded, runIdleSpillIfIdle, drainFlushWaiters).
  • No semantic change: the rows are already durable from ingestRowsNow; noteOwnCommit was already recorded by it. noteMetaFp is a no-op in writer-off anyway.

Regression test

tests/issue6782-same-tick-ingest-dedup.test.ts — 3 tests:

  1. Same-tick ingestRowsNow + markDirty → the writer-off build skips the redundant guardedBuild (shadow.stats.ingests stays at 1, not 2).
  2. A content change between ingestRowsNow and the microtask (a new lastMessages reference) → the build is NOT skipped (ingests is 2).
  3. A revision bump beyond +1 (an extra markDirty) → the build is NOT skipped (ingests is 2).

Verification

  • npm run typecheck clean.
  • Targeted tests: tests/issue6782-same-tick-ingest-dedup.test.ts (3/3), tests/issue3397-durability-before-render.test.ts (2/2), tests/storage-save-path.test.ts (3/3), tests/storage-full-flip.test.ts (10/10).
  • Full suite: remote CI on the pushed head.

Closes #6782

@ranxianglei ranxianglei added bug Something isn't working p2 scope/storage labels Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-15_6782-same-tick-ingest-dedup (1d763bd)

Latest build — install this exact version

npm install -g billion-context@0.0.10098-pr.6786.5e47413

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
1d763bd billion-context@0.0.10098-pr.6786.5e47413 build
74c6aa5 billion-context@0.0.10087-pr.6786.c472e6a build
35a6af1 billion-context@0.0.10056-pr.6786.8244259 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6786.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: same-tick ingest dedup — skip the redundant writer-off build after ingestRowsNow (#6782) [PR] fix: same-tick ingest dedup — skip the redundant writer-off build after ingestRowsNow (#6782) Nov 15, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — screened the 100 most recent open PRs: #6786 is the only open PR referencing #6782; no parallel fix for this thread.

PR state (verified live, 2026-11-15 ~02:20Z): open, not draft, mergeable (no conflicts, 72 commits behind master f21b601893), head 2026-11-15_6782-same-tick-ingest-dedup @ 35a6af1724, 1 commit, 2 files (+241/−2): src/persist.ts (+48/−2), tests/issue6782-same-tick-ingest-dedup.test.ts (+195, new). Labels already set by the author: bug / p2 / scope/storage.

Code review (full diff read against the exact PR base 3858b6956a):

The fix implements direction 1 from #6782 (same-tick dedup, minimal change) exactly as proposed:

  • ingestRowsNow (src/persist.ts:3223-3262 on the PR branch) now calls noteIngested(session) after a successful shadow.record in BOTH branches (stale-disk :3244-3248, normal :3250-3258), recording session.revisionEpoch + the session.lastMessages reference into two new per-id maps (ingestedRevision / ingestedTailRef, :597-609). A failed ingest (catch) does NOT record — dedup can never suppress a save whose rows did not land.
  • The writer-off scheduleSave run() (:3297-3351) skips guardedBuild only when proofRevision === ingestedRevision + 1 (exactly the one unconditional markDirty bumped) AND the tail reference is unchanged — and still runs the full settle protocol: noteDurable, lastSaveLanded, runIdleSpillIfIdle, drainFlushWaiters.

PR claims verified against the code:

Path audit (which callers can/can't hit the dedup):

  • The deduped pattern is the prepare-phase one — snapshotMessages → ingestRowsNow → markDirty (e.g. src/prepare-anthropic.ts:619-628; same shape in prepare-google/openai/responses).
  • src/incremental-lane.ts:1055-1070 (ingest + scheduleSave, no markDirty) yields proofRevision === ingestedRevision (not +1) → the dedup does NOT fire — conservative, behavior unchanged.
  • convergeToDisk (src/persist.ts:3158-3186) may replace lastMessages (:3168) and calls markDirty (:3185) — the stale-disk branch records the post-converge state, sound.
  • Cross-generation safety: a reloaded session gets fresh arrays, so a stale entry from a previous generation can never match the tail-ref check.
  • Non-writer-off (debounced) stores: entries are recorded but never read — no behavior change there.

Tests (local, targeted — full suite is CI's job): checked out the PR head files onto the exact PR base and ran:

CI (verified live via mirror cache, ~02:20Z): 30 checks — 0 failed, 14 still pending (guard lanes queued, test matrix running), e2e lanes skipped per their gates, gate green. Verdict: NOT yet green — the full matrix is still in progress; I'm not claiming green.

One minor finding (non-blocking, for your call): ingestedTailRef holds the session's lastMessages array reference (the full raw request array) per distinct session id and is never deleted — an evicted session's last tail array is retained for the process lifetime. This matches the existing per-id map pattern (durableRevision/lastSaveLanded etc. are also never evicted-cleared), but those hold scalars while this holds large arrays. Options: (a) delete the entry in run() right after the dedup check — the signal is same-tick by design, and the only theoretical dedup miss (incremental-lane pattern) re-records every turn; (b) clear at the loadSync generation boundary like durableRevision (:1836); (c) gate noteIngested on writerOffActive() || envelopeFrozen() so non-writer-off stores never record. I did not push a change — per the [PR] rule I'm reporting, not modifying, unless you ask. My recommendation is (a): 2 lines, zero behavior risk.

Verdict: mergeable pending the full CI matrix going green. The fix is correct, minimal, well-tested, and matches the direction proposed in #6782. Suggested path: rebase onto master (72 behind), merge when the matrix is green; the memory-hygiene finding can be folded in as a follow-up commit or tracked separately — your call.


结构化八问(供人工 reviewer):

  1. 是什么问题: writer-off 模式下,每次保存的 ingest 事务都会无条件全量读取会话的 3 张账本表(固定成本 O(会话大小));0.1.214 起同一 tick 内这个重事务在 forward 前跑两遍——第二遍是空 delta 的冗余 build,白付全部固定成本(0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 A/B 中约占 ~3.2s pre-forward 成本的一半)。
  2. 什么原因导致的: ingestRowsNow 推进了水位,但随后的无条件 markDirty 仍触发 writer-off 合并 build,而 build 路径无法得知"水位已在同 tick 推进且内容未变"——于是对空 delta 重跑完整 delta-ingest。
  3. 影响范围与严重程度: 仅 writer-off(信封冻结)会话;每个请求轮次在 forward 前多付一次 O(会话大小) 的事务成本。性能退化,非正确性问题,会话完全可用(p2)。
  4. 怎么修复的: 同 tick 判重信号——ingestRowsNow 记录自己 ingest 的 revision + tail 引用;writer-off build 仅在 proof revision 恰好 +1(只有那次无条件 markDirty 递增)且 tail 引用未变时跳过冗余 guardedBuild,完整 settle 协议(noteDurable/spill/waiters)照常执行。ingest 失败不记录,判重永远不可能抑制未落盘的保存。
  5. 是否有回归问题: 已审——非 writer-off(debounce)路径不读新 map(无变化);incremental-lane 模式(无 markDirty)判重不触发(无变化);stale-disk 分支记录 converge 后状态(成立);跨代 reload 的新数组不可能匹配陈旧条目;noteOwnCommit/noteMetaFp/spill 门语义不变(已逐条验证)。一处小的内存卫生发现见上(非阻塞)。
  6. 是否破坏缓存: 否——未触碰任何缓存/重试/游标/去重键语义;判重键只是同 tick 内存信号。
  7. 是否改了设计方向: 否——按 Every save requires a full read of the three ledger tables (refs/blocks/ccr) for final-state reconciliation (O(session s | 每次保存都要全量读 refs/blocks/ccr 三张账本表做终态对账(O(会话规模) 固定开销);0.1.214 起该重事务每轮在转发前跑两次,第二次是空转 #6782 列出的 4 个候选方向中的方向 1(最小同 tick 判重)原样实现。
  8. 是否引入了新的配置: 否——无新 env/config/默认值。

收工判定: 可合并(等全量 CI 矩阵转绿 + rebase;内存卫生发现由 owner 另行决定)。

The mechanism this PR implemented (skip the redundant writer-off
guardedBuild after ingestRowsNow already covered the tick's state)
landed on master via #6791 (the #6780 fix) with a stricter gate
(durabilityArms: epoch+1, tail ref, no decompositionLog, no
contentStoreDirty, no pendingLedgerAlign, metadata bytes unchanged,
single-shot arm). The conflict rebase resolved src/persist.ts to the
master version; the PR's weaker ingestedRevision/ingestedTailRef gate
would have skipped builds master's gate deliberately runs, so it was
dropped. Only the behavioral regression tests remain — they assert
shadow.stats.ingests and pass against the master implementation.
@ranxianglei
ranxianglei force-pushed the 2026-11-15_6782-same-tick-ingest-dedup branch from 74c6aa5 to 1d763bd Compare November 15, 2026 05:49
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/vllm-qwen/qwen3.8-27b)

Merge conflict resolved and pushed. The PR was dirty because it and master's #6791 (the #6780 fix, merged today 12:48 +0800 — 9 minutes after this PR's commit was created) independently implemented the same same-tick ingest-dedup mechanism in the same region of src/persist.ts (ingestRowsNow call sites + the scheduleSave writer-off branch). I rebased onto master, resolved the conflict in favor of master's implementation, and force-pushed 74c6aa5429 → 1d763bd4c4. The PR is now mergeable (state blocked = awaiting human merge). CI will re-run on the new head.

Key judgment in the resolution: master's #6780 mechanism (durabilityArms / armDurabilitySkip) is a strict superset of this PR's (ingestedRevision / ingestedTailRef / noteIngested) — same epoch+1 + tail-ref conditions, plus four extra safety gates (no decompositionLog, no contentStoreDirty, no pendingLedgerAlign, metadata bytes unchanged), single-shot arm semantics (review F2), and arm invalidation in removeSession / consumePendingLedgerAlign (F1/F5). This PR's gate is strictly looser: merged alongside master's, it would have skipped builds that master's gate deliberately runs (decomposition payload / ledger-align restamp / CCR re-key would go unpersisted). So the PR's src/persist.ts changes were dropped; the behavioral test file was kept — it asserts shadow.stats.ingests (not implementation) and passes against master's mechanism. The commit message was amended to reflect the now test-only content.

Verified locally: npm run typecheck clean; targeted tests green — tests/issue6782-same-tick-ingest-dedup.test.ts 3/3, tests/issue6780-pre-forward-dedup.test.ts 4/4, tests/issue3397-durability-before-render.test.ts + tests/storage-save-path.test.ts 5/5. Full-suite verdict is the remote CI lanes on the pushed head.

七问:

  1. 是什么问题:PR [PR] fix: same-tick ingest dedup — skip the redundant writer-off build after ingestRowsNow (#6782) #6786(Every save requires a full read of the three ledger tables (refs/blocks/ccr) for final-state reconciliation (O(session s | 每次保存都要全量读 refs/blocks/ccr 三张账本表做终态对账(O(会话规模) 固定开销);0.1.214 起该重事务每轮在转发前跑两次,第二次是空转 #6782 的 same-tick ingest 判重修复)与 master 合并冲突(merge state: dirty),是唯一阻塞项——旧 head 上 30 项 CI check 全绿。
  2. 什么原因导致的:本 PR 与 master 上的 [PR] 0.1.214 Flush to disk twice before each forwarding round: skip the second no-op ingest when state is unchanged (#6780) | 0.1.214 每轮转发前落盘两次:状态没变时跳过第二次的空转 ingest(#6780) #6791(0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 修复,今天 12:48 +0800 合入,比本 PR 的 commit 晚 9 分钟)各自独立实现了同一个判重机制,落在 src/persist.ts 的同一代码区(ingestRowsNow 两处调用点 + scheduleSave writer-off 分支),文本直接冲突。
  3. 影响范围与严重程度:表面只影响本 PR 无法合并;真正的风险在冲突解法——若保留本 PR 的机制与 master 并存,本 PR 较松的门(只查 epoch+1 + 尾引用)会跳过 master 较严的门(durabilityArms 另查 decompositionLog / contentStoreDirty / pendingLedgerAlign / metadata 指纹,且单发 + arm 失效)刻意要跑的 build,导致 decomposition payload、ledger-align restamp、CCR rekey 落不了盘。
  4. 怎么修复的:rebase 到 master,src/persist.ts 整体取 master 版(master 的 0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 实现是本 PR 机制的严格超集,本 PR 的 src 改动无任何 master 缺失的内容,且门更松,故丢弃);保留行为测试文件 tests/issue6782-same-tick-ingest-dedup.test.ts 原样(断言 shadow.stats.ingests 而非实现细节,对 master 实现通过);commit message 改写为 test-only 以匹配实际内容。
  5. 是否有回归问题:无。合入内容现在只有测试(+195 行,0 行 src 改动),master 行为逐项不变。已验证:typecheck 干净;定向测试 issue6782 3/3、issue6780 4/4、issue3397 + storage-save-path 5/5 全绿。
  6. 是否破坏缓存:不涉及(无 src 改动,master 上的判重机制未动)。
  7. 是否改了设计方向 / 剩余什么:设计方向未变。剩余 owner 决策:本 PR 现在功能上是 [PR] 0.1.214 Flush to disk twice before each forwarding round: skip the second no-op ingest when state is unchanged (#6780) | 0.1.214 每轮转发前落盘两次:状态没变时跳过第二次的空转 ingest(#6780) #6791 的重复(修复已在 master)。选项 A:合并这个 test-only PR(补 3 个行为 pin,观测点与 master 的 4 个 case 不同——ingest 计数 vs sidecar record 调用;注意 PR 正文写 "Closes Every save requires a full read of the three ledger tables (refs/blocks/ccr) for final-state reconciliation (O(session s | 每次保存都要全量读 refs/blocks/ccr 三张账本表做终态对账(O(会话规模) 固定开销);0.1.214 起该重事务每轮在转发前跑两次,第二次是空转 #6782",而 Every save requires a full read of the three ledger tables (refs/blocks/ccr) for final-state reconciliation (O(session s | 每次保存都要全量读 refs/blocks/ccr 三张账本表做终态对账(O(会话规模) 固定开销);0.1.214 起该重事务每轮在转发前跑两次,第二次是空转 #6782 的方向 2–4 仍未做,合入会自动关 Every save requires a full read of the three ledger tables (refs/blocks/ccr) for final-state reconciliation (O(session s | 每次保存都要全量读 refs/blocks/ccr 三张账本表做终态对账(O(会话规模) 固定开销);0.1.214 起该重事务每轮在转发前跑两次,第二次是空转 #6782,请确认是否合适);选项 B:将 [PR] fix: same-tick ingest dedup — skip the redundant writer-off build after ingestRowsNow (#6782) #6786 按 [PR] 0.1.214 Flush to disk twice before each forwarding round: skip the second no-op ingest when state is unchanged (#6780) | 0.1.214 每轮转发前落盘两次:状态没变时跳过第二次的空转 ingest(#6780) #6791 的重复关闭(master 的 tests/issue6780-pre-forward-dedup.test.ts 已覆盖该行为家族,含 single-shot 与 ledger-align case)。我的建议:B。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/vllm-qwen/qwen3.8-27b)

Stale blocker re-trigger — nothing to do this run. Verified live just now: the PR head is 1d763bd4c4 (the conflict-rebase pushed in my previous run), master is still 1ee510c528, and the mirror cache (fetched 05:51:33Z) reports mergeable_state: blocked — i.e. the conflict is gone; blocked means mergeable and waiting on the human merge, not dirty. CI on the new head is running: 30 checks, 0 failed, 5 pending. No action taken, no push made.

七问:

  1. 是什么问题:系统再次转发「merge state: dirty」阻塞通知;经实时核验,这是上一轮 force-push 之前的旧状态(stale trigger),当前无冲突。
  2. 什么原因导致的:镜像轮询的阻塞检测基于 force-push 前的旧 dirty 状态触发,晚于修复到达。
  3. 影响范围与严重程度:无影响——仅一次误触发通知,PR 实际状态健康。
  4. 怎么处理:本轮未做任何改动;实时核验了 PR head(1d763bd)、master(1ee510c)、镜像缓存 mergeable_state(blocked)与 check 状态(0 failed / 5 pending)。
  5. 是否有回归问题:无(未动任何代码)。
  6. 是否破坏缓存:不涉及。
  7. 剩余什么:等 CI 在新 head 上跑完 + owner 合并(或按上轮建议按 [PR] 0.1.214 Flush to disk twice before each forwarding round: skip the second no-op ingest when state is unchanged (#6780) | 0.1.214 每轮转发前落盘两次:状态没变时跳过第二次的空转 ingest(#6780) #6791 重复关闭);owner 决策点不变。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment