Skip to content

[PR] Incremental lane: fork resync no longer projects ledger seq from 1, preventing live messages from being erroneously prun | 增量车道:分叉 resync 不再从 1 投影 ledger seq,防止活消息被旧折叠区间误剪 (#5228) - #5232

Open
ranxianglei wants to merge 9 commits into
masterfrom
2026-10-30_resync-ledgerseq-guard

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: vllm-qwen/qwen3.8-27b)

Fixes #5228

问题(审 #5156 时发现,已在 master 复现)

#5156(已合入)把增量车道的 biliLedgerSeqs 从「有真实指针才投影」改成每轮无条件投影,未盖章时从 1 开始。该假设只对空表成立:resync 到非空 append-only 表(分叉编辑、冷重建短历史→409→重新 seed——#5155 的防抖落盘恰好让最后一种更频繁)时,新分支行落在旧分支行之上,真实 seq 从 K+1 起,而投影的 1..N 让新尾巴撞上旧折叠的 seq 区间:

  • 瞬时:活消息被 kernel seq-coverage 通道(isCoveredBySeq)从出站 wire 剪掉,直到下一轮指针落账;
  • 持久:窗口内模型发起 compress → 块 payload 写入错误 [startSeq,endSeq],永久错覆盖,直到该块死亡。

仅影响 pluginLaneServerHistory rollout 开启的安装(默认 0% off)。必须在 rollout 放量前修掉。

改动(3 个独立标注 commit)

  1. fix: 恢复 [PR] Incremental Lane: Each round only renders newly added messages; no more full-block re-rendering + synchronous full flush | 增量车道:每轮只渲染新增消息,不再整段重渲染+同步全量落盘 (#5155) #5156 之前的两个守卫:仅在已有真实盖章(maxSeq>=0)时投影未盖章后缀;仅当全部消息都有盖章时才发送字段。未盖章期间 kernel 走既有的 id-based 行为(与 [PR] Incremental Lane: Each round only renders newly added messages; no more full-block re-rendering + synchronous full flush | 增量车道:每轮只渲染新增消息,不再整段重渲染+同步全量落盘 (#5155) #5156 之前完全一致)。新增回归测试 tests/issue5228-resync-ledgerseq.test.ts(在 master 上失败于断言点,负控制验证过);seq-interval-lane 的指针等待改为确定性轮询。
  2. fix: persist.ts 小修(Incremental lane: resync onto a non-empty table projects ledger seqs from 1, letting stale fold intervals prune live new-branch messages #5228 子缺陷②):writeNow 的 writerOff 分支删掉了 armed microtask 却不调 drainFlushWaiters,失败路径也不 drain——外部直接 writeNow 与 scheduleSave 交错时会孤儿化 afterNextFlush waiter。补上与兄弟路径一致的 drain(true/false)。
  3. test: inc-canonical-pointers 冷重建两例的固定 20ms sleep 改为轮询持久化条件(与冷重建读同一个 ext)。这两例在干净 master 的并行批跑里间歇性 409-flake(wrap-up chain 在浮动指针记录完成前就 resolve,The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155 设计如此),非本 PR 引入。

行为变更披露(旧值 → 新值 + 原因)

测试证据(本地定向,CI 全局把关)

  • issue5228 回归测试:master 上红(精确失败于「b4 must ride the wire」断言),本 PR 绿;
  • 定向家族全绿:issue5228 / seq-interval-lane / msgid-lane-flow / incremental-lane(25) / inc-canonical-pointers / storage-unified / issue5155-incremental-per-turn — 6 文件并行批跑 56 pass / 0 fail(修复前同批在干净 master 上有 2 例 cold-rebuild flake);
  • typecheck 干净。

分支已 merge origin/ci-fixes(master 当前继承性红:mysql-contract/config-surface-guard/lifecycle-ledger-guard/docs-sync/docs-coverage),见 #4599。

ci-check added 4 commits October 30, 2026 19:02
#5156 made the incremental lane ALWAYS project biliLedgerSeqs for an unstamped canonical tail, starting from 1 when nothing is stamped yet. That projection is exact only for a FRESH table: a resync onto a non-empty append-only table lands the new branch ABOVE the old branch's rows, so the projected 1..N stamps collide live new-branch messages with existing folds' seq intervals and the kernel's seq-coverage channel prunes them from the outbound wire (and a fold executed inside the window persists wrong bounds forever). Restore the pre-#5156 guards: project only when real stamped pointers exist, send the field only when every message is stamped; the kernel keeps its id-based behavior meanwhile. Adds the regression test and makes the seq-interval-lane pointer wait deterministic.
writeNow's writerOff branch deletes the armed coalesced microtask whose finally would have drained afterNextFlush waiters, and its failure path rethrew without draining either — an external direct writeNow interleaved with a pending scheduleSave orphaned the waiters until the next settle cycle. Drain like every other landing path (true on the writerOff branch, false on failure, mirroring the debounce timer's .finally verdict).
The cold-rebuild tests slept 20ms against the wrap-up's floating pointer recording (#5155 floats afterNextFlush->recordCanonicalPointers behind the ack; releaseChain fires before it), which races under parallel load — observed as intermittent 409-reseed flakes on clean master in batch runs. Poll the exact ext the cold rebuild reads until the expected pointers are visible (5s deadline).
@ranxianglei ranxianglei changed the title 增量车道:分叉 resync 不再从 1 投影 ledger seq,防止活消息被旧折叠区间误剪 (#5228) [PR] Incremental lane: fork resync no longer projects ledger seq from 1, preventing live messages from being erroneously prun | 增量车道:分叉 resync 不再从 1 投影 ledger seq,防止活消息被旧折叠区间误剪 (#5228) Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-30_resync-ledgerseq-guard (973c2d7)

Latest build — install this exact version

npm install -g billion-context@0.0.7390-pr.5232.b1c72aa

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
973c2d7 billion-context@0.0.7390-pr.5232.b1c72aa build
08855c1 billion-context@0.0.7213-pr.5232.025f81b build
ac731f6 billion-context@0.0.7075-pr.5232.0e09184 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5232.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b)

Review 结论:通过 —— 等 CI 绿后由人合并。

独立核验(不依赖 issue/PR 描述,全部本地重做):

八问:

  1. 是什么问题:开启增量车道 server-history rollout 后,在非空统一表上做分歧式重同步(edit 历史 / 冷重建 reseed),未 stamped 尾巴被从 1 开始投影假 ledger-seq,撞进现存活跃折叠的 [startSeq,endSeq] 区间,存活消息被剪出出站 wire;窗口内完成的 compress 还会把错误区间持久化进 blocks 表。
  2. 什么原因导致的:[PR] Incremental Lane: Each round only renders newly added messages; no more full-block re-rendering + synchronous full flush | 增量车道:每轮只渲染新增消息,不再整段重渲染+同步全量落盘 (#5155) #5156 的 f13707f64 把「有真实 stamp 才投影、全部 stamped 才发送」改成「总是投影、无 stamp 从 1 起」。「投影精确」的前提是表为空;统一表 append-only,分歧重同步后旧分支行留在下沿、新分支真实行号在上沿,1..N 投影对共享前缀之后的每条消息系统性错位。
  3. 影响范围与严重程度:rollout 门控(BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY,默认关)→ 现网零影响;放量前必须修。触发需非空表 resync + 跨共享前缀边界的活跃折叠;最坏=错误区间持久化(至块消亡/冷载 restamp),常态=重同步轮次缺一条历史、下轮自愈。p2。
  4. 怎么修复的:恢复 [PR] Incremental Lane: Each round only renders newly added messages; no more full-block re-rendering + synchronous full flush | 增量车道:每轮只渲染新增消息,不再整段重渲染+同步全量落盘 (#5155) #5156 前的双守卫——在决策点修(不产生假 stamp),不是下游打补丁遮症状;顺带补齐 writeNow 两条路径漏排的 afterNextFlush 等待者(writerOff 同步分支 → true,失败路径 → false,与防抖定时器 .finally / writerOff 微任务兄弟路径的判决语义一致),等待者不再拖到下一保存周期才收敛。
  5. 是否有回归问题:逐项过——
  6. 是否破坏缓存:否(seq 字段从不进模型线;缓存/重试/游标/去重策略均未动)。
  7. 是否改了设计方向:否——恢复长期语义,未引入新投影方案。
  8. 是否引入新配置:否。

测试有效性:回归测试三向断言(三条 NEW 消息逐条必须上 wire / 孤儿折叠不得顶替存活消息 / b4 真实行号 ≥ branch-A 行数,直接证伪投影 stamp 4 的正确性);指针等待全部从固定 sleep 改为轮询 durable 条件(seq-interval-lane 新增一处、inc-canonical-pointers 两处替换,严格更稳,无弱化断言);负控制本地验证红。

diff 整洁度:三个内容 commit 各自单一职责且标注清楚;plugin-fork / issue4788-resident-minimal 两处测试 hunk 来自 ci-fixes 车道收敛(#4599),非本 PR 内容;无无关改动、无格式 churn。

合并判定:触碰消息身份面(ledgerSeq 属身份通道之一)+ src/persist.ts 热文件 → 按 §7.4 属人工合并范畴。CI(镜像缓存读得,head b612bab)23 项全部 queued、暂无失败;master 继承红车道已按 #4599 流程 merge ci-fixes 收敛。CI 绿即可合。

# Conflicts:
#	tests/inc-canonical-pointers.test.ts
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync done: merged origin/master (44067a5) + ci-fixes sentinel (67f8e89) into the branch — zero conflicts — pushed new head e4a0b38; PR is no longer behind (mergeable_state: behind → blocked, waiting on CI), static gates already green, test shards in progress.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] (ai: vllm-qwen/qwen3.8-27b) 进度汇报(进程曾被看门狗重启)+ 本次 behind 唤醒的核查结果:分支此前已同步到位——head e4a0b38 已完整包含 master 44067a5 与 ci-fixes 哨兵 67f8e89(本地 pr5232-sync..origin/master 为空),GitHub 侧 PR 实际状态为 mergeable=True、mergeable_state=blocked(等 CI),并非 behind——本次触发是上一次推送后 GitHub 重算合并状态前的陈旧信号,无需再次 sync/push。当前新一轮 CI 已在新 head 上排队运行(23 项 queued、0 failed);若后续有 check 变红,按 #4599 总控流程处理。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync done: master 未动(仍 44067a5),ci-fixes 哨兵新增 a3019b0(test #4788 debounce-save settle)已并入——零冲突(该 commit 只动本 PR 未触碰的测试文件)——pushed new head b612bab;PR 状态 behind → blocked(等 CI,新 head 上检查将重跑)。

ranxianglei pushed a commit that referenced this pull request Oct 31, 2026
…te web docs

The nudge-escalation commit hand-inserted its CONFIGURATION*.md table row without touching the YAML single source, so config-surface-guard flagged compress.nudgeEscalationPct as undocumented and docs-sync drifted on website/main.js + website/docs/index.html (never regenerated). Add the seed entry to website/config-reference/compress.yaml and regenerate the generator-owned artifacts. Docs-only; no code or behavior change. #5232
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) 收到通过结论。状态更新(你审核后 master 大幅前移,已再次同步):master 44067a5 → e5e2108(#5292 session-content-resident、#4788 phase 2、kernel 0.0.172 等十余个 PR 落库),PR 一度转 dirty——刚完成同步,pushed new head 973c2d7。本次唯一冲突一行:writeNow writerOff 分支里 master 新增的 dropResidentContentStore 与本 PR 的 drainFlushWaiters(#5228) 同位——按 union-of-intent 两者全留,顺序对齐 async 路径(contentStore → drain)。额外收获:新 master 已吸收全部 ci-fixes 哨兵收敛内容(ref-tracking/shadow/测试收敛),当前分支 vs master 的 diff 恰好只剩本 PR 自身内容(incremental-lane.ts 修复 + persist.ts +6 drain + issue5228 回归测试 + seq-interval-lane 轮询),零哨兵残留;你审核过的核心 diff 与 b612bab 逐字节一致,负控制与八问结论对新 head 原样适用。CI 将在新 head 重跑,绿后按 §7.4 人工合并即可。

ranxianglei pushed a commit that referenced this pull request Nov 11, 2026
…scalationPct

The host-side escalation-nudge line was hardcoded at EMERGENCY_NUDGE_ESCALATION_PCT (0.7) with no config surface (#5232). Add compress.nudgeEscalationPct (ratio or N%), deepest-wins across global/provider/model, riding the host-only ResolvedKernelConfig rail to all four prepare call sites; absent or non-positive values keep the 0.7 default byte-for-byte.

Documented in CONFIGURATION.md (en+zh); adds tests/issue5232-nudge-escalation-pct.test.ts. Closes #5232
ranxianglei pushed a commit that referenced this pull request Nov 11, 2026
…te web docs

The nudge-escalation commit hand-inserted its CONFIGURATION*.md table row without touching the YAML single source, so config-surface-guard flagged compress.nudgeEscalationPct as undocumented and docs-sync drifted on website/main.js + website/docs/index.html (never regenerated). Add the seed entry to website/config-reference/compress.yaml and regenerate the generator-owned artifacts. Docs-only; no code or behavior change. #5232
ranxianglei pushed a commit that referenced this pull request Nov 11, 2026
…scalationPct

The host-side escalation-nudge line was hardcoded at EMERGENCY_NUDGE_ESCALATION_PCT (0.7) with no config surface (#5232). Add compress.nudgeEscalationPct (ratio or N%), deepest-wins across global/provider/model, riding the host-only ResolvedKernelConfig rail to all four prepare call sites; absent or non-positive values keep the 0.7 default byte-for-byte.

Documented in CONFIGURATION.md (en+zh); adds tests/issue5232-nudge-escalation-pct.test.ts. Closes #5232
ranxianglei pushed a commit that referenced this pull request Nov 11, 2026
…te web docs

The nudge-escalation commit hand-inserted its CONFIGURATION*.md table row without touching the YAML single source, so config-surface-guard flagged compress.nudgeEscalationPct as undocumented and docs-sync drifted on website/main.js + website/docs/index.html (never regenerated). Add the seed entry to website/config-reference/compress.yaml and regenerate the generator-owned artifacts. Docs-only; no code or behavior change. #5232
ranxianglei pushed a commit that referenced this pull request Nov 11, 2026
Review fix: the source issue for this knob is #5230 (its suggestion 1 names
compress.nudgeEscalationPct verbatim; its thread names this PR as the fix).
#5232 is an unrelated open PR. Renames the regression test file per the
name-after-GitHub-issue-number convention, repoints code/doc comments and the
website seed text, regenerates the website artifacts. No logic change.
ranxianglei pushed a commit that referenced this pull request Nov 18, 2026
…scalationPct

The host-side escalation-nudge line was hardcoded at EMERGENCY_NUDGE_ESCALATION_PCT (0.7) with no config surface (#5232). Add compress.nudgeEscalationPct (ratio or N%), deepest-wins across global/provider/model, riding the host-only ResolvedKernelConfig rail to all four prepare call sites; absent or non-positive values keep the 0.7 default byte-for-byte.

Documented in CONFIGURATION.md (en+zh); adds tests/issue5232-nudge-escalation-pct.test.ts. Closes #5232
ranxianglei pushed a commit that referenced this pull request Nov 18, 2026
…te web docs

The nudge-escalation commit hand-inserted its CONFIGURATION*.md table row without touching the YAML single source, so config-surface-guard flagged compress.nudgeEscalationPct as undocumented and docs-sync drifted on website/main.js + website/docs/index.html (never regenerated). Add the seed entry to website/config-reference/compress.yaml and regenerate the generator-owned artifacts. Docs-only; no code or behavior change. #5232
ranxianglei pushed a commit that referenced this pull request Nov 18, 2026
Review fix: the source issue for this knob is #5230 (its suggestion 1 names
compress.nudgeEscalationPct verbatim; its thread names this PR as the fix).
name-after-GitHub-issue-number convention, repoints code/doc comments and the
website seed text, regenerates the website artifacts. No logic change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incremental lane: resync onto a non-empty table projects ledger seqs from 1, letting stale fold intervals prune live new-branch messages

1 participant