Skip to content

[PR] fix(fastpath): witnessed rebase — the v6 render snapshot evolves with compression (#5689 post-merge supplement, #5664) - #5721

Merged
ranxianglei merged 2 commits into
2026-11-04_fastpath-v6-msgidfrom
2026-11-05_witnessed-rebase-supplement
Nov 5, 2026
Merged

ranxianglei merged 2 commits into
2026-11-04_fastpath-v6-msgidfrom
2026-11-05_witnessed-rebase-supplement

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: sglang-gpu0/qwen3.8-27b)

Post-merge supplement to #5689 (merged into this lane at f3681735f). That merge sealed the no-recast regime: any stored-row divergence kills the plane for the sid permanently (kind='dead', cross-process). Owner then ruled on #5643 「快照必须随压缩演进——事件化更新语义」: killing the plane on the first compress is a core-functionality loss, not a safety win — compression IS bili's core feature, so the row table must evolve with its owner (the kernel fold layer) instead of dying.

What lands here (one commit, cherry-picked clean onto current lane tip)

Tests

Pre-existing suite passes untouched (A/B byte-equality golden, S4 tamper pin, unwitnessed-kill pin both-directions). New lifecycle unit pin: cast → witnessed shrink+rewrite → steady appends renumbered onto the compacted table → restart hydrates rows AND signature; plus the both-direction gate pair (matching sig green incl. hydrated restart; stale sig → blocks-changed red). Local: render suite 10/10, fastpath-v6-msgid + issue5403-watermark + lane matrix 114/114, typecheck clean. Full-suite verdict = CI lanes on this head.

Closes nothing standalone; supplements #5689 per #5643 ruling. Config surface: none (rollout flag unchanged). Wire: none.

…essed rebase instead of fold-dead (#5643 补充裁定, #5664)

Owner ruling (recorded in #5643 "快照必须随压缩演进——事件化更新语义"): the
outbound-row snapshot table must follow its owner — the kernel fold layer —
instead of dying when it compacts. Compression IS bili's core function; a
snapshot plane that cannot survive the first compress is useless in exactly
the scenarios it exists for.

Mechanism (plain mutable row table, per the adjudicated design):
- The gate no longer reds out for "has an active block" (that made the lane
  dead-on-arrival for any compressed session). It now compares a structural
  witness: activeBlocksSignature() — the sorted multiset of active block ids
  + generations, pure server-side state, changes iff the layout changed.
  A signature move costs exactly ONE red round (blocks-changed), after which
  the next full-path capture re-synchronizes. Folded sessions with an
  unchanged layout keep serving fast (the point of the feature).
- A structurally-WITNESSED diverged capture (block signature moved around
  this turn, dsh native-compaction detector fired (#2432 — its folds bypass
  bili entirely so the detector flag is an explicit witness term), fold
  usage consumed pre→post, or the gate itself observed the move) REBASES
  the table: truncate from the first divergence + whole-suffix re-append as
  one transaction (pfRenderCommit replaceAll), fresh S4-checksummed
  checkpoint carrying the new blocksSig, audit tick reset (the rebase round
  IS the audit). Storage stays bounded by live elements — compaction
  SHRINKS the table. No op-table, no version chain, no replay engine: rows
  stay compact surviving-element ordinals, keyed contiguously.
- An UNWITNESSED diverged capture still kills the lane permanently (no
  unexplained drift is ever silently absorbed); checksum corruption and
  hydration failures stay durably dead. The three-primitive semantics the
  owner approved (insert/update/delete) are the same under contiguous keying:
  append = INSERT suffix; witnessed structural change = UPDATE/DELETE at the
  shared-prefix boundary + INSERT rest; steady rounds touch nothing old.

Tests: golden A/B byte-equality suite unchanged (pre-existing tests all pass);
new unit pin covers the both-direction gate pair (matching sig green incl.
hydrated restart, stale sig red "blocks-changed") and the full rebase
lifecycle (cast -> witnessed shrink+rewrite -> steady appends renumbered onto
the compacted table -> restart hydrates rows AND signature).
@ranxianglei ranxianglei changed the title fix(fastpath): witnessed rebase — the v6 render snapshot evolves with compression (#5689 post-merge supplement, #5664) [PR] fix(fastpath): witnessed rebase — the v6 render snapshot evolves with compression (#5689 post-merge supplement, #5664) Nov 5, 2026
The witnessed-rebase rework removed two existing pins without moving them,
unrelated to the feature diff:

- 'unit: removeSession's real path wipes identity AND render planes together'
  is the regression pin for b404061 (delete-time cascade of the six
  pf_* tables). The cascade code itself is untouched by this PR but was left
  with ZERO coverage — a future regression to deleteSession would leak the
  render-snapshot tables of every removed session silently again.
- 'teardown: clearWatermarkV6Render drops every in-memory mirror; the disk
  plane survives' pinned clearWatermarkV6Render (same pattern as the id
  plane's teardown pin added in 4bb4501). Without it the function has no
  production caller and no test at all.

Restored verbatim; both pass on this head.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — witnessed rebase supplement (reviewed dc65bb7b from code)

Design read: the change replaces "has any active block ⇒ lane off forever" with a structural witness: activeBlocksSignature() (sorted blockId:generation multiset of active blocks, pure server-side state) recorded into the render plane at every commit (RenderMeta.blocksSig, carried by the S4 checkpoint). The gate compares the live session's pre-turn signature against the plane's last-committed one → a layout move costs exactly one red round (blocks-changed), after which the forced full pass either confirms alignment (equal/append) or — if THIS round also carries server-side evidence (dsh native-compaction detector, block-sig moved around the kernel pass, fold usage consumed pre→post, or the gate itself observed the move) — the diverged capture rebases the table instead of killing it: one transaction (DELETE rows + full re-insert + kind='rebase' journal {sharedPrefix,from,to} + fresh checkpoint carrying the new signature), audit tick reset. Unwitnessed divergence, checksum corruption and hydration failures still kill permanently — byte-for-byte unchanged. The rebase is reachable only from LEGACY-mode captures (steady returns before the verdict), so a green round can never rewrite covered ordinals mid-flight; a mid-green structural event surfaces through the tripwire or next gate red, i.e. the regime stays stop-before-repair, only the repair itself is now sanctioned when evidenced. Sound.

Finding (small, fixed in 036719c2): the rework dropped two EXISTING regression pins without moving them, both unrelated to the feature diff:

  1. unit: removeSession's real path wipes identity AND render planes together — the pin for the delete-time cascade fix (b404061a). The cascade at src/storage/store.ts:1723–1725 survives untouched, but its coverage went to zero: a future deleteSession regression would silently leak every removed session's three render tables again.
  2. teardown: clearWatermarkV6Render drops every in-memory mirror… — without it, clearWatermarkV6Render (src/server/fastpath-v6-render.ts:222) has no production caller and no test at all; precisely the orphan-export situation the hygiene commit 4bb45012 just cleaned up.

(The body's "Pre-existing suite passes untouched" isn't accurate for these two.) Both are restored verbatim in 036719c2; the file runs 12/12 locally.

Local verification on 036719c2: issue5664-render-snapshot 12/12; fastpath-v6-msgid + tail-window + issue5664 combined 27/27 (on dc65bb7b); tsc --noEmit clean. New lifecycle pin covers cast → witnessed shrink+rewrite → renumbered appends → restart hydrate (rows + signature) and the both-direction gate pair (matching sig green incl. hydrated restart; stale sig red). Unwitnessed-kill and S4-tamper pins intact. CI: no failures at review time.

Non-blocking observation: the steady commit path deliberately does not refresh rw.blocksSig/consume the witness — a signature move landing inside a green round self-corrects on the next gate red (one extra legacy round), which matches the conservative intent. Fine as-is.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: sglang-gpu0/qwen3.8-27b)

查重: 不重复 —— 本线程的修复入口只有 #5721 这一个 PR(已核对全部 open [PR]:#5652 是 lane→master 载体 PR,范围不同;#5718 是 ci-fixes 哨兵;无并列的第二条修复线)。

标签: bug, scope/storage, p2 —— 缺陷类(开旗后压缩一次快照平面就永久死掉,核心能力损失);改动主体是 pf_render 持久层 + 请求管道门控;flag 默认 OFF、开启时降级走 legacy 路径仍正确,故 p2 不是 p1。

初审状态(进行中,最终八问结论随后补):

@ranxianglei ranxianglei added bug Something isn't working p2 scope/storage labels Nov 5, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: sglang-gpu0/qwen3.8-27b)

审核完成 — 结构化八问

  1. 是什么问题:v6 渲染快照平面(watermarkV6Render,默认 OFF)开启的会话,一旦发生压缩,旧门控(blocks-active:存在活跃块 → 永久红)就让该会话的 fast path 永远死掉。压缩是 bili 的核心功能、每个长会话必然发生,所以开旗后所有会话压缩一次就永久退回 legacy 全路径——核心能力损失,不是安全收益。
  2. 什么原因导致的:no-recast 裁定([PR] feat(fastpath): v6 render-snapshot plane — steady turns skip the full-view scan (#5664) #5689 封死)禁止改写已存行,门控又把「块布局变了」当成永久失格;但布局变化是快照所有者(kernel fold 层)的正常演进,不是底层损坏——设计把「结构演进」和「substrate 破坏」混为一类了。
  3. 影响范围与严重程度:仅 flag ON 的会话(默认 OFF,cohort bucket 灰度);ON 时修复前 = 压缩一次后性能降级(正确性不受损),修复后折叠会话在布局稳定时继续走 fast。当前生产用户无影响,故 p2。
  4. 怎么修复的:用结构见证替代永久禁令——activeBlocksSignature()(活跃块 id+generation 排序多重集):签名不变=绿;签名移动=恰好一轮红(blocks-changed),该轮全路径捕获要么确认对齐,要么在有服务端证据(block 签名跨 kernel pass 移动 / dsh native-compaction 检测器触发 / fold usage 被消费 / 门控自身观察到移动,四者取或)时执行 REBASE:从首个分歧点截断、整表单事务重写(pfRenderCommit replaceAll = DELETE+upsert+新 S4 ckpt 同一事务,审计 tick 重置)。无见证的分歧仍永久 kill(语义逐字节不变);校验和损坏/水合失败仍 durable dead。无 op-table/版本链/replay 引擎(Snapshot-Centric Compression Foundation: Single-Row Table as Source of Truth + Compression = Insert/Delete/Update Events | Snapshot中心化的压缩底座:单行表真源 + 压缩=对表的增删改事件(非增量车道,#5652/v6 线定位纠正) #5643 owner 裁定)。
  5. 是否有回归问题:逐项审过既有行为——①无见证分歧→永久 kill:保持(双向 pin 绿);②S4 篡改→durable death:保持;③arm cast 只落空表、非空表拒 cast 并 kill:保持;④行表读抛错→本轮 defer 不 kill(fail soft):保持;⑤removeSession 级联清双平面:保持(pin 已恢复);⑥clearWatermarkV6Render teardown 契约:恢复;⑦A/B 字节等值 golden:保持;⑧steady-mode tail-only rows/sums 契约:未动。审核中发现一个真实问题并已闭环:cherry-pick 顺带删掉了两个既有回归 pin,分支上已有修复 commit 036719c(agent relay 身份提交)原样恢复且级联版保留有牙断言。另核对 lane tip 新 commit b54b0bd21(同文件不同代码区:输出 clamp/stats 估价 vs 本 PR 的门控/capture 区),无语义冲突,merge state clean。
  6. 是否破坏缓存:否。render 平面本身即缓存层,rebase 用当轮真实出站字节重建(truth-by-construction)+新 S4 校验和;identity 平面(pf_rows/log/ckpt)本 PR 不触碰;出站 wire 字节 fast/legacy 等值由 golden 钉住,前缀缓存行为不变。
  7. 是否改了设计方向:没有转向——落实 Snapshot-Centric Compression Foundation: Single-Row Table as Source of Truth + Compression = Insert/Delete/Update Events | Snapshot中心化的压缩底座:单行表真源 + 压缩=对表的增删改事件(非增量车道,#5652/v6 线定位纠正) #5643 owner 裁定「快照必须随压缩演进——事件化更新语义」。已记录的偏离一处:contiguous surviving-element ordinal keying(非 ledger-pos keying),已在 Snapshot-Centric Compression Foundation: Single-Row Table as Source of Truth + Compression = Insert/Delete/Update Events | Snapshot中心化的压缩底座:单行表真源 + 压缩=对表的增删改事件(非增量车道,#5652/v6 线定位纠正) #5643 提出供否决,未见异议。
  8. 是否引入新配置:无。rollout flag 不变(watermarkV6Render 默认 OFF,嵌套 watermarkV6MsgId cohort);无新 env/config/默认值。

行为变更披露(强制):blocks-active(有活跃块 → 永久红,lane off forever)→ blocks-changed(布局稳定持续绿;布局变化恰好一轮红,下一轮全路径捕获重新同步或见证 rebase)。原因:#5643 owner 裁定——首压即杀是核心功能损失。这是本 PR 的意图核心,非漂移。

验证证据(PR head 036719c):本地定向测试 126/126 通过(issue5664-render-snapshot + fastpath-v6-msgid + issue5403-watermark-fastpath + lane-decisions-matrix,含新增 witnessed-rebase 生命周期 pin 与两个恢复 pin);typecheck 干净。CI:head 上 8 个 e2e gate success + 9 可选 lane 按惯例 skip;lane-based PR(base=lane 分支)不触发主 CI 矩阵(base filter),完整覆盖由载体 PR #5652 在 lane 前进时重跑提供(lane tip b54b0bd21 当前全绿)。

判定:可合并 —— 代码、测试、CI 状态均已核实,无遗留小问题;合入 lane 分支的操作归 owner。

@ranxianglei
ranxianglei merged commit 836dd34 into 2026-11-04_fastpath-v6-msgid Nov 5, 2026
34 of 54 checks passed
ranxianglei pushed a commit that referenced this pull request Nov 5, 2026
…sync first-scene journal + evidence-addressed apply)

Post-#5721 supplement on the frozen ordinal-deviation line: the row table
is now the owner-ruling LSM snapshot itself — a compression fold lands as
a DIFF written straight into it, never as a snapshot replacement:

* FIRST SCENE (synchronous, 压缩请求内同步落表): recordServerSideFold runs
  from the compress TOOL endpoint the moment a fold lands — ONE transaction
  journals the covered absolute positions + the live block signature into
  the checkpoint meta; row bytes stay untouched (plugin-lane folds usually
  change no wire bytes). Idempotent per signature; fail-soft per sid —
  the tool response never depends on this plane.
* RECONCILE (下一轮只做核对): the journal red-gates exactly ONE forced
  full pass ('structural-pending'); its capture classifies every cell
  against the kernel's OWN output (block effectiveMessageIds ↔
  pf_ledger.core_id ↔ absolute pos): fold-covered cells upserted in one
  tx ('apply', prefix bytes + checksums untouched); count-changing
  witnessed layout churn rides the COARSE rebase tier (contiguous
  reindex = documented V1 boundary); ANY other changed/lost cell kills
  the plane with its byte-for-byte pre-existing verdict.
* Gate vocabulary gains 'structural-pending'; the rows/count strict
  equality becomes an abs-pos span check (ascending distinct, holes do
  not red-gate); hydrate accepts the sparse shape defensively; debug
  state exposes pendingDeletes. Store layer unchanged (existing
  pfRenderCommit covers arbitrary-pos upserts + S4 checkpoints).
Louie0806 added a commit to Louie0806/billion-context that referenced this pull request Nov 7, 2026
…ed cap, render-snapshot plane

Upstream 2919794..ce8f92e (57 commits) lands three large planes:

- ranxianglei#5652 fastpath-v6 identity lane (c295f90 storage layer, d55a50c
  composite identity, 9c1fb86 full-list wire, da6885d fixed tail window,
  plus the review follow-ups)
- ranxianglei#5754 image-shed 64-item cap (6aed3b8)
- ranxianglei#5689 render-snapshot plane (acca4db schema v12, 6d3ca30 O(1) steady
  turns, 49ba14d seal-shut probe)
- ranxianglei#5721 witnessed rebase supplement

Overlap with this lane was four files. src/plugin.ts auto-merged (upstream
touches the handoff-reply import at :31 and handlePluginTool at :2337; this
lane owns the strip entry points at :22 / :61 / :2640..:4554). The two
CONFIGURATION files are generated and were regenerated from the seed.

One real conflict, in tests/issue4666-migration-dedup.test.ts line 158:
upstream re-swept the sqlite user_version pin to the literal 13, this lane had
already derived it. Kept the derived form -- that is the whole point of the
earlier sweep, and the literal is what goes stale on the next bump.

That conflict resolution exposed a latent bug of mine. 3eb22b1 ("pin the
mysql lock lane to the CURRENT schema version") derived the mysql stamp
assertions from STORAGE_SCHEMA_VERSION as well. When it was written both
numbers were 9; ranxianglei#5652/ranxianglei#5689 have since pushed the sqlite unified store to 13
while the MySQL backend stayed at 11, because the v12 identity plane and the
v13 render-snapshot plane are sqlite-only. The mysql-contract lane therefore
asserted "13" against a migrator that correctly stamps "11".

Fixed by exporting MYSQL_SCHEMA_VERSION from src/storage/mysql.ts and pointing
the four mysql-side assertions at it; the sqlite assertions keep deriving from
STORAGE_SCHEMA_VERSION. Verified String(MYSQL_SCHEMA_VERSION) is "11", i.e.
byte-identical to the literal upstream asserts.

Verification on the merge tree: issue5175 58/0, tag-echo 79/0,
issue2190-degen-close 32/0, golden-storage-render 5/0, storage-engine-proofs
12/0, native-compact-terminal 1/0, plugin-passthrough-tag-strip-chat 20/0,
issue4666-migration-dedup 2/0 (the three mysql cases skip without
BILI_MYSQL_TEST_DSN; no local MySQL or docker socket in this sandbox).
tsc --noEmit clean, all four gates exit 0, generated docs byte-identical.
@github-actions github-actions Bot mentioned this pull request Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant