Repository navigation
[PR] fix(fastpath): witnessed rebase — the v6 render snapshot evolves with compression (#5689 post-merge supplement, #5664) - #5721
Conversation
…essed rebase instead of fold-dead (#5643 补充裁定, #5664) Owner ruling (recorded in #5643 "快照必须随压缩演进——事件化更新语义"): the outbound-row snapshot table must follow its owner — the kernel fold layer — instead of dying when it compacts. Compression IS bili's core function; a snapshot plane that cannot survive the first compress is useless in exactly the scenarios it exists for. Mechanism (plain mutable row table, per the adjudicated design): - The gate no longer reds out for "has an active block" (that made the lane dead-on-arrival for any compressed session). It now compares a structural witness: activeBlocksSignature() — the sorted multiset of active block ids + generations, pure server-side state, changes iff the layout changed. A signature move costs exactly ONE red round (blocks-changed), after which the next full-path capture re-synchronizes. Folded sessions with an unchanged layout keep serving fast (the point of the feature). - A structurally-WITNESSED diverged capture (block signature moved around this turn, dsh native-compaction detector fired (#2432 — its folds bypass bili entirely so the detector flag is an explicit witness term), fold usage consumed pre→post, or the gate itself observed the move) REBASES the table: truncate from the first divergence + whole-suffix re-append as one transaction (pfRenderCommit replaceAll), fresh S4-checksummed checkpoint carrying the new blocksSig, audit tick reset (the rebase round IS the audit). Storage stays bounded by live elements — compaction SHRINKS the table. No op-table, no version chain, no replay engine: rows stay compact surviving-element ordinals, keyed contiguously. - An UNWITNESSED diverged capture still kills the lane permanently (no unexplained drift is ever silently absorbed); checksum corruption and hydration failures stay durably dead. The three-primitive semantics the owner approved (insert/update/delete) are the same under contiguous keying: append = INSERT suffix; witnessed structural change = UPDATE/DELETE at the shared-prefix boundary + INSERT rest; steady rounds touch nothing old. Tests: golden A/B byte-equality suite unchanged (pre-existing tests all pass); new unit pin covers the both-direction gate pair (matching sig green incl. hydrated restart, stale sig red "blocks-changed") and the full rebase lifecycle (cast -> witnessed shrink+rewrite -> steady appends renumbered onto the compacted table -> restart hydrates rows AND signature).
The witnessed-rebase rework removed two existing pins without moving them, unrelated to the feature diff: - 'unit: removeSession's real path wipes identity AND render planes together' is the regression pin for b404061 (delete-time cascade of the six pf_* tables). The cascade code itself is untouched by this PR but was left with ZERO coverage — a future regression to deleteSession would leak the render-snapshot tables of every removed session silently again. - 'teardown: clearWatermarkV6Render drops every in-memory mirror; the disk plane survives' pinned clearWatermarkV6Render (same pattern as the id plane's teardown pin added in 4bb4501). Without it the function has no production caller and no test at all. Restored verbatim; both pass on this head.
ranxianglei
left a comment
There was a problem hiding this comment.
Review — witnessed rebase supplement (reviewed dc65bb7b from code)
Design read: the change replaces "has any active block ⇒ lane off forever" with a structural witness: activeBlocksSignature() (sorted blockId:generation multiset of active blocks, pure server-side state) recorded into the render plane at every commit (RenderMeta.blocksSig, carried by the S4 checkpoint). The gate compares the live session's pre-turn signature against the plane's last-committed one → a layout move costs exactly one red round (blocks-changed), after which the forced full pass either confirms alignment (equal/append) or — if THIS round also carries server-side evidence (dsh native-compaction detector, block-sig moved around the kernel pass, fold usage consumed pre→post, or the gate itself observed the move) — the diverged capture rebases the table instead of killing it: one transaction (DELETE rows + full re-insert + kind='rebase' journal {sharedPrefix,from,to} + fresh checkpoint carrying the new signature), audit tick reset. Unwitnessed divergence, checksum corruption and hydration failures still kill permanently — byte-for-byte unchanged. The rebase is reachable only from LEGACY-mode captures (steady returns before the verdict), so a green round can never rewrite covered ordinals mid-flight; a mid-green structural event surfaces through the tripwire or next gate red, i.e. the regime stays stop-before-repair, only the repair itself is now sanctioned when evidenced. Sound.
Finding (small, fixed in 036719c2): the rework dropped two EXISTING regression pins without moving them, both unrelated to the feature diff:
unit: removeSession's real path wipes identity AND render planes together— the pin for the delete-time cascade fix (b404061a). The cascade at src/storage/store.ts:1723–1725 survives untouched, but its coverage went to zero: a future deleteSession regression would silently leak every removed session's three render tables again.teardown: clearWatermarkV6Render drops every in-memory mirror…— without it,clearWatermarkV6Render(src/server/fastpath-v6-render.ts:222) has no production caller and no test at all; precisely the orphan-export situation the hygiene commit4bb45012just cleaned up.
(The body's "Pre-existing suite passes untouched" isn't accurate for these two.) Both are restored verbatim in 036719c2; the file runs 12/12 locally.
Local verification on 036719c2: issue5664-render-snapshot 12/12; fastpath-v6-msgid + tail-window + issue5664 combined 27/27 (on dc65bb7b); tsc --noEmit clean. New lifecycle pin covers cast → witnessed shrink+rewrite → renumbered appends → restart hydrate (rows + signature) and the both-direction gate pair (matching sig green incl. hydrated restart; stale sig red). Unwitnessed-kill and S4-tamper pins intact. CI: no failures at review time.
Non-blocking observation: the steady commit path deliberately does not refresh rw.blocksSig/consume the witness — a signature move landing inside a green round self-corrects on the next gate red (one extra legacy round), which matches the conservative intent. Fine as-is.
(ai: sglang-gpu0/qwen3.8-27b) 查重: 不重复 —— 本线程的修复入口只有 #5721 这一个 PR(已核对全部 open [PR]:#5652 是 lane→master 载体 PR,范围不同;#5718 是 ci-fixes 哨兵;无并列的第二条修复线)。 标签: bug, scope/storage, p2 —— 缺陷类(开旗后压缩一次快照平面就永久死掉,核心能力损失);改动主体是 pf_render 持久层 + 请求管道门控;flag 默认 OFF、开启时降级走 legacy 路径仍正确,故 p2 不是 p1。 初审状态(进行中,最终八问结论随后补):
|
(ai: sglang-gpu0/qwen3.8-27b) 审核完成 — 结构化八问
行为变更披露(强制): 验证证据(PR head 判定:可合并 —— 代码、测试、CI 状态均已核实,无遗留小问题;合入 lane 分支的操作归 owner。 |
836dd34
into
2026-11-04_fastpath-v6-msgid
…sync first-scene journal + evidence-addressed apply) Post-#5721 supplement on the frozen ordinal-deviation line: the row table is now the owner-ruling LSM snapshot itself — a compression fold lands as a DIFF written straight into it, never as a snapshot replacement: * FIRST SCENE (synchronous, 压缩请求内同步落表): recordServerSideFold runs from the compress TOOL endpoint the moment a fold lands — ONE transaction journals the covered absolute positions + the live block signature into the checkpoint meta; row bytes stay untouched (plugin-lane folds usually change no wire bytes). Idempotent per signature; fail-soft per sid — the tool response never depends on this plane. * RECONCILE (下一轮只做核对): the journal red-gates exactly ONE forced full pass ('structural-pending'); its capture classifies every cell against the kernel's OWN output (block effectiveMessageIds ↔ pf_ledger.core_id ↔ absolute pos): fold-covered cells upserted in one tx ('apply', prefix bytes + checksums untouched); count-changing witnessed layout churn rides the COARSE rebase tier (contiguous reindex = documented V1 boundary); ANY other changed/lost cell kills the plane with its byte-for-byte pre-existing verdict. * Gate vocabulary gains 'structural-pending'; the rows/count strict equality becomes an abs-pos span check (ascending distinct, holes do not red-gate); hydrate accepts the sparse shape defensively; debug state exposes pendingDeletes. Store layer unchanged (existing pfRenderCommit covers arbitrary-pos upserts + S4 checkpoints).
…ed cap, render-snapshot plane Upstream 2919794..ce8f92e (57 commits) lands three large planes: - ranxianglei#5652 fastpath-v6 identity lane (c295f90 storage layer, d55a50c composite identity, 9c1fb86 full-list wire, da6885d fixed tail window, plus the review follow-ups) - ranxianglei#5754 image-shed 64-item cap (6aed3b8) - ranxianglei#5689 render-snapshot plane (acca4db schema v12, 6d3ca30 O(1) steady turns, 49ba14d seal-shut probe) - ranxianglei#5721 witnessed rebase supplement Overlap with this lane was four files. src/plugin.ts auto-merged (upstream touches the handoff-reply import at :31 and handlePluginTool at :2337; this lane owns the strip entry points at :22 / :61 / :2640..:4554). The two CONFIGURATION files are generated and were regenerated from the seed. One real conflict, in tests/issue4666-migration-dedup.test.ts line 158: upstream re-swept the sqlite user_version pin to the literal 13, this lane had already derived it. Kept the derived form -- that is the whole point of the earlier sweep, and the literal is what goes stale on the next bump. That conflict resolution exposed a latent bug of mine. 3eb22b1 ("pin the mysql lock lane to the CURRENT schema version") derived the mysql stamp assertions from STORAGE_SCHEMA_VERSION as well. When it was written both numbers were 9; ranxianglei#5652/ranxianglei#5689 have since pushed the sqlite unified store to 13 while the MySQL backend stayed at 11, because the v12 identity plane and the v13 render-snapshot plane are sqlite-only. The mysql-contract lane therefore asserted "13" against a migrator that correctly stamps "11". Fixed by exporting MYSQL_SCHEMA_VERSION from src/storage/mysql.ts and pointing the four mysql-side assertions at it; the sqlite assertions keep deriving from STORAGE_SCHEMA_VERSION. Verified String(MYSQL_SCHEMA_VERSION) is "11", i.e. byte-identical to the literal upstream asserts. Verification on the merge tree: issue5175 58/0, tag-echo 79/0, issue2190-degen-close 32/0, golden-storage-render 5/0, storage-engine-proofs 12/0, native-compact-terminal 1/0, plugin-passthrough-tag-strip-chat 20/0, issue4666-migration-dedup 2/0 (the three mysql cases skip without BILI_MYSQL_TEST_DSN; no local MySQL or docker socket in this sandbox). tsc --noEmit clean, all four gates exit 0, generated docs byte-identical.
(ai: sglang-gpu0/qwen3.8-27b)
Post-merge supplement to #5689 (merged into this lane at
f3681735f). That merge sealed the no-recast regime: any stored-row divergence kills the plane for the sid permanently (kind='dead', cross-process). Owner then ruled on #5643 「快照必须随压缩演进——事件化更新语义」: killing the plane on the first compress is a core-functionality loss, not a safety win — compression IS bili's core feature, so the row table must evolve with its owner (the kernel fold layer) instead of dying.What lands here (one commit, cherry-picked clean onto current lane tip)
blocks-active(any active block → lane off forever) is replaced by a structural witness —activeBlocksSignature(): sorted multiset of active block id+generation, pure server-side state, changes iff the layout changed. A signature move costs exactly ONE red round (new reasonblocks-changed) after which the next full-path capture re-synchronizes; folded sessions with stable layout keep serving fast.pfRenderCommit replaceAll), fresh S4-checksummed checkpoint carrying the new signature, audit tick reset. Storage stays bounded by live elements — compaction SHRINKS the table. Witness sources are server-side evidence only: block-signature move around this turn's kernel pass / dsh native-compaction detector fired ([bug] DSH 桌面端 0.1.188:宿主原生压缩落地摧毁折叠基底 → 同一区间永久不可锚定、compress 连片失败并熔断(compaction-basic auto:false 仍无效) #2432 folds bypass bili entirely) / fold usage consumed pre→post / gate itself observed the move.Tests
Pre-existing suite passes untouched (A/B byte-equality golden, S4 tamper pin, unwitnessed-kill pin both-directions). New lifecycle unit pin: cast → witnessed shrink+rewrite → steady appends renumbered onto the compacted table → restart hydrates rows AND signature; plus the both-direction gate pair (matching sig green incl. hydrated restart; stale sig →
blocks-changedred). Local: render suite 10/10, fastpath-v6-msgid + issue5403-watermark + lane matrix 114/114, typecheck clean. Full-suite verdict = CI lanes on this head.Closes nothing standalone; supplements #5689 per #5643 ruling. Config surface: none (rollout flag unchanged). Wire: none.