Repository navigation
[PR] storage: snapshot table compression foundation P0+P1 (schema v12 4-table + shadow write, all off by default, zero behavi | storage: 快照表压缩底座 P0+P1(schema v12 四表 + 影子写,默认全关、零行为变化) - #5901
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.10050-pr.5901.7c58bc8This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5901.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 接单。查重结论:不重复——开放 [PR] 列表里没有第二个快照表底座实现 PR;相关但有区别的两条:#5873(设计提案 PR,同主题的 design 阶段,本 PR 是其实现)与 #5855(fastpath re-land,与本 PR 争 v12/v13 schema 版本号,PR 正文已自行标出)。 标签:p3 + scope/storage(默认全关、零行为变化的存储层底座,当前无用户面影响;非缺陷类不打 bug)。 正在做完整 review(方向/回归/代码/测试有效性/diff 整洁度五关)+ 核对 CI 实时状态,结构化八问结论随后在本线程给出。PR 里三个待裁定点我会各自给独立判断供你拍板。 |
(ai: vllm-qwen/qwen3.8-27b) CI 自动修复 lane(attempt 1/3):原 head 修复内容(2 个独立标注 commit)
结构化八问
收工判定:新 head |
(ai: vllm-qwen/qwen3.8-27b) Review 结论(PR #5901,现 head 先报状态(均实时核过):
结构化八问:
三个待裁定点的我的判断:
另两点观察(非阻塞,留作放量前记录):
判定:方向正确、实现扎实、测试有牙。CI 在新 head 转绿后即可合入;但本 PR 触及持久化格式(schema v12)+ 配置面,按 §7.4 属人工合并门禁——请人工合并:#5901 |
0e0804a to
2cae3e4
Compare
(ai: vllm-qwen/qwen3.8-27b) CI-lane run (conflict resolution, attempt 1/3): the PR's conflict with master is resolved. Merged What I found and did
结构化八问
收工判定:冲突已解,PR 现为可合并态(待 CI 重跑转绿 + owner review/合并)。 |
…ries The design doc pair (proposals/snapshot-compression-foundation) moves from the proposal branch into the implementing PR, per the docs-centralization standard: - bilingual README.md / README.zh-CN.md + 4 arch diagram pairs (svg+png) + the arch-migration generator script - status line updated: P0+P1 implemented in #5901 (schema v12, shadow-write soak, zero egress change); P2 egress cutover still pending owner ruling - §5 version-stacking corrected to reality (landed v12 on v11 master while #5855 is still open; later-landing side renumbers mechanically) - §9 keys finalized (storage.snapBase.enabled / .shadowWrite, config-file only) - §12/§13 notes: P0+P1 shipped as one PR; #5855 prerequisite not waited on - cross-links to the per-table map docs/storage-relations.{en,zh}.svg - registered in docs/index.md + docs/index.zh-CN.md
…th (#5874) docs-sync red on fb17874: the committed generated rows claimed default 'on (soak; off in release)' while the canonical seeds (website/config-reference/process.yaml) say default false — the generated artifact had drifted from its source. Rerun tools/gen-config-docs.mjs generate and commit its output (2 rows per file, en+zh). Note for reviewers: the #5901 soak (dab9292) makes the code default ON when the config file is silent until the owner-instructed pre-merge revert; this doc describes the seed state, which equals the post-revert code.
|
Field incident + fix (2026-11-08, commit The #5855 collision flagged in the PR description happened in a real shared environment today: a database co-written by instances running the #5855 re-land code had been migrated to Fix: the v12 DDL is probe-first now (table-existence check, the discipline v2/v5/v6/v8 already use) instead of version-gated — the DDL is pure Verification: 14/14 suite + 35 storage regressions green, tsc clean; redeployed the shared 2.1 GB database live — tables now coexist with |
…CI)
readPkgVersion is polled every 500ms by waitFor in the registry/advisory e2e suites; a poll landing inside the updater's remove-and-reinstall window hits a transiently absent package.json and the raw readFile throws ENOENT out of the predicate, killing the subtest at ~1.6s despite a 60s budget (ubuntu red job 128697545236; windows green on identical code = race). All four copies of the helper now return null on ENOENT ('not yet', not an error); one-shot assert sites still fail loudly on absence.
[bot] 🏷 This run — two red CI lanes fixed on PR #5901 (new head
|
(ai: vllm-qwen/qwen3.8-27b) Re-verified the live check state first — the reported Live state at current head
The fix landed as 七问
判定:本轮无需进一步动作,等最后一个测试分片(windows 22 shard 1)自行收尾即可。 |
(ai: vllm-qwen/qwen3.8-27b) [bot] 🏷 Already posted above — this run's full summary (live-state verification + 七问) is my previous comment in this thread; nothing new to add. Outcome in one line: the reported |
(ai: vllm-qwen/qwen3.8-27b) CI run summary (head Re-verified live first: by the time this wake ran, every gate previously reported red had been resolved by your own commits, so I pushed nothing. What happened across this session's wakes:
The one remaining red (wake 4): Actions taken: zero branch changes (nothing mechanical to fix); one-line conclusion posted to #4599 per CI-coordination policy; targeted Carried-forward disclosure (unchanged since floor 18): the temporary SOAK fallback at src/storage/store.ts:648 (config silent ⇒ plane ON) is marked "revert before merge per owner instruction" — until it's reverted, code-on-silent ≠ doc default 七问
收工判定:head |
(ai: vllm-qwen/qwen3.8-27b) Base conflict resolved — Live re-verify first: master had advanced to Conflicts: exactly 3 files, all one issue — Local verification before push: the lock-sync gate's own recipe ( Push wrinkle, handled per discipline: your Post-push live state: 七问
收工判定:冲突已清( |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) CI auto-fix lane (attempt 1/3) — all 4 red test shards on PR #5901 fixed in one commit, pushed as head Live re-verify first: the mirror cache was frozen at a 14:55Z snapshot (stale), so I took one direct check-runs read — the event's 4 failures were still live at head Attribution (from the job logs): all four shards fail on the SAME single deterministic test — Root cause: your P2 commit
A cut landing between the two halves of a surrogate pair serializes as an unpaired Fix (commit Local verification before push: Push wrinkle, handled per discipline: your Carried-forward disclosure (unchanged): the temporary SOAK fallbacks — 七问
收工判定:根因已修并推送(新 head |
验证汇总:真实模型 soak + 自评审修复 + 100% CI + 100% 本地 gate1. 真实任务 soak(qwen3.8-27b,dsh 车道,tmux 三任务)用本 PR 的构建起了 3 个真实任务:评审本 PR、写双语 README 高管摘要、追踪入站→出站管线。三个任务全部完成、产出质量合格(评审抓到了真 bug,见下)。压缩轮全部正常触发(累计 5 个 block,~11 万 token),出站 egress=cutover 36 次、red 0 次。 2. soak 评审发现的问题(全部核实、全部已修)
3. CI
4. 本地真模型 gate(release-gate-local,--with-update)单次运行 100% 绿:A 9/9(折叠+折叠后连续性,含中途窗口收缩)B 6/6(中继 glm-5.2 大上下文)U 2/2(dsh 托管安装+私有 registry 更新)。 出站证据:bili.log 中
|
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 已做 live 复核,与你的汇总一致:
|
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Merge-conflict lane (attempt 1/3) — all conflicts resolved, merge commit Live re-verify first: the mirror check cache snapshot was stale (frozen Nov-8 18:55Z, known freeze), so I took git ground truth instead: branch What conflicted and how each was resolved (merge of
Everything else (docs, Local verification before push: full Carried-forward disclosure (unchanged): the temporary SOAK fallbacks — 七问
收工判定:冲突已收敛并推送(merge commit |
Resolves the PR's dirty merge state (CI had stopped re-triggering on pushes). The one conflict — src/persist.ts afterLoad: master's #5608-D deferred the #4316 ledger alignment from cold load to the next fold; take the master side and keep the #5874 forensics log line there so every cold load stays observable with session-object identity.
…ps retired blocks Rebased head is red on master's own pin (3de75ff, tests/snapshot-render.test.ts): a retired block stays active=true (kernel sync.ts: 'the retired flag is the only switch'), and every other #5765 consumer suppresses its summary anchor — D-lane snapshot-render.ts 'if (retired) continue', kernel collectSummaryAnchors prune.ts — but the plane egress anchor loop only checked b.active, so a direct-authority egress re-rendered the retired block's anchor (its covered rows stayed suppressed, the anchor came back). The loop now skips b.retired exactly like the two kernel/D-lane sites; knownSummaryIds already covers all blocks so stale rendered copies are still dropped. snapshot-render 13/13, snaplane p0+p1+p2+fastpath 17/17, snapshot-render-chain, storage-engine-proofs 12/12, opencode-rolling 10/10 all green.
… back to master state f25f772 exported TRUNCATION_CLIP_RE + buildTruncationClipMarker 'so hosts can recognize their own legacy bytes as kernel clips (the egress tripwire adopts them)'. 533b72f (direct-authority ruling) deleted the tripwire: the plane now stores the kernel's own tagged processTurn output, so the clip marker arrives already in the served bytes and no host ever needs to recognize it. Grep-verified zero consumers outside kernel/ (src, tests, pi-subagents); invariant-judge.ts keeps its own independent looser regex (#3842). Reverting the four kernel files + both lockfiles to master state makes the kernel diff of this PR exactly zero — no kernel-change gate, no human-gated kernel surface left in the diff.
…nel-input assembly, delta outbound (#6181) Implements the 2×2 brief (discussion 5874 comment 19544722) on top of the P2 plane, kernel unchanged. The snapshot table (l1_world + row rows) stays the single truth; the two selection points are independent: - Inbound (stage 0, prepare-openai lane): the client's (sess_raw, msgid[]) pair is matched against the stored world canon (matchWorld: echo/probe/scatter/issue/diverge/resync). Probe rounds assemble the kernel input from plane rows + only the NEW span, so M1 (full resend) and M2 (incremental) reach byte-identical kernel cores. R3 identity faults (declared msgids vs element count) 400 with zero side effects, 3-strike window latch degrades to the legacy path. - Outbound (egress stage, snap-egress + prepare-responses): the snapshot view carries a per-element side channel; the responses lane (provider retains context via previous_response_id) selects seed/delta/no-op via selectOutbound — first round full (incl. anchors), next rounds only new rows, isFullResend trips a reseed. Degrade to legacy resets the delta frontier (one-round convergence, #7). Files: src/snapshot/{assemble,checksums,delta,identity,plane,types}.ts (new), store.ts snapStage0/snapStage0Cores/snapStage0Reset facades, snap.ts l1 envelope + clearSnapPlaneDead, msgid-pin.ts seqs channel, session.ts responsesDelta persistence, architecture.md map. Tests: tests/issue5874-2x2-p3.test.ts (33: verdicts ±, M1/M2 byte pin, T5 scatter pin, #7 resync, #8 delta pin, R3 red gates, store round trip). Regression: p1/p2/fastpath 23/23, snapshot-render 24/24, storage-engine-proofs 12/12, opencode-rolling 10/10. tsc clean, unused-export gate clean, architecture-docs guard green.
…sponse_id only after this proxy established a delta frontier CI run 43079490144 red: 'ACP_KEEP_RESPONSE_ID=1 keeps the id forwarded and stays silent' — the Part B strip condition (!deltaServed) removed the id on EVERY non-delta round with the knob on, including the native case (no snapshot frontier ever established) where the pinned #1954 contract says the id is forwarded intact. Fix: capture hadFrontier (a persisted delta state existed before this round's selection). The id is now stripped only when hadFrontier && the round served no partial delta — i.e. the provider's retained context equals exactly the sent prefix, so a full replay would duplicate it. Before the first frontier the native semantics hold regardless of the outbound (seed / full view / legacy). deltaServed tightened to mean 'a PARTIAL delta went out': a reseed and a full-resend (delta grew to the whole view) both send the full view and now read as no-delta. onSelection carries the total element count so the lane can tell them apart. The dev-message gate (skip on partial delta) keeps its meaning.
…nel, frontier commit-on-accept, pure continuation, shared stage-0, codec-level pin) Review of PR #6299 found four blockers and six majors; this is the complete fix round. B0 (root): a chained turn on the responses lane has no full-view reconstruction (the D lane is skipped for pluginMode and #4804 augmentation is WS-only), so its kernel view is a delta, not the world. Flushing that view to the plane deleted every row it did not contain. Now a chained turn that is neither cohort (msgid-pinned stage-0) nor WS-anchored sets session.skipPlaneFlush (one-shot, consumed by the persist build — snapShadowSync returns before any plane work) and passes bypass:true to applySnapEgress (the outbound is the legacy bytes; the plane stays stale-but-intact until the next full/cohort turn, which re-syncs it one-round). B1: FoundationRejectError was swallowed by the lane 'kernel transform failed, forwarding unchanged' catch -> 200. Both lane catches now rethrow it first; the server funnel maps it to a 400 with x-bili-snapshot: identity and {error:{type:'snapshot_identity_fault'}} before the generic 503 path. B3: a plugin chained round with EMPTY input (a pure provider-chain continuation) is routed to the raw-body passthrough (prepareResponsesCompact) at the dispatcher — no dev-message injection, no chain-id strip, no kernel round. The injected dev message + stripped id on a no-content round produced a blind provider response that the host then chained from (permanent context loss). B4: the delta frontier advanced at egress-build time with no rollback on upstream failure. The selection now stashes pendingResponsesDelta on the Prepared and commits session.responsesDelta only in the relay when the upstream response status is < 400 (4xx -> null = reseed next round; never advances on failure). M1: stage-0 arming now requires the pluginLaneServerHistory rollout (featureEnabled in runStage0), tying R3 to the rollout that makes it reachable. M2: checkR3Identity is now the single R3 arbiter, called in runStage0. M3: the R3 breaker latch persists (l1_world.r3_latched_at, probe-first ALTER, upsert — the first fault precedes the session's first l1 row) and survives a process restart; a latched session degrades forever, never 400s again. M4: A1d pins the kernel-core byte-identity at the CODEC level (the new element's core is byte-identical between the M1 full-array parse and the M2 standalone parse, wireIndex aside). M6: session.responsesDelta is cleared at every stage-0 resync (s0.resynced). M5: PR body rewritten to match the code. Shared stage-0: the pin + R3 + stage-0 decision + assembly moved out of the openai lane into src/snapshot/stage0.ts (runStage0), wired into BOTH the openai and responses lanes (the responses lane reads biliMsgIds/biliLedgerSeqs from its body; cohort = pluginMode + declared). Tests: tests/issue5874-2x2-p3.test.ts 40/40 (was 33): new A1d (M4 codec pin), B0a (skipPlaneFlush leaves the plane untouched), B0b (egress bypass ref-identity), M3 (latch persistence across store instances), and HTTP e2e B1 (400 + header + zero upstream forwards), B3 (pure continuation raw passthrough), B4 (400 turn does not commit the frontier; the next round re-seeds the full view). Regression: snaplane-p2+snapshot-render+chain+shadow 41/41, storage-engine-proofs +opencode-rolling 22/22, snapbase-p0+snaplane-p1+fastpath+ responses-previous-response-id 23/23, tsc clean.
…-body M2 retries no longer wipe the plane, and the first chained round seeds its frontier at the stored-world boundary (gated on a trusted, never-refused chain) - Residual 1a: an empty delta (frontier already covers the world — the host retried the same body after an upstream failure) is now classified as a NOOP: the chain id is kept, no developer message is injected, and the empty input rides the provider's own cache. Before: id stripped + dev injected = a blind response the host would chain from forever. - Residual 1b: stage 0 recognizes a full replay (every incoming id already in the stored canon, carrier seqs at/below the plane frontier) as echo+replay — zero side effects. Before: an M2 retry ([m5] vs head m1) read as headMoved -> diverge -> snapStage0Reset: a harmless retry wiped the plane. - Residual 2: the first CHAINED round of a keepResponseId chain carried no committed frontier (round 1 had no id), so it re-seeded the whole world the provider's chain already holds from this proxy's own round-1 full view (2x the world's tokens, one time). Stage 0 now proves the safe case (probe + nothing gone + new elements -> the chain provably holds the stored world) and seeds the frontier at the stored-world boundary; everything else keeps the plain re-seed. - B4 hardening: a REFUSED selection round now sets a persisted responsesChainBroken marker (committed with the session). While set, the frontier seed is blocked and the round re-seeds the full view — the in-request flush wrote the refused turn's rows before the forward, so the plane holds more than the provider's chain and trusting storedFrontierPos would lose the refused content. The flag clears on a full-view/noop acceptance; the pending delta now carries its kind (full/noop/partial) so the relay can maintain it. - Tests: p3 42/42 (new: residual-2 unit shape pin, residual 1a+2 e2e through the proxy, A1/A1d legs now use isolated DBs — the two legs are ALTERNATIVE client behaviors on the same turn and sharing a file let leg M1's canon contaminate leg M2's verdict). - Regression: p0 10, p1 4, p2 17, fastpath 2, snapshot-render 13+6, snapshot-shadow 5, storage-engine-proofs 12, opencode-rolling 10, responses-previous-response-id 7 — all green.
…a delta frontier only for an id this proxy minted (#6181) previous_response_id proves nothing about which world the provider's chain holds. A host forking/rewinding to an older response id (or carrying an id another proxy minted) can pass the strict-extension proof yet lose the gap between the old chain's world and the stored world when we seed at the stored boundary. - session.lastMintedResponse {id, frontierPos}: the latest response id THIS proxy emitted for the session, paired with the plane frontier the chain provably held — recorded in the relay from the id observed on the outbound stream (SSE response.created/completed scan on the exact bytes written to the client, json.id for non-streaming), accepted forwards only (a refused round mints nothing the client chains from). - store facade snapPlaneMaxPos (store/shadow/persist) — the lane records the post-flush plane max pos as prepared.mintFrontierPos. - seed gate: the first-chained-round frontier seed now also requires lastMintedResponse.id === previous_response_id AND lastMintedResponse.frontierPos === storedFrontierPos. - persisted-frontier trust: a chained round whose id this proxy did not mint last drops the persisted frontier and re-seeds the full view (id kept, dev injected — the pre-seed status quo, at-least-once, never a loss); the persisted state is cleared so the next round re-establishes verifiability from a fresh seed. - B3 latent bug: the pure-chain-continuation route reuses the compact-passthrough base, which hard-codes resetAfterSuccess — scheduling a native-compaction rebase (resetSessionCompression) for the next turn although nothing compacted. Dropped for the B3 route. - p3: per-round upstream ids in the e2e harness; the Residual 1a+2 e2e chains from the ids the responses actually carried and gains a round-4 fork case (old id + full current world + new element → full re-seed, id kept). 41/41.
…ate) The Residual-2 hardening commit added a raw mintSseTail.slice(-32768) which the text-safe family gate (tests/text-safe.test.ts) rejects — raw negative string slices reintroduce the #816/#1615 lone-surrogate family. Replace with safeSuffix(mintSseTail, 32768) (same last-32768-code-units semantics, surrogate-pair-safe). safeSuffix was already imported.
… rule) + mint tap spin Two regressions surfaced when the branch was re-based onto the #5901 head (f000197, which carries #6326 durability-before-render): 1. B4 e2e: #6326's ingestRowsNow persists the ledger rows BEFORE render as an UNTAGGED save; its final-state diff then deleted/re-homed the pinned plane rows (ids the tagged plane cannot address — R2 doctrine) and the next tagged save marked the plane sticky-dead. snapShadowSync now guards untagged saves against a sid that holds active plane rows with a PREFIX RULE: the save is allowed only when its ledger-id view prefix-identically extends the stage-0 l1 envelope (that save is the one that materializes the rows stage-0 declared — ids agree, idempotent); a stale view (re-derived kernel ids, pre-turn transcript, compaction) is skipped for the plane, the disk record still lands, and the plane converges at the next tagged save or stage-0 resync. The guard runs BEFORE the transaction (after it the destructive diff is already committed). 2. Residual-2 e2e: the mint-id scan (the record that lets the next chained round seed the delta frontier at the stored-world boundary) existed only in the compress-loop (injected) branch; plugin and non-injected responses turns never entered it, so the first chained round re-seeded the whole world. The scan is factored into MintIdTap and rides a tee'd twin of the plugin-lane stream and of the plain (non-rewrite) proxy stream — both branches where the client-facing bytes ARE the upstream bytes. The tap fix also repairs a latent infinite-spin in the old inline scan: the event-boundary recompute ran only at the loop bottom, which the non-marker skipped — a non-marker event preceding a marker event pinned the boundary on a shrinking tail and spun the loop forever (event loop blocked; no timers fired). The recompute is now unconditional on every pass. p3 42/42 (incl. B4 frontier-commit-on-accept + Residual 1a+2); p0 10/10, p1 4/4, fastpath 2/2, p2 17/17, responses-previous-response-id 7/7, snapshot-render 13/13, render-chain 6/6, shadow 5/5, storage-engine-proofs 12/12 (kernel rebuild for #5901's compress.ts), opencode-rolling 10/10.
…wline in the #5901 merge The 941447a merge joined the egress-cohort desc_ja value with the next '- key: sessions' entry onto one line, which tools/gen-config-docs.mjs rejects (quoted value not valid JSON) — docs-sync + docs-coverage red. Split the line back; generator checks pass (coverage ok: 204 env + 63 keys; i18n ok: 254 keys).
… v11 test title, comment indent - pr-body-6299.md: #6299's PR-body draft (already merged) was committed at the repo root — authoring artifact, not repo content (violates the docs layout: root keeps README*/AGENTS/CHANGELOG only) - tests/storage-ingest-watermark.test.ts:594: title said 'migrates to v11' while the body has asserted v12 since this PR bumped STORAGE_SCHEMA_VERSION - src/server/prepare-openai.ts:204: comment block started at column 0
…6480's pure-sqlite default flipped the absent-selection JSON writer off, and these suites assert the legacy JSON persistence surface (session files / cold-disk-restore content-store reads). First exposure: the ci:full gate ran them on the #6480-merged tree; reproduced on bare master, fixed by the explicit dual-write pin. #6480
… the CI storage pin in notice-registry (master #6480 regressions) Two master-side regressions first exposed by this branch's full test shards: 1. fork-ccr-adoption reads the companion session ENVELOPE off disk, but the #6480 pure-sqlite default stops writing JSON envelopes. Pin dual-write in the test (same one-liner as the native e2e suites). 2. The notice-registry suite is hermetic by design (absent config file, fresh XDG) but the CI test baseline pins BILI_STORAGE_UNIFIED=legacy (preload-storage-baseline.mjs, #4011). An explicit selection is a user decision that legitimately hides the fresh-install default-flip notice, so the two 'fresh install sees the default flip' assertions failed on every shard. Lift the pin for this file only. Master merged #6480 without a main-CI run (the push event was dropped), so neither regression had been seen in CI before this branch.
… pin (Windows EBUSY/EPERM) persist-metadata-fold-gate.test.ts to keep the legacy envelopes their assertions read. unified=full opens index.db, and Windows cannot unlink a file whose SQLite handle is still open — both suites rm'd their temp dirs without closing the stores, so the windows-latest shards failed deterministically (EBUSY unlink index.db / EPERM rmSync tmpdir) while every ubuntu shard passed. Close the stores (closeQuietly) before rmrf. Same master-side regression class as the notice/pin fixes in 7b6eb98: master merged #6480 without a main-CI run, so none of these had been seen.
…t-off revert, test coverage, docs - B0: on non-cohort chained bypass rounds the in-request flush was the only consumer guarded by the one-shot skipPlaneFlush; the relay success path re-arms the debounced save and that save rode turn.messages' DELTA view as a tagged world — snapViewEvents (final-state diff) deleted every plane row the delta lacks. The lane now clears session.lastPlaneMessages on bypass rounds, so later saves derive untagged from the DB final state (append-only rows — correct). - Default off (discussion #5874 section 9): the 2026-11-08 100% soak default is reverted in all four spots — store.ts field + open() fallback, admin.ts status payload, snap-egress.ts gate (egress now opt-in, v?.egress === true) — and the p0 flag test now pins default-off. e2e suites arm the flags explicitly via a dedicated config file. - Test gaps from the review: T2 covers the in-transaction rollback leg (valid append + R2-unknown update in one batch => nothing lands); T9 raises N 40->160 so SNAP_FOLD_DEPTH (128) actually fires and asserts a fold happened (ckpt_seq >= 1) — at 40 saves the bound was vacuous; the byte-identity oracle's LEGACY_TABLES now includes kv_store (was 11 of 12); p2 G2-G5 set egress: true explicitly (they had relied on the soak default). - Docs: process.yaml egress default false + 3-lang regeneration; proposal READMEs (EN/zh) status line, DDL block (actual v12 columns incl. envelope / r3_latched_at), probe-first migration note, stage-5 row, section 9 key list, section 10 metrics all updated to the direct-authority + default-off state; config.ts schema comment; stale comments fixed (session.ts lastPlaneMessages, handle.ts B3 string-input admission, snap.ts l1 upsert, p0 T6).
The #5901 rebase auto-merge kept BOTH ../persist.js imports: master's standalone 'import { getStore }' (added for the #3397 ingestRowsNow durability-before-render call) and this branch's 'import { SessionStore, getStore }' (added for the in-request SessionStore.flushOrSnap plane path). Both are needed symbols (getStore at the ingestRowsNow + budget calls, SessionStore at the flushOrSkip call), so the superset import stays and the redundant standalone getStore line is dropped. Fixes TS2300 duplicate identifier surfaced by the post-rebase typecheck.
d30da8b to
f42f251
Compare
…seeds (docs-sync guard) The #5901 rebase resolution of website/config-reference/process.yaml dropped the desc_ja lines for storage.snapBase.enabled and storage.snapBase.shadowWrite. Master's gen-config-docs.mjs now requires desc_ja on every seed entry, so the docs-sync CI job fails at the 'generate' step (exit 1) before the drift check. Restored both lines verbatim from the pre-rebase PR head; generate now exits 0 with no drift and docs-coverage stays green.
…conflict) Union of both intents: - #5901 (HEAD): snapshot-plane count columns (snapRows/snapLogEntries/snapCkpts/ l1Entries) + the #5874 snap_rows inline-footprint term. - master: sessions/conversations count columns + CAST AS BLOB on every LENGTH() (accurate byte count for multi-byte content). Resolved: 15 sessionId params (15 SQL columns), full result type, CAST AS BLOB applied to the session_ext term for consistency with the auto-merged sessions/blocks terms; the #5874 snap_rows term is kept verbatim.
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决并推送(2026-11-15) master 在 owner 上次合并后又前进了(到 做了什么
验证
八问
合并判定:冲突已解决、已推送( |
这是啥 / 做什么
讨论 #5874「快照表式压缩底座」的干净实现,一次 PR 做到 P0 + P1 两个阶段,全程默认关闭、不改变任何现有行为:
l1_world身份账本 /snap_rows快照行 /snap_log事件日志 /snap_ckpt折叠点)+ 一个索引,外加单事务提交原语(事件按序落行 → 逐条记日志并写 S4 校验和 → 可选折叠进 checkpoint,任何一步出错整个事务回滚,不留半态)。读侧重建会重算校验和,对不上就永久关死该会话的平面(R1)。回滚 = 关旗(两表自动空转)或 DROP 四张表 + user_version 退回,没有任何数据迁移。
关键设计点
src/storage/snap.ts):单事务;seq 严格单调且恒 > max(log 最大 seq, ckpt_seq);折叠时写三件套 meta(内核 kstate 原文 + 块覆盖映射 + S4 聚合),日志在折叠点截断,深度有界。active=0,pre/post 双形态保留;物理清理只在该行的覆盖块消失后发生(GC 释放阀)。storage.snapBase,只走 config-file,不新增 env 旋钮,彻底整治环境变了滥用,除非必要都走配置文件 #2030 纪律):enabled(master)⊃shadowWrite(lane),任一层在 knobs 和 store 两处都强制嵌套,默认全关。/__bili/status加snaplane字段(旗状态 + 各表计数 + 按 sid 的死锁列表),sessionStorageStats加平面维度。测试
tests/issue5874-snapbase-p0.test.ts(9 个):T1 v11→v12 纯加性升级(11 张 legacy 表 dump 逐字节不变)+ DROP 回滚复原;T1b 新装直达 v12;T2 原子性/无半态;T2b 跨折叠 seq 单调;T2c S4 篡改 ⇒ R1 粘性死锁;T2d delete/compact 不物理删;T6 级联无孤儿;flag 嵌套两层各 5/4 例。tests/issue5874-snaplane-p1.test.ts(4 个):T3 B1 陷阱钉(同一 wire 重放 ×3 次重启,l1 只发一次、一轮武装、重启稳定);T4 A/B 影子比对(每次保存后平面活跃行 == legacy live 行,逐 pos 逐字节;含 crash gap 自愈);T7 块覆盖区间落进 ckpt 覆盖映射([["blk-c1", 0, 7]])且被盖行双形态保留;T9 40 轮 B1 重放下日志深度 ≤256、行数 O(pos)。tsc --noEmit干净,check:unused-exports干净。config surface
新增
storage.snapBase两个键(config-file only,不新增 env,#2030 纪律):storage.snapBase.enabledfalsestorage.snapBase.shadowWritefalsestorage.snapBase.egressfalsetrue才生效storage.snapBase.egressCohortsproviders表管上游接入、compress.*三级层次管压缩引擎参数——快照表平面是存储层新机制,两处都没有可挂载的键位;按纪律走 knobs 的 config-file key(env > file > default),不加 env。需要 owner 裁定的点(先按我的默认做,讨论后改)
Refs discussion #5874.
2026-11-11 update — P2 (mapEgress) landed + P2-final in-request snapshot write
P2 (earlier in this PR): the outbound compressed context is now served from the S4-verified snap plane instead of the in-memory history. Owner ruling (2026-11-11): the plane is the DIRECT outbound authority — no per-round diff, no tripwire, no seam-walk re-alignment; the only degrade paths are the defined no-plane ones (no plane → legacy bytes, assembly error → legacy for one round, degrade-reason counters on
/__bili/status). Opt-in viastorage.snapBase.egress: true(+egressCohortsprotocol list) — the 2026-11-08 100% soak ran it on and was reverted to default-OFF on 2026-11-13; flipping egress off is a zero-byte rollback.P2-final (this push): the six-stage chain now runs in the proposal's own order — raw bytes → protocol conversion → kernel rewrite → snapshot write (in-request) → egress rewrite → egress. Previously the plane was fed by the 500ms debounce AFTER the response, so the egress stage always read one save behind and had to splice this round's own resend as a client tail. Each kernel lane now flushes the store synchronously right after
processTurn(snapshotMessagesfirst, thenSessionStore.flushOrSkip); a failed flush degrades the round to the legacy egress path (tripwire-safe), and the debounce is re-armed only if the session mutated after the flush. A compression turn's brand-new block is plane-readable in the same round and this round's own resend carries an empty tail. The watermark window pin (wpHit.windowCores) moved with the write, expression unchanged; nothing between the two positions readssession.lastMessages, so pin semantics are byte-identical.Docs updated: configuration en/zh (regenerated from the process.yaml seed), proposal README en/zh status lines.
kernel-change: kernel/src/relocate-compress.ts + parse-compress-input.ts + tests — relocated fold summaries ride the #335 live-carrier protection; parse classifies a re-sent fold-point pointer (already-folded-echo) and salvages a stringified single-entry content (kernel version follows master, workspace path unchanged).
2026-11-11 update — master merge (#5608-C/D) parity fix
Master's #5608-C/D (resume-timestamp-ledger-carry) was merged in: a fold's ledger alignment raises the timestamp visibility high-water mark, after which the kernel's render intentionally drops the
timeattribute from old refs (prefix-cache economics). Two follow-ups:<acp>render tags. # Cache hit collapses to 1–12% for a whole generation, then the very next request returns to 100% — repeated 15× in one day #5608-C changes those tag bytes exactly at the fold, which reset the queue and replayed thecompressdirective (the "compress called exactly once" assertion). The queue now keys on the tag-stripped content — metadata-churn-proof.2026-11-11 update — image-plane fix (fafca54)
The plane rows persist the media-offloaded form (
imageBase64 = {"$media": hash}, #3762), and every row accessor hydrates it on read — the newsnapEgressViewwas the one accessor that returned the raw split form. Any image in the history fed that offload object intosha256inside the seam walk / tripwire, the TypeError escaped into the lane catch, and the lane forwarded the unprocessed client bytes (kernel render, watermark and tripwire all skipped). Symptom in CI: the full-resend arm lost its<acp>render tags while the incremental arm kept them —rolling cache_control: image payloads stay exact when marks movered, deterministic.store.snapEgressViewnow hydratespostJsonthrough the samehydrateMediathe other accessors use (CAS bytes, byte-identical).applySnapEgressnow enforces the never-fail contract: any assembly error degrades (counter + warn + legacy bytes) and can no longer throw into the lane.identitySigis robust to the offloaded form (the$mediahash is the byte identity — both forms sign identically) and cannot throw.Verified: the previously red suite green 3/3 locally (CI preload chain), P0/P1/P2 + snapshot-render 45/45, media/ingest regression 18/18, full CI green on fafca54.