Repository navigation
[PR] Snapshot-table compression foundation proposal (design, refs #5643) - #5873
ranxianglei wants to merge 6 commits into
Conversation
Design proposal per owner re-positioning on #5643: six-stage fixed chain (raw bytes -> protocol conversion -> rewrite -> snapshot tables -> egress mapping -> outbound) with ONE snapshot table set as the storage truth. Kernel zero-change. Includes zh+en design docs, current/target architecture diagrams (svg+png), DDL skeleton, red-gate/degradation matrix, test matrix, P0-P3 phasing, and pending owner rulings table (A-L + D3-D5). Docs only: no code, no config surface changes (config keys intentionally left un-named pending P0 config-surface section).
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8512-pr.5873.0e06357This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5873.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…t lands (refs #5643) Per owner request: the current-architecture picture viewed under the new foundation — ten components mapped one-to-one to their disposition (keep/reuse, role-change, replace/converge, yield-pending-ruling), plus the three structural pain points mapped to their fixing anchors.
(ai: vllm-qwen/qwen3.8-27b) PR #5873 审核结论(设计提案,docs-only) 先给判定:这是一份方向正确、对现状描述准确、结构完整的设计提案,零行为变更,可作为讨论稿合并。逐项如下(八问体例)。 ① 是什么问题 / 要做什么:把今天三处并行状态源(进程内存 lastMessages + kstate JSON + SQLite 表族)收敛成一套快照表作为唯一存储真源;每个模型请求走一条固定六段链(原始字节→协议转换通用核→改写/压缩→应用快照表→出站映射→出站);压缩内核零改动;有 session+msgid 的宿主全走新链,无 msgid 降级到「顺序>hash」写同一套表;任何失步整轮回退 legacy(降级不猜)。回滚 = DROP 表 + 关 flag。 ② 什么原因导致的(现状根因):三个结构性根因点得准——(1) 状态存三处形成双写分叉(#5403 OOM 的直接生产者就是 lastMessages 整包钉住);(2) 宿主 (sess_raw,msgid) 入站即剥、只在单轮在途、落不了盘,跨进程/重启无法回答「见过这个元素吗」;(3) v5/增量/D-lane 三条快路径各自补补丁、没有共同存储底座。这是根因层判断而非症状修补。方向成立。 ③ 影响范围与严重程度:本 PR 零行为变更(纯文档,src//kernel//tests/ 零改动),对用户面影响 = 无。它只是把 #5643 重定位后的方向落成一份可评审的设计契约。真正的行为面在后续 P0+(P0 明确「零行为变更」,P1 shadow 只写不读)。 ④ 怎么修复的(方案思路):单一提交原语(单事务 upsert snap_rows → append snap_log → compact snap_ckpt)作为唯一写入口;ckpt.meta_json 三元组承载内核态序列化 + 覆盖图 + S4 校验和;红门 R1–R4 任一命中整轮回落 legacy;P0→P1→P2→P3 分期、每期独立 PR + 人审关卡。 ⑤ 是否有回归问题:无。docs-only,不存在可被破坏的既有行为;我额外确认了没有任何测试引用 proposals/ 或扫描仓库文件树,连测试套件都不受影响。提案对现状的描述我逐条核对了 origin/master,全部准确(见下方核对清单)。 ⑥ 是否破坏缓存:不涉及——本 PR 不改任何运行时代码,缓存路径零触碰。提案本身反而冲着「稳定前缀缓存」去(egress 前缀字节跨轮稳定是结构性保证,非巧合)。 ⑦ 是否改了设计方向:不改现有方向,是新增一条底层线。它与 #5643 owner 2026-11-07 的重定位逐点对齐(六段链原文一致、内核零改动、msgid 全走 / 无 msgid 降级同表、#5855 只是工程基底非终态、三个待裁定项都收进了 §14 裁定表)。配置面纪律守住了:key 命名推迟到 P0 config-surface 章节、不新增 env knob、owner-gated、web 三态开关 + i18n + registry dead-key guard + yaml seed 全套到位。 ⑧ 是否引入新配置:本 PR 引入零配置(刻意留白到 P0),符合 owner-gated config surface 纪律,是对的。 准确性核对(我对 origin/master 实测,非照抄提案文案):
给 P0 / 裁定的几个补充角度(非阻塞,建议并入 §14 裁定与 P0 契约吸收):
CI 状态(读镜像缓存,未用 gh 轮询):static-gates 全绿;test 分片在跑,最近一次快照无 failure(11 skipped / 7 success / 12 in_progress);早期 windows-latest/22/1 那个孤红在本次快照已不在失败列表。本 PR 是 docs-only 且无任何测试引用新目录,test 分片结果与本 diff 无关。 交付完整性:已按本 PR 自身 diff 打上标签 收工判定:可合并(作为讨论稿发布)——零行为变更、方向正确、现状描述经实测准确、zh/en 完整、配置面纪律到位。真正的实现在 P0+,以 §14 裁定(至少 F/G/B 定表骨架)+ #5855 先落地为前提;上面 6 个角度建议并入对应裁定 / P0 契约,不构成发布阻塞。是否合并由 owner 拍板。 |
…5643) All markdown references (READMEs, PR description, discussion) already point at arch-migration.{svg,png}; the generated files were named arch-mapping.*, so the migration map 404'd everywhere.
79cb2f3 to
bdcbcfe
Compare
… split, unify four-table rollback wording (refs #5643)
What this is
A design proposal (docs only, no code) for the new snapshot-table compression foundation line that #5643 was re-positioned into on 2026-11-07: replace today's parallel state sources (process memory + kstate JSON + table spread) with ONE snapshot table set as the storage truth, with every model request walking one fixed chain:
raw bytes → protocol conversion (canonical cores) → rewrite/compression → apply to snapshot tables → egress mapping → outbound
The compression kernel stays untouched. Sessions whose host supplies session id + per-element msgids run fully on the new chain; hosts without msgids degrade to order-then-hash identity and write into the same tables. Any mismatch falls back to today's legacy full-resend path (degrade, never guess). Rollback = DROP the four new tables + flip the flag off = byte-identical return to today.
Contents (
docs/proposals/snapshot-compression-foundation/)README.zh-CN.md/README.md— full design, zh/en mirrors (§0–§16): stage-by-stage design, DDL skeleton (4 tables), invariants, red-gate degradation matrix (R1–R4), coexistence routing with existing lanes (v5 watermark / incremental plugin-turn / D-lane Snapshot rendering: proxy-lane outbound is now rendered authoritatively server-side; existing sessions are immune to ker | 快照渲染:proxy 车道出站从服务端正史渲染,存量会话对内核升级免疫(#3752 根治) #3756), rollout & config surface (owner-gated, no new env knobs), observability, test matrix T1–T12, P0–P3 phasing, and the pending owner rulings table (A–L + D3–D5)assets/arch-current.{svg,png}— current architecture (four lanes patching independently, kstate JSON as parallel source, three structural pain points)assets/arch-target.{svg,png}— target full view, pixel-aligned one-to-one with arch-current so the changed area and blast radius read at a glance (grey untouched · cyan new path/tables · amber role changes · red-dashed fallback; impact boxes on the right, one-line scope on the bottom strip)assets/arch-stages.{svg,png}— stage-by-stage expansion of the six-stage chain (commit primitive / DDL chips / red gates R1–R4 / coexistence routing & rollback)assets/arch-migration.{svg,png}— component migration map (every current component → its disposition under the new foundation: keep / replace / role-change / retired-pending-ruling; the three structural pain points mapped one-to-one to their fixes)Current architecture:
Target architecture:
Stage expansion (inside of the six-stage chain):
Component migration map (today's components → where they land under the new foundation):
Why now / sequencing
#5854 fully reverted the v6 line out of master; #5842 (identity wiring) is already merged. Timing dependency: this plan's P0 DDL stacks on top of #5855's transaction skeleton — #5855 should land first, then P0 starts. This proposal intentionally leaves config key names un-named until the P0 config-surface section, per the owner-gated config surface discipline.
Scope check
src/,kernel/,tests/