Skip to content

test: pin persistence write-amplification shape and mode cost ordering (#3049) - #3275

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-15_storage-write-amplification-pin
Oct 16, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-15_storage-write-amplification-pin

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Follows #3049 (production measurements: full 57MB snapshot rewrites for +270B deltas; 1.42TB cumulative writes over 15h on a shadow-mode host). Adds a hermetic regression test that pins the persistence write-cost model so the upcoming storage work (#2671 unified storage / incremental persistence) has a measurable target to flip.

What it pins

tests/storage-write-amplification.test.ts drives 16 small deltas through three storage legs in one process and asserts:

  1. Legacy shape — every save rewrites the whole envelope: total JSON bytes within [0.5, 1.5] × K × final-size, i.e. O(session size) per save, not O(delta). Measured on the folded golden corpus: 226,976B rewritten for 1,424B changed (~159× amplification). An incremental-persistence implementation MUST flip this assertion.
  2. Mode cost ordering (session padded to ~815KB by raising BILI_PERSIST_TAIL_TOKENS so the pads survive the tail-budget trim): composite bytes (envelope + SQLite peak footprint) satisfy full + jsonWriter=off < legacy < shadow. Measured: 5.79MB < 13.06MB < 18.85MB. Also pins that shadow pays exactly the legacy envelope cost (±5%) and that jsonWriter=off writes zero envelope files while still ingesting every delta into index.db.

No exact-byte pins (timestamps, WAL internals are flaky); the pins are orderings and bands that hold across engines.

Methodology notes

  • Envelope bytes: stat-sum of .json files under the session dir after each writeNow (kernel commits via tmp-file + rename; sizes monotonic within the run).
  • SQLite side: peak footprint of index.db/-wal/-shm. The WAL grows monotonically within a checkpoint-free window, so peak ≈ cumulative for these runs.
  • The first save is the corpus's initial materialization and is excluded from delta metrics by construction (deltaBytes = lastSize − firstSize).

Finding worth recording (crossover)

The SQLite leg is NOT free at small sizes: each ingest transaction dirties a fixed set of b-tree pages (whole 4KB WAL frames), so index-side growth is ~O(1) per save regardless of session size, while the legacy JSON rewrite is O(session size). Below roughly 100–200KB of session payload the index-only leg can cost MORE than legacy; above it (the 30–60MB production cases in #3049) it wins by an order of magnitude and keeps winning as sessions grow. This argues for keeping both write paths available until incremental persistence lands, rather than forcing every host onto the index early.

Behavior changes

None. Additive test file only — no src/ changes, no config surface touched (reads the existing knobs BILI_STORAGE_UNIFIED / BILI_STORAGE_JSON_WRITER / BILI_PERSIST_TAIL_TOKENS read-only), no version bump.

Verification

  • node --import tsx --test tests/storage-write-amplification.test.ts → 2/2 pass locally (~1.2s)
  • neighboring tests/storage-save-path.test.ts → 3/3 pass
  • npm run typecheck + npm run build green on this head

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-15_storage-write-amplification-pin (b52b7a4)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3864-pr.3275.f4a08f6

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3275.tgz
npm install -g package

Install instructions are refreshed on each push.

ranxianglei pushed a commit that referenced this pull request Oct 15, 2026
…l after every idle save (#3275)

writeNow -> foldResidentTail replaces session.lastMessages with a new
bounded array right after each idle save, so the pre-loop capture went
stale after save 1: deltas pushed into the orphaned array never reached
the envelope, deltaBytes pinned at 0 and the legacy-shape assertion
failed on master (CI test shard 2 red on ubuntu/windows). Re-reading
session.lastMessages inside the loop follows the post-save fold contract.

Local (master + PR): storage-write-amplification 2/2 pass
(legacy 226,736B/1,394B = 163x; composites 1.9MB < 13.1MB < 15.0MB),
storage-save-path 3/3, typecheck clean.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 open+closed issue/PR 全量检索,"write-amplification" 仅命中本 PR;相关但范围不同:#3049(生产测量,本 PR 的动机)、#2671(增量持久化本体,本 pin 服务的目标)。

审核结论:方向正确,发现一个真问题——测试在当前 master 上红,已直接在分支修好。

方向与质量(独立审核,非复读 issue):

  • 用「band + 排序」而非精确字节做 pin,是对的——时间戳/WAL 内部态天然 flaky,精确值必炸;
  • 交叉点(crossover)发现有价值:SQLite 每次 ingest 弄脏固定 b-tree 页(整帧 4KB WAL),小会话下 index-only 可能比 legacy 更贵,这为「两条写路径都保留到增量持久化落地」提供了实测依据;
  • hermetic 手法(BILI_CONFIG_FILE 指向空文件、双 tier 显式钉死、engine 缺失时优雅 SKIP、Windows best-effort rm)与相邻 tests/storage-save-path.test.ts 完全一致;
  • 断言有牙:legacy 形状 pin 在增量持久化落地时必须翻掉(amp 从 ~160× 塌向 1),mode 排序 pin 会挡住任何悄悄改变写路径成本结构的改动。

发现的问题(CI 已红):test (…, 2) shard 在 ubuntu-22/24、windows-22 失败(1/3 shard 绿)。根因不是环境:

修复(已在 PR 分支追加 commit 1ddb593fb,纯测试改动,不动 pin 语义):循环内每轮重读 session.lastMessages,并留了 why-comment 说明 R3 折叠契约,防止后人把它"简化"回单次捕获再踩一遍。本地验证(master + PR 合并态):

storage-write-amplification: 2/2 pass (~2s)
  legacy: 226,736B rewritten for 1,394B changed (163x)
  composites: index-only=1.92MB < legacy=13.06MB < shadow=14.98MB
storage-save-path: 3/3 pass
npm run typecheck: clean

行为变更披露:无产品行为变更(纯加测试文件,src/ 零改动,配置面只读复用现有 knob)。PR 描述中引用的三组 composite 数值(5.79/13.06/18.85MB)是修复前测的:旧引用下 index-only leg 每轮重复 ingest 一个逐 save 变化的折叠视图,修复后改为增量 ingest 16 个小 delta,index-only 降到 ~1.9MB;legacy/shadow 基本不变,排序与 ±5% band pin 均不受影响。


结构化八问

  1. 是什么问题:非缺陷修复——新增回归测试,把当前持久化写成本形状(legacy 每次 save 重写整个 envelope = O(session size);三种模式合成成本排序 index-only < legacy < shadow)钉死,给 Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671 增量持久化一个可测量的翻转目标。
  2. 什么原因导致的(审核中发现的问题):测试基于旧 master 编写,Memory plane (R3): stop keeping compressed originals, CCR and snapshots fully in RAM now that the unified store has them — measure first, then lazy-load #3050 R3 step 2 合入后 writeNow 每次 idle save 都会替换 session.lastMessages 数组,测试的循环前引用捕获从第二次 save 起失效,deltas 未落盘导致 deltaBytes=0 断言失败。
  3. 影响范围与严重程度:仅测试自身,不影响任何生产代码路径和用户;严重度 = PR 红灯(test shard 2),其余 lane 不受阻。
  4. 怎么修复的:循环内每轮重读 session.lastMessages(遵循 post-save 折叠契约)+ why-comment 防回归;pin 的对象和阈值一字未动。
  5. 是否有回归问题:无。触碰路径的既有行为逐项过:src/persist.ts 三种存储模式行为未变(测试对 knob 只读)、相邻 storage-save-path 套件 3/3 保持绿、typecheck 干净;无 fail-fast 变吞错、阈值/时序/输出格式漂移。
  6. 是否破坏缓存:不涉及(未触碰缓存/重试/游标/去重逻辑)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:无(复用 BILI_STORAGE_UNIFIED / BILI_STORAGE_JSON_WRITER / BILI_PERSIST_TAIL_TOKENS / BILI_PERSIST_ZSTD)。

判定:新 head(1ddb593fb)CI 转绿后即可合并;按规矩合并由你操作(我不能 merge PR)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

跟进:新 head 1ddb593fb 的 CI 已全绿——39/39 全部 success,包括之前红的 test (…, 2) shard(ubuntu-22/24、windows-22/24 全过)。修复生效确认。PR 现在可合并(合并由你操作)。

@ranxianglei
ranxianglei force-pushed the 2026-10-15_storage-write-amplification-pin branch 2 times, most recently from 1ddb593 to 149e519 Compare October 15, 2026 10:52
@ranxianglei
ranxianglei force-pushed the 2026-10-15_storage-write-amplification-pin branch from 149e519 to d403969 Compare October 15, 2026 13:01
@ranxianglei

Copy link
Copy Markdown
Owner Author

冲突

@ranxianglei
ranxianglei force-pushed the 2026-10-15_storage-write-amplification-pin branch from ec1be03 to b52b7a4 Compare October 15, 2026 17:21
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解决并推送。新 head:b52b7a422。

冲突原因:分支此前 rebase 到 6b5b28317,之后 master 又前进了(#3268 / #3053 / #3289 等)。冲突点在 tests/issue3148-upstream-circuit.test.ts:分支上的 ec1be0302 把 #3148 clamp pin 从 1360 重钉到 4096(#3071 capability floor),而 master 侧已合入了同样的重钉(断言值完全一致,仅注释措辞不同)——两边改了同一处。

解法:把分支重建在当前 master(6ee4a2ac9)之上,只保留一个提交 b52b7a422 = tests/storage-write-amplification.test.ts(含 R3 post-save fold 的重读修复的改进变体);冗余的 issue3148 重钉提交直接丢弃——master 已有等价修复,丢弃后本 PR 收敛为「纯新增一个测试文件」,diff 最干净。未动 master,force-with-lease 仅作用于 PR 分支(推送前已确认远端无更新提交)。

本地验证(合并态):

storage-write-amplification: 2/2 pass(legacy 224,546B/1,394B = 161×;composites 1.92MB < 13.06MB < 14.97MB)
storage-save-path: 3/3 pass
npm run typecheck: clean

结构化八问

  1. 是什么问题:PR 分支与 master 在 tests/issue3148-upstream-circuit.test.ts 上文本冲突——两边各自独立做了同一个修复([bug] 上游在会话中途不可用时,bili 把一次故障放大成四重代价:固定节奏 20 连击全部冷发、输出预算被静默钳制 83 次(最狠 65536→1360)、紧急收缩与 nudge 停摆均不可见 #3148 clamp pin 1360→4096,跟随 [bug] 0.1.195 输出预算三路饿死仍复现:无 output 上限模型 restore 无下限(1→252、fork 原样转发 1)、#2490 planning floor 钳到 1479–2623 → 3 会话 9 次 finish=length 空轮不重试(#546/#1665/#1840/#2904/#2756 残留) #3071 floor)。
  2. 什么原因导致的:分支基于旧 master(6b5b283)时该文件还是旧 pin,分支上补了重钉;随后 master 合入了自己的等价重钉,分支再往前推进时就撞上了。典型的「同一 CI 红被两条线各自修掉」。
  3. 影响范围与严重程度:仅测试文件,无生产代码/用户面影响;严重度 = PR 处于冲突态无法合并,CI 不跑。
  4. 怎么修复的:rebase 到当前 master,保留唯一有增量价值的提交(write-amplification 测试文件),丢弃与 master 等价的 issue3148 重钉——按「union of intent」取 master 已审版本,不引入重复改动。
  5. 是否有回归问题:无。触碰路径既有行为逐项过:issue3148 测试文件现在与 master 逐字节一致(master 版本本身是绿的);write-amplification 测试在新 base 上本地 2/2,pin 的 band/排序全部保持;相邻 storage-save-path 3/3、typecheck 干净。无阈值/时序/输出格式漂移。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:无。

判定:冲突已清,等 b52b7a422 上 CI 转绿即可合并(合并由你操作)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解决,分支重建到当前 master,新 head b52b7a422。

冲突原因:我推修复提交(1ddb593)后,另一条线把分支重组到了旧 base(6b5b283)上并附带了一个 #3148 重钉提交(ec1be03);同时 master 已前进到 6ee4a2a——而 master 侧已经合入了一个等价的 #3148 重钉(改同一处 assert)。两边都动了 tests/issue3148-upstream-circuit.test.ts,产生冲突。

处置(union-of-intent):以当前 master(6ee4a2a)为底重建分支,只保留本 PR 的增量测试文件提交(内含改进版 R3 修复:每次 save 前实时读 session.lastMessages,不再捕获一次旧引用);丢弃重复的 #3148 重钉——master 上已有等价版本且经过其自身评审,措辞从 master。--force-with-lease 仅推到本 PR 分支(推前核对远端仍是 ec1be03,未覆盖任何人的新提交)。

本地验证(merge-state = master + 本文件):storage-write-amplification 2/2(legacy 224,546B rewritten for 1,394B ≈ 161×;composite 1,919,040 < 13,055,016 < 14,974,056)、storage-save-path 3/3、typecheck 干净。

CI 现状(b52b7a4):本 PR 的两个新测试在 CI 通过(shard 3,ok 1741/ok 1742)。当前 7 条红 lane 全部是 master 自带的红,与本 PR diff 无关——证据:base 提交 6ee4a2a 的 master push CI(run 38767006158)同样 12 条 test lane 挂 10 条,失败签名逐字节一致:

  1. golden-storage-render ×4(chat/anthropic/responses/google,state 段 stats 键漂移)→ 已有 storage-render 金标快照未随 #3295 重新生成,master 与所有在途 PR 的测试分片全红 [no-dispatch] #3313,修复 PR test: rebaseline storage-render goldens after #3295 dropped dead image-stats fields from state.stats #3314 在途(rebaseline);
  2. #1086 T5 liveness(25 !== 60)→ 已在 CI 测试 lane 出现与代码改动无关的失败:存储渲染快照漂移 + 两个偶发用例 #3315 清单内(原判「偶发」)。我做了定向复现:确定性失败——根因是 :546 假上游每轮回显 "x".repeat(360),撞上今天刚合入的 Add single-codepoint repetition lane to the output-side runaway guard (Closes #3260) #3289 输出侧 text-run 护栏(src/runaway-guard.ts:72 阈值 200,内部常量),本地复现 35 次 guard tripped、签名与 CI 一致。详见我补到 CI 测试 lane 出现与代码改动无关的失败:存储渲染快照漂移 + 两个偶发用例 #3315 的证据,几行级修复 PR 随后补上。

八问

  1. 是什么问题:PR 分支与 master 冲突(两条线各自修了同一个过期钉死值);解完后剩余 CI 红均为 master 既有问题。
  2. 什么原因导致:双方在不同 base 上独立修了 [bug] 上游在会话中途不可用时,bili 把一次故障放大成四重代价:固定节奏 20 连击全部冷发、输出预算被静默钳制 83 次(最狠 65536→1360)、紧急收缩与 nudge 停摆均不可见 #3148 过期断言,master 期间又合入了等价版本;剩余红的根因分别是金标未随 state 形状变更重生成(storage-render 金标快照未随 #3295 重新生成,master 与所有在途 PR 的测试分片全红 [no-dispatch] #3313/feat!: remove image pre-compression and the image_full tool (#3266) #3295)与 T5 夹具撞上新护栏(Add single-codepoint repetition lane to the output-side runaway guard (Closes #3260) #3289)。
  3. 影响范围与严重程度:本 PR 零产品面影响(纯增量测试);合并门禁被 master 自带红阻塞,需 test: rebaseline storage-render goldens after #3295 dropped dead image-stats fields from state.stats #3314 + T5 修复落地后全绿。
  4. 怎么修复的:重建分支到当前 master,取意图并集(留增量测试、弃重复重钉),force-with-lease 仅限 PR 分支。
  5. 是否有回归问题:无 src/ 变更;两个新测试 CI 实测通过;相邻 storage-save-path 绿。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入新配置:不涉及。

判定:本 PR 内容就绪、自身全绿;待 #3314 与 T5 小修合入后 CI 转绿即可合并(合并由你操作)。T5 修复 PR 链接稍后补在本帖。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

T5 修复 PR 已开:#3317(修 #3315 子缺陷 2)——tests/issue1086-chain-self-exemption.test.ts:546 填充由 "x".repeat(360) 改等长混合字符 "xyz".repeat(120),护栏行为不动。本地 21/21 通过、0 次 guard tripped(修前 35 次);纯测试文件变更。

至此本 PR 合并路径上的已知红全部有归属:金标漂移 → #3314(在途),T5 → #3317(新)。两者合入后本 PR CI 应全绿;本 PR 自身无需再动。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

这条冲突通知是过期信号——分支已经在当前 master 上,无需 rebase,未做任何推送。

核实过程(全部实测,非推断):

  1. GitHub API:PR test: pin persistence write-amplification shape and mode cost ordering (#3049) #3275 mergeable: true、mergeable_state: "blocked" —— blocked 是「必需检查未绿」,不是文本冲突(冲突时 mergeable 会是 false)。
  2. 本地核对:远端 master HEAD = 6ee4a2ac9;PR head b52b7a422 的 merge-base 恰好就是 6ee4a2ac9 —— 分支直接坐在最新 master 上,对 master 的完整 diff 只有一个文件 tests/storage-write-amplification.test.ts(+266 行)。
  3. 时间线:这个 rebase 在昨晚 18:21 的上一轮冲突处理里已经完成(见本线程 floor 9/10,新 head 即 b52b7a422);此后 master 一个提交都没动(ls-remote 仍为 6ee4a2ac9)。本次 dirty 通知应是 rebase 之前触发、延迟送达的旧信号。按「冲突解完不重复 rebase」的规矩,没有做无内容的空推送。

当前 CI 现状(b52b7a4):本 PR 自己的两个测试在 CI 通过(ok 1741 legacy shape / ok 1742 mode ordering)。8 条红 lane 我逐条拉了 job 日志核对失败签名,全部是 master 自带的红,与本 PR diff 无关:

失败签名 lane 归属
#1086 T5 liveness(25 !== 60,#3289 护栏误伤夹具填充) ubuntu/windows × node22/24 shard 2 #3315 子缺陷 2,修复 PR #3317 open
storage-render golden ×4(chat/anthropic/responses/google,state 段漂移) ubuntu/windows × node22/24 shard 3 #3313,rebaseline PR #3314 open

交叉验证:base 提交 6ee4a2ac9 自身的 master push CI(run 38767006158)失败签名逐字节一致(昨晚已核过,master 此后未动,结论仍然成立)。#3314 / #3317 目前都还是 open,未合入。


结构化八问

  1. 是什么问题:系统转发「PR 与 master 冲突(dirty)」,要求 rebase。
  2. 什么原因导致的:通知是过期信号——rebase 昨夜已完成并推送(b52b7a422,基于 6ee4a2ac9),master 此后零移动,GitHub 判定 mergeable=true;通知在解决前就已生成、今天才送达。
  3. 影响范围与严重程度:无实际影响——分支状态本来就是干净的;真正阻塞合并的是 master 自带两条既有红(test: rebaseline storage-render goldens after #3295 dropped dead image-stats fields from state.stats #3314/修 CI 无故变红:#1086 T5 用例的 'x'×360 填充触发 #3289 输出护栏,含 #3289 的每棵树必挂 #3317 未合入),与本 PR 内容无关。
  4. 怎么修复的:无需修复。做了三方证据核实(API mergeable + merge-base + ls-remote),确认无冲突后按「不重复 rebase」纪律收尾,不做空推送。
  5. 是否有回归问题:不涉及(零改动、零推送)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:无。

判定:本 PR 自身就绪且全绿,无需再动;等 #3314、#3317 合入后 CI 转全绿即可合并(合并由你操作)。若平台侧 dirty 标记不自动清除,属检测器刷新问题,不是分支问题。

@ranxianglei
ranxianglei merged commit b23a02e into master Oct 16, 2026
31 of 39 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.479-master.b23a02e (rolling master tag).

npm install -g billion-context@0.0.479-master.b23a02e  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

LoveDoLove pushed a commit to LoveDoLove-Forked-Projects/billion-context that referenced this pull request Oct 16, 2026
…-run lane (ranxianglei#3315 sub-defect 2)

The T5 liveness fake upstream echoed "x".repeat(360) per turn. ranxianglei#3289's
output-side runaway guard trips at TEXT_RUN_THRESHOLD=200 identical
codepoints (src/runaway-guard.ts), aborting every such stream — so each
turn lost its visible reply and the suite failed deterministically
(25 !== 60) on every tree containing ranxianglei#3289: master push run 38767006158
and PR ranxianglei#3275 run 38776693851, plus local repro (35 guard trips).

Mixed-char filler "xyz".repeat(120) keeps the identical 360-byte payload
size (all assertions depend on length/byte count only) while maxTextRun
drops to 1. Guard behavior itself is untouched — ranxianglei#3289 is intentional
production behavior; the fixture simply must not emit pathological runs.

Verified: node --import tsx --test tests/issue1086-chain-self-exemption.test.ts
-> 21/21 pass, 0 guard trips (was 35).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant