Repository navigation
test: pin persistence write-amplification shape and mode cost ordering (#3049) - #3275
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.3864-pr.3275.f4a08f6This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3275.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…l after every idle save (#3275) writeNow -> foldResidentTail replaces session.lastMessages with a new bounded array right after each idle save, so the pre-loop capture went stale after save 1: deltas pushed into the orphaned array never reached the envelope, deltaBytes pinned at 0 and the legacy-shape assertion failed on master (CI test shard 2 red on ubuntu/windows). Re-reading session.lastMessages inside the loop follows the post-save fold contract. Local (master + PR): storage-write-amplification 2/2 pass (legacy 226,736B/1,394B = 163x; composites 1.9MB < 13.1MB < 15.0MB), storage-save-path 3/3, typecheck clean.
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复。 open+closed issue/PR 全量检索,"write-amplification" 仅命中本 PR;相关但范围不同:#3049(生产测量,本 PR 的动机)、#2671(增量持久化本体,本 pin 服务的目标)。 审核结论:方向正确,发现一个真问题——测试在当前 master 上红,已直接在分支修好。 方向与质量(独立审核,非复读 issue):
发现的问题(CI 已红):
修复(已在 PR 分支追加 commit 行为变更披露:无产品行为变更(纯加测试文件,src/ 零改动,配置面只读复用现有 knob)。PR 描述中引用的三组 composite 数值(5.79/13.06/18.85MB)是修复前测的:旧引用下 index-only leg 每轮重复 ingest 一个逐 save 变化的折叠视图,修复后改为增量 ingest 16 个小 delta,index-only 降到 ~1.9MB;legacy/shadow 基本不变,排序与 ±5% band pin 均不受影响。 结构化八问
判定:新 head( |
(ai: vllm-qwen/qwen3.8-27b) 跟进:新 head |
1ddb593 to
149e519
Compare
149e519 to
d403969
Compare
|
冲突 |
ec1be03 to
b52b7a4
Compare
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决并推送。新 head: 冲突原因:分支此前 rebase 到 解法:把分支重建在当前 master( 本地验证(合并态): 结构化八问
判定:冲突已清,等 |
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决,分支重建到当前 master,新 head 冲突原因:我推修复提交(1ddb593)后,另一条线把分支重组到了旧 base(6b5b283)上并附带了一个 #3148 重钉提交(ec1be03);同时 master 已前进到 6ee4a2a——而 master 侧已经合入了一个等价的 #3148 重钉(改同一处 assert)。两边都动了 处置(union-of-intent):以当前 master(6ee4a2a)为底重建分支,只保留本 PR 的增量测试文件提交(内含改进版 R3 修复:每次 save 前实时读 本地验证(merge-state = master + 本文件):storage-write-amplification 2/2(legacy 224,546B rewritten for 1,394B ≈ 161×;composite 1,919,040 < 13,055,016 < 14,974,056)、storage-save-path 3/3、typecheck 干净。 CI 现状(b52b7a4):本 PR 的两个新测试在 CI 通过(shard 3,
八问
判定:本 PR 内容就绪、自身全绿;待 #3314 与 T5 小修合入后 CI 转绿即可合并(合并由你操作)。T5 修复 PR 链接稍后补在本帖。 |
(ai: vllm-qwen/qwen3.8-27b) T5 修复 PR 已开:#3317(修 #3315 子缺陷 2)—— 至此本 PR 合并路径上的已知红全部有归属:金标漂移 → #3314(在途),T5 → #3317(新)。两者合入后本 PR CI 应全绿;本 PR 自身无需再动。 |
(ai: vllm-qwen/qwen3.8-27b) 这条冲突通知是过期信号——分支已经在当前 master 上,无需 rebase,未做任何推送。 核实过程(全部实测,非推断):
当前 CI 现状(b52b7a4):本 PR 自己的两个测试在 CI 通过(
交叉验证:base 提交 结构化八问
判定:本 PR 自身就绪且全绿,无需再动;等 #3314、#3317 合入后 CI 转全绿即可合并(合并由你操作)。若平台侧 dirty 标记不自动清除,属检测器刷新问题,不是分支问题。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.479-master.b23a02e # this exact build
npm install -g billion-context@master # always the newest merged build
|
…-run lane (ranxianglei#3315 sub-defect 2) The T5 liveness fake upstream echoed "x".repeat(360) per turn. ranxianglei#3289's output-side runaway guard trips at TEXT_RUN_THRESHOLD=200 identical codepoints (src/runaway-guard.ts), aborting every such stream — so each turn lost its visible reply and the suite failed deterministically (25 !== 60) on every tree containing ranxianglei#3289: master push run 38767006158 and PR ranxianglei#3275 run 38776693851, plus local repro (35 guard trips). Mixed-char filler "xyz".repeat(120) keeps the identical 360-byte payload size (all assertions depend on length/byte count only) while maxTextRun drops to 1. Guard behavior itself is untouched — ranxianglei#3289 is intentional production behavior; the fixture simply must not emit pathological runs. Verified: node --import tsx --test tests/issue1086-chain-self-exemption.test.ts -> 21/21 pass, 0 guard trips (was 35).
Follows #3049 (production measurements: full 57MB snapshot rewrites for +270B deltas; 1.42TB cumulative writes over 15h on a shadow-mode host). Adds a hermetic regression test that pins the persistence write-cost model so the upcoming storage work (#2671 unified storage / incremental persistence) has a measurable target to flip.
What it pins
tests/storage-write-amplification.test.tsdrives 16 small deltas through three storage legs in one process and asserts:BILI_PERSIST_TAIL_TOKENSso the pads survive the tail-budget trim): composite bytes (envelope + SQLite peak footprint) satisfyfull + jsonWriter=off<legacy<shadow. Measured: 5.79MB < 13.06MB < 18.85MB. Also pins that shadow pays exactly the legacy envelope cost (±5%) and thatjsonWriter=offwrites zero envelope files while still ingesting every delta into index.db.No exact-byte pins (timestamps, WAL internals are flaky); the pins are orderings and bands that hold across engines.
Methodology notes
.jsonfiles under the session dir after eachwriteNow(kernel commits via tmp-file + rename; sizes monotonic within the run).index.db/-wal/-shm. The WAL grows monotonically within a checkpoint-free window, so peak ≈ cumulative for these runs.deltaBytes = lastSize − firstSize).Finding worth recording (crossover)
The SQLite leg is NOT free at small sizes: each ingest transaction dirties a fixed set of b-tree pages (whole 4KB WAL frames), so index-side growth is ~O(1) per save regardless of session size, while the legacy JSON rewrite is O(session size). Below roughly 100–200KB of session payload the index-only leg can cost MORE than legacy; above it (the 30–60MB production cases in #3049) it wins by an order of magnitude and keeps winning as sessions grow. This argues for keeping both write paths available until incremental persistence lands, rather than forcing every host onto the index early.
Behavior changes
None. Additive test file only — no
src/changes, no config surface touched (reads the existing knobsBILI_STORAGE_UNIFIED/BILI_STORAGE_JSON_WRITER/BILI_PERSIST_TAIL_TOKENSread-only), no version bump.Verification
node --import tsx --test tests/storage-write-amplification.test.ts→ 2/2 pass locally (~1.2s)tests/storage-save-path.test.ts→ 3/3 passnpm run typecheck+npm run buildgreen on this head