Skip to content

[PR] fix: make snapshot-render D-lane test deterministic and surface degrade reason (#3771) - #3774

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-19_snapshot-render-ci-flake
Oct 19, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-19_snapshot-render-ci-flake

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Closes #3771

What was broken

The snapshot-render D-lane suite (tests/snapshot-render.test.ts) failed non-deterministically in CI (all five D-on tests) while passing locally. The trip-wire was D observability latch counted turns, but the real signal was replicasAvailable=false rows=null seqs=null — the unified replica was simply absent for the harness store.

Root cause (corrected from the earlier settle-timing framing)

Not a commit-timing race. The test never pinned its storage mode: top-of-file sets only NODE_ENV / BILI_PERSIST_ZSTD / BILI_CONFIG_FILE (pointing at a non-existent no-config.json). When the ambient BILI_STORAGE_UNIFIED resolves to anything other than shadow (0/false/legacy/garbage, or a config value), resolveStorageMode returns "legacy", where UnifiedShadow.maybeOpen returns null silently (no log) and the harness store has no unified sidecar. Every D-on test then silently ran the legacy path and tripped the latch / anchor / prefix asserts. Local runs had the var unset → default shadow → passed, which is what made it look environment-dependent.

Repro: BILI_STORAGE_UNIFIED=0 node --import tsx --test tests/snapshot-render.test.ts → # pass 2 # fail 5, byte-identical to CI.

Fix

  • Pin BILI_STORAGE_UNIFIED=shadow at the top of the test so the sidecar opens deterministically regardless of ambient env (the D-lane requires a readable unified transcript by construction).
  • Add an additive snapshotRenderDegradeReason marker on session.metadata: each post-arm degradation records why (store-unavailable / transcript-empty / row-unparsable / view-pointer-missing-row / transcript-error / chain-not-started), cleared on a successful render. Makes future D-lane degradation diagnosable instead of silent. No wire/message-shape change — every degrade still returns clientMsgs unchanged. Pinned by the storage-unavailable test.

Rebased onto current master; no conflict with f296b0e's settleRows.

Verification

  • Green under ambient BILI_STORAGE_UNIFIED = 0 / false / unset (was # pass 2 # fail 5 before), incl. the google lane added since.
  • npm run typecheck clean.

Behavior change (flagged)

  • New additive field session.metadata.snapshotRenderDegradeReason (string; set only on a post-arm degrade, cleared on success). Diagnostic-only — does not touch outbound bytes. Old: degrades were indistinguishable. New: the reason is recorded.
  • Test now explicitly pins storage mode to shadow (previously inherited the ambient value). No production default changed.

@ranxianglei ranxianglei changed the title fix: make snapshot-render D-lane test deterministic and surface degrade reason (#3771) [PR] fix: make snapshot-render D-lane test deterministic and surface degrade reason (#3771) Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-19_snapshot-render-ci-flake (3b45e67)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4479-pr.3774.b39aac7

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3774.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

范围缺口提醒(来自并行的 PR #3770 review):你的修复没有覆盖 turnGoogle()。

PR #3770(google 车道接入 snapshot render)在 tests/snapshot-render.test.ts 里新增了 turnGoogle() helper,携带与你修掉的 turn() 完全相同的固定 60ms settle 模式。你 patch 里的 awaitTranscriptReady/assertDActive 只应用到 openai 车道的测试(s-steady/s-upg/s-drop/s-rw/s-switch)。

证据:全量矩阵 run 39196237026(#3770 head 322502e)中,google 变体以与 openai 三件套完全相同的签名失败——D on · google: steady state 挂在 'D observability latch counted turns',D on · google: identity-seed upgrade 挂在 prefix diverges at outbound message 1。即同一根因、同一失败形态,只是换了车道。

建议:合并前把 awaitTranscriptReady + assertDActive 扩展到 turnGoogle(google 会话 label),否则两 PR 都合入后 google D-on 测试仍会在 CI 非确定失败,#3771 只能算修了一半。

合并顺序提示:两个 PR 改同一测试文件,#3770 的 google 测试是按修复前 harness 写的——后合的一方 rebase 时记得把新 helper 用上。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 本 PR 即 #3771 的修复 PR(唯一 Closes #3771);在途的 #3770(snapshot-render google-lane)与 #3753(lite-fold anchors)虽同属 snapshot-render 域,但症状与机制均不同,不构成重复。

标签: bug / scope/ci / p2(CI 非确定性红灯 + 假覆盖信号;功能可用但每次命中分片都可能红,需绕行)

审核进行中:已独立通读全部 diff(2 文件,+59/−13)与两个文件全文;master 自 PR base 前进了 5 个提交(hermes-plugin context window + 另两套测试的 teardown 收口),与本 PR 文件零重叠、无语义交互。本地定向测试与 CI 结果出来后给结构化八问结论。

@ranxianglei ranxianglei added bug Something isn't working p2 scope/ci labels Oct 19, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

结论:原 head(2eedf61)CI 红灯未过——但根因已独立钉死并修复,commit 828bea1 已推到本 PR 分支,等 CI 复验。

CI 事实(PR 自身 head 的 run 39198572850 @ 2eedf61)

根因(独立重推,不采纳 PR 正文的诊断)

tests/preload-storage-baseline.mjs:16 有一行 process.env.BILI_STORAGE_UNIFIED ??= "legacy",经 --import 注入每个 CI 测试子进程(PRELOAD_ARGS,scripts/run-ci-tests.mjs:146;scripts/test-suite-lock.mjs:234 同引用)。后果:CI 里 harness 的 SessionStore 一律走 mode==="legacy" 静默分支(src/storage/shadow.ts:76 maybeOpen 直接返回 null,无任何日志)→ unified sidecar 从不打开 → D 车道结构性永不激活(rows 永远 null,replicasAvailable 永远 false)。轮询 5 秒也救不了一个从来没有 sidecar 的 store。

证据链(本地可复现,非推断):

  1. 用 CI 完全相同的命令本地跑:node --import ./tests/preload-storage-baseline.mjs --import ./tests/preload-zone-slot.mjs --import tsx --test tests/snapshot-render.test.ts → 5/5 失败,签名与 CI 逐字节一致(含 seed-upgrade 的 prefix diverges——那正是 test 6 文档化的 legacy churn,D-off 下断言的就是它);去掉 preload → 7/7 通过。确定性、与环境负载无关。
  2. 两条失败腿的完整日志里没有任何 sidecar failed to open / disabled after N failures / 加密阻断 warn → 排除打开失败、ingest 故障、加密阻断,只剩静默 legacy 分支一条路。
  3. 为何能合入:[PR] Snapshot rendering: proxy lane outbound prefix rendered from server-side canonical history; existing sessions immune to | 快照渲染:proxy 车道出站前缀从服务端正史渲染,存量会话对内核升级免疫(#3756) #3759 的 pre-merge 绿矩阵是 skip-scoped 空矩阵(Release PRs should default to running the full matrix: the ci:full label has a merge-timing race condition (v0.1.200's | 发版 PR 应默认跑全量矩阵:ci:full 标签存在合并时序竞态(v0.1.200 全量被取消后白付两遍) #3688 label race),该套件从未在真实 preload 下跑过 CI;本地跑不加载 preload,所以「本地恒绿 + CI 恒红」。PR 正文的「60ms 提交延迟/CPU starvation」诊断不成立。

修复(已推 828bea1,独立标注 commit)

在 preload 内做文件级豁免:node --test 每文件一个子进程,argv 恰好指向一个套件;仅 tests/snapshot-render.test.ts 豁免(名单带注释、可扩展),其余文件保持 legacy baseline(probe 验证:非豁免子进程仍收到 PROBE_ENV=legacy)。修后同一 CI 命令本地 7/7;纯本地跑保持 7/7。你原 commit 的 polling + loud guard 保留不动——guard 正是让真签名可见的东西,polling 继续防御真实的提交延迟。Windows 安全:harness teardown 先 store.closeQuietly() 再 rmrf(tests/snapshot-render.test.ts:146→147),sidecar tmpdir 无 EPERM 风险。

行为变更披露(old → new)

  • 原 PR:新增 additive session.metadata["snapshotRenderDegradeReason"](诊断专用,成功即清;随既有 free-form metadata key 走持久化/ingest 通道,无 wire/面板影响)。
  • 我的 commit:仅测试基建——该套件在 CI 中改按生产默认(shadow)运行而非 legacy baseline;生产代码零改动。

结构化八问

  1. 是什么问题:D 车道 e2e 套件在 CI 里不是 flaky,是结构性死灯——CI 预载把存储模式钉死 legacy,sidecar 从不打开,车道从未激活过。
  2. 什么原因导致的:Inbound markers: one declarative strip engine for all four wires (#2789, #2798, #2811 plan B) #2815 落地时给全体测试语料铺了 legacy baseline(preload ??=),snapshot-render 套件依赖生产默认(shadow)却没进豁免名单;[PR] Snapshot rendering: proxy lane outbound prefix rendered from server-side canonical history; existing sessions immune to | 快照渲染:proxy 车道出站前缀从服务端正史渲染,存量会话对内核升级免疫(#3756) #3759 的空矩阵绿让它带着这个矛盾上了船。
  3. 影响范围与严重程度:任何排到该文件的 CI 腿必红(曾两度阻塞 [PR] fix: hash-free lite fold anchors for past-cap tool cores (#3752) #3753 rollup)+ 假覆盖信号(绿 CI ≠ D 车道被验证)。用户面零影响(纯测试基建);严重度 p2。
  4. 怎么修复的:在 baseline 源头(preload)按文件路径豁免需要 sidecar 的套件,单一事实源同时覆盖两个引用方;polling/loud guard 保留为第二道防线。
  5. 是否有回归问题:审过的既有行为——全体语料仍收 legacy baseline(probe 实证)、加密阻断/打开失败路径未动、wire/消息形状未动(degrade 返回原数组引用、成功路径仅多一次 metadata delete)、metadata 新字段走既有通道、面板不受影响(命名字段白名单)。未发现无意回归;残留风险仅「该套件首次在 Windows 上真实开 sidecar」(handle 收口已核实)。
  6. 是否破坏缓存:不涉及——未触碰任何缓存/重试/游标/去重键;metadata 字段不在 wire 字节内,前缀缓存稳定性不受影响。
  7. 是否改了设计方向:不涉及——additive 诊断字段 + 测试基线作用域收敛。
  8. 是否引入新配置:不涉及——无新 env/config/CLI 面;BILI_STORAGE_UNIFIED 语义与 ??= 逃生口不变。

余留事项

判定:828bea148 CI 转绿 + #3770 缺口处置明确后即可合并;合并动作留给人工。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解决,rebase 到最新 master 并 force-push(新 head e01aff57f)。

冲突根因:本 PR 和已合入的 #3749(f296b0e0d)各自独立修了同一个 flake——tests/snapshot-render.test.ts 里写死的 60ms sleep 在 CI 高负载下等不到 SQLite commit 落地。两边改的是同一行,所以撞了。解法取舍:master 的 settleRows(按 label 轮询行集变化、rewind 收缩也算数、5s 超时响亮报错)比本 PR 原来的 awaitTranscriptReady(只等 rows>0,第 2 轮起立刻返回、保证不了当前轮 commit 落地)严格更强,所以确定性机制整体采用 master 版,删掉本 PR 被取代的 helper(不留死代码)。

八问:

  1. 是什么问题:PR 分支与 master 冲突——双方对同一处测试代码给了两套修法。
  2. 什么原因导致的:同一个 flake(CI red: snapshot-render D-lane tests fail non-deterministically ('D observability latch counted turns') while passing locally — #3759 merged with unverified-in-CI tests #3771/[PR] Payload-addressed image message IDs: seed adds media fingerprint slot (#3751) | 图片消息 id 按载荷寻址:种子增加媒体指纹槽(#3751) #3767 家族)被两条并行工作流独立修复,改在同一行上;且本 PR 原方案本身比 master 方案弱。
  3. 影响范围与严重程度:纯测试文件 + 一个观测性 metadata 字段,无用户可见行为;此前只影响该套件在 CI 上的确定性。
  4. 怎么修复的:rebase 后取「master 的 settleRows 机制 + 本 PR 独有增量」的并集——assertDActive 守卫(5 个 D-on 用例显式证明 D-lane 真的渲染过,没激活就抛出 degradeReason + store 状态,堵住「静默走 legacy 也绿」的盲区)、src/server/snapshot-render.ts 的 legacyFor 降级原因标记、preload 豁免(snapshot-render 套件不再被钉死 legacy 存储基线)。
  5. 是否有回归问题:无生产行为变更——legacyFor 只写 session.metadata 观测字段、从不碰出站字节,pre-arm 退出路径刻意不打标;其余所有套件的 legacy 基线不受豁免影响(双向探针实测:非豁免 argv → 仍钉 legacy,snapshot-render argv → 不钉)。缓存/重试/游标均不涉及。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及(测试基建 + 可观测性)。
  8. 是否引入新配置:无——BILI_STORAGE_UNIFIED 是既有变量,只是测试 preload 不再对这一个套件钉它。

诚实说明一句:rebase 之后,PR 标题里「make deterministic」那半已经被 master 上 #3749 的合入覆盖,本 PR 现在的实际范围 = 降级原因显性化 + 存储基线豁免。

本地验证:typecheck 干净;tests/snapshot-render.test.ts 定向跑 7/7 全绿(D-on 用例全部通过 assertDActive,即车道真实激活而非静默回退);build 通过。全量回归以远程 CI 在新 head 上的结果为准。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

进展简报(floor 6 之后):master 已前进到 2b40342——#3770(google lane,#3764)已合入,你的 f296b0e「settle on the store tail view」也已在 master。PR 分支现与 master 冲突(mergeable_state=dirty),正在按「union of intent」rebase:

  • 采纳你的 settleRows(等 tail view CHANGES,天然覆盖 null→rows 与 rewind 收缩两种迁移)→ 我原 commit 里的 awaitTranscriptReady(只查非空,对 r2+ 的「行数早已非空」场景有洞)弃用;
  • 保留本 PR 的两块不可替代部分:① preload 豁免(828bea1)——master 的 tests/preload-storage-baseline.mjs 仍是裸 ??="legacy",不豁免则 CI 里 sidecar 结构性不打开,settleRows 只会 5s 后超时抛错(响亮但仍是红的);② assertDActive 响亮守卫 + snapshotRenderDegradeReason 元数据——settleRows 只证明「行落了」,不解释「行为什么没渲染」,且 google 变体目前无任何激活守卫。

rebase 完成后会跑 typecheck + 精确 CI 命令定向验证再推,CI 结果出来补八问收尾。

@ranxianglei
ranxianglei force-pushed the 2026-10-19_snapshot-render-ci-flake branch 3 times, most recently from c57144b to 0193499 Compare October 19, 2026 05:36
awork added 2 commits October 19, 2026 13:40
The D-lane e2e (tests/snapshot-render.test.ts) failed non-deterministically
in CI while passing locally: most functional asserts also hold in legacy
mode, so a run where the D lane never activated read as "flaky latch"
instead of "lane never activated". The fixed-window nap replacement now
lives on master as settleRows (f296b0e: wait until the session's tail
view CHANGES); this commit lands the two pieces that remain:

- assertDActive(): loud guard in every D-on test (openai AND google
  variants). If the lane did not actually render, it throws naming the
  degrade reason plus store state (replicasAvailable/rows/seqs), closing
  the silent-legacy blind spot so a future genuine degrade fails
  informatively instead of tripping an opaque latch assert.

- src/server/snapshot-render.ts: additive session.metadata['snapshotRenderDegradeReason']
  marker set on each post-arm degrade (store-unavailable/transcript-empty/
  row-unparsable/view-pointer-missing-row/transcript-error/chain-not-started)
  and cleared on success. Pre-arm exits (rollout off / nothing armed) are
  untouched. No wire or message-shape change: degrades still return
  clientMsgs unchanged.

- Contract pins both directions (#3617 gate discipline): the
  storage-unavailable test asserts the surfaced reason ('transcript-empty');
  steady-state asserts the marker clears after successful D renders.
The test never pinned its storage mode. An ambient BILI_STORAGE_UNIFIED of
legacy/0/false (or a config value) resolves mode to "legacy", where maybeOpen
returns null SILENTLY and the harness store has no unified sidecar. Every D-on
test then silently ran the legacy path and failed on the D latch / anchor /
prefix asserts — CI-only red because local runs had the var unset (default
shadow). Pinning shadow makes the sidecar open deterministically; verified green
under =0/=false/unset incl. the google lane.
@ranxianglei
ranxianglei force-pushed the 2026-10-19_snapshot-render-ci-flake branch from 0193499 to 3b45e67 Compare October 19, 2026 05:44
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

最终状态:分支已收敛为两个 commit(e9b71de + 3b45e67,rebase 到当前 master 2b40342),本地全绿,等 CI 复验。

过程披露(重要,如实说明)

  1. master 移动了:[PR] feat: wire google lane into snapshot render (#3764) #3770(google lane)已合并,且你本人的 f296b0e(harness 用 settleRows 替代 60ms nap)也落进了 master。我按 union-of-intent 重做:采纳你的 settleRows(替代我早前的 awaitTranscriptReady 轮询——对 r2+ 的 tail 变化有盲区,你的版本更准),保留 loud guard 并把它接到全部 7 个 D-on 测试(5 openai + 2 google)——floor 4 提的 [PR] feat: wire google lane into snapshot render (#3764) #3770 turnGoogle() 缺口就此关闭。
  2. 发现并行会话在同分支迭代:期间另一 agent 会话先后推了 e01aff5 → c06a263(仅 env pin)→ 0193499(pin + degradeReason metadata,无 loud guard)。我没有盲目覆盖,逐文件 diff 后取并集:
    • 采纳它的 suite-local env pin(tests/snapshot-render.test.ts 头部无条件 BILI_STORAGE_UNIFIED = "shadow"):它严格覆盖我早前的 preload 豁免方案——无条件赋值胜过任何 ambient 值(含 CI preload 的 ??= legacy),且符合存储家族「每个套件自钉模式」的既有惯例,覆盖所有调用路径。preload 文件恢复 master 原样。
    • 采纳它的 src 打磨:degrade 日志行带 session=<id>、doc 注释显式点名两个 metadata 字段。
    • 保留我的 assertDActive loud guard(它的树里没有——那正是本 PR 正文承诺的第二半)。
  3. 最终两 commit 各自独立标注,信息无损。

根因(floor 7 已详述,一句话版)

CI preload 把 BILI_STORAGE_UNIFIED ??= "legacy" 注入每个测试子进程 → harness store 走 maybeOpen 的静默 legacy 分支、sidecar 从不打开 → D 车道结构性永不激活;PR 原诊断「60ms 提交延迟」不成立(本地已用 CI 同款命令逐字节复现)。

行为变更披露(old → new)

  • 新增 additive session.metadata["snapshotRenderDegradeReason"](诊断专用,成功渲染即清;随既有 free-form metadata key 走持久化/ingest 通道,无 wire/面板影响;所有 degrade 仍返回 clientMsgs 原引用)。
  • 测试基建:该套件显式钉死 shadow(生产默认),不再受任何 ambient legacy 值影响;生产代码除上述标记外零改动。

结构化八问

  1. 是什么问题:D 车道 e2e 在 CI 里不是 flaky,是结构性死灯——存储模式被 baseline 钉成 legacy,sidecar 从未打开,车道从未激活。
  2. 什么原因导致的:Inbound markers: one declarative strip engine for all four wires (#2789, #2798, #2811 plan B) #2815 的 legacy baseline(preload ??=)未给依赖生产默认(shadow)的本套件留口子;[PR] Snapshot rendering: proxy lane outbound prefix rendered from server-side canonical history; existing sessions immune to | 快照渲染:proxy 车道出站前缀从服务端正史渲染,存量会话对内核升级免疫(#3756) #3759 的空矩阵绿让它带着矛盾上了船。
  3. 影响范围与严重程度:排到该文件的 CI 腿必红(曾两度阻塞 [PR] fix: hash-free lite fold anchors for past-cap tool cores (#3752) #3753 rollup)+ 假覆盖信号;用户面零影响,p2。
  4. 怎么修复的:套件自钉 shadow(确定性打开 sidecar)+ post-arm degrade 原因上 metadata + 每个 D-on 测试 loud guard(未激活即抛错并报 degradeReason/store 状态)。三层各管一段:pin 管激活、marker 管可观测、guard 管断言不撒谎。
  5. 是否有回归问题:wire/消息形状未动(degrade 返回原数组引用、成功路径仅多一次 metadata delete);面板不受影响(命名字段白名单);preload 恢复原样,其余语料 baseline 不变;类型检查绿。未发现无意回归。
  6. 是否破坏缓存:不涉及——未触碰缓存/重试/游标/去重键;metadata 字段不在 wire 字节内。
  7. 是否改了设计方向:不涉及——additive 诊断字段 + 测试基线作用域收敛。
  8. 是否引入新配置:不涉及——无新配置面;BILI_STORAGE_UNIFIED 语义与逃生口不变。

本地验证(最终树 3b45e67)

判定:代码侧已闭环(CI 复验为最后一步);合并动作留给人工。

@ranxianglei
ranxianglei merged commit 5afa77f into master Oct 19, 2026
32 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.598-master.5afa77f (rolling master tag).

npm install -g billion-context@0.0.598-master.5afa77f  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/ci

Projects

None yet

1 participant