Repository navigation
Open
Labels
Description
同一根本原因(performance-memory)的 open issue 汇聚,由根因归类板自动维护(10min 刷新,勿手编)。
Members (15)
- 镜像 [PR] fix(dsh): parse the pre-send request body once per request #5798(GitHub [perf][dsh] 25.8-32.7 s of each step elapses before bili sees the body (bili itself: 5.5-7.2 s) | TTFT 归因:每步 25.8-32.7 s 发生在 bili 收到请求体之前 #5792) — [perf][dsh] 25.8-32.7 s of each step el…
根因: TTFT attribution on a 29,619-msg/51.2MiB dsh session: aligning DSH step/start e… - 镜像 [PR] fix(persist): stop re-reading the envelope on every index-only save (#5805) #5808(GitHub [perf][pool] Default residency budget of 512 MiB causes a 33 MB session to be evicted every round alongside a 578 MB lar | [perf][pool] 默认驻留预算 512 MiB 让 33 MB 会话在 578 MB 大会话旁每轮被驱逐(每次重载 3.4 s,全在事件循环上) #5806) — [perf][pool] Default residency budget o…
根因: default sessions.residentMaxBytes=512MiB sits below a single large session's me… - 镜像 [PR] Revert "[PR] feat(fastpath): v6 composite-identity incremental fastpath lane (full rework, #5576 line voided)" #5854(GitHub decision request: resident-pool default budget — keep 512 MiB, raise to 2 GiB, or scale with physical RAM (+ two related pending calls) | 决策请求:常驻池默认预算维持/上调/按物理内存自适应(附两项关联待决) #5852) — decision request: resident-pool default…
根因: decision request: default sessions.residentMaxBytes=512MiB vs A2 static ~2GiB v… - 镜像 [PR] fix(nudge): no billing sample this round → hold pressure-band injection one round (phantom-pressure guard) #5860(GitHub [Bug][dsh] 0.1.212 / Windows: host exit 134 (2.2 GB V8 heap OOM) on a 3.8 MiB long session — likely #5811 + #5813 + #5418 combined | 3.8 MiB 长会话触发 host OOM #5858) — [Bug][dsh] 0.1.212 / Windows: host exit…
根因: dsh desktop HOST process (not proxy) exits 134: V8 heap OOM at 1.7-2.2GB reside… - 镜像 #4516's oversize circuit breaker has a dual false positive: B/4 token estimation overestimates CJK content by 7x + windo | #4516 的 oversize 熔断存在双重误判:B/4 token 估算对 CJK 内容高估 7 倍 + 窗口比较未使用实际请求模型的上下文窗口 #5871(GitHub 0.1.212 still reproduces the structural issue from #4528: single-request local preprocessing peaks at 144.7s; conversati | 0.1.212 仍复现 #4528 的结构性问题:单请求本地预处理最高 144.7s,关闭插件后对话明显变快 #5869) — 0.1.212 still reproduces the structural…
根因: Long sessions starve the event loop: a 4,600-message session blocks the proxy and the web UI for 7–50 s per turn #4528 structural cost persists in 0.1.212 despite [PR] fix(#4528): yield to the event loop during long turns; warn on slow sessions #4544 stopgap: every turn run… - 镜像 [Bug][dsh lane] 0.1.213 macOS: a detached proxy that outlives the host keeps running but loses Local Network authorization (all LAN upstreams EHOSTUNREACH, cloud fine), and host restart re-attaches to it so it never heals | 守护进程跨宿主存活后失去本地网络授权,attach 复用旧进… #6257(GitHub [Bug][storage] CPU profile of v0.1.213 slow-ingest: canonicalJson + serialize + canonicalDigest dominate the delta save path #6254) — [Bug][storage] CPU profile of v0.1.213 …
根因: a CPU profile of a live v0.1.213 proxy attributes the recurring slow-ingest wal… - 镜像 Kernel authorization request: On first write, the visibility watermark for the time attribute is set below the stamped r | Kernel authorization request: 时间属性可见水位线首次写入时定在已盖戳引用之下(从 v0.1.213 更新后的第一个请求会把所有现有标签的 time= 去掉) #6327(GitHub Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325) — Performance issue: saving a sub-2 KB co…
根因: single session of 33,332 messages, zero concurrency, unified storage (index.db … - 镜像 zstd request decoding blocks cancellation on native-capable Node runtimes #6488(GitHub zstd request decoding blocks cancellation on native-capable Node runtimes #6488) — zstd request decoding blocks cancellati…
根因: synchronous fzstd push on native-capable runtimes ignores AbortSignal; gzip/br/… - 镜像 [PR] fix: interpret cache-diff byte offsets in original text coordinates #6496(GitHub Log time-window queries retain an unbounded pre-match context buffer #6486) — Log time-window queries retain an unbou…
根因: time-window query retains every pre-match line in leadIn before the bounded tai… - 镜像 # billion-context Long Session Crash Report | # billion-context 长会话崩溃报告 #6757(GitHub # billion-context Long Session Crash Report | # billion-context 长会话崩溃报告 #6757) — # billion-context Long Session Crash Re…
根因: surface only-grows (a fold only skips on send, it does not reduce surfaceTokens… - 镜像 0.1.214 moved the two
slow-ingestcalls per round ahead offorward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次slow-ingest挪进了forward POST之前,首字从 2.3s 涨到 7.7s #6780(GitHub 0.1.214 moved the twoslow-ingestcalls per round ahead offorward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次slow-ingest挪进了forward POST之前,首字从 2.3s 涨到 7.7s #6780) — 0.1.214 moved the twoslow-ingestcal…
根因: 0.1.214 (PR [PR] fix: persist ledger rows before the model call (durability-before-render) #6326 durability-before-render + pure-SQLite writer-off microtask) … - 镜像 [Drive-by] forced-path image-rescale docs contradict the code on within-maxDim handling #7150(GitHub 性能清理提案:同一份内容每请求被重复哈希 3-5 次 —— 盘点结论与分批清理方案 #7149) — [Ideas] 性能清理提案:同一份内容每请求被重复哈希 3-5 次 —— 盘…
根因: performance proposal (audit: ~70 files, ~80 hash sites): the same content hashe… - 镜像 Compression replay retries consume unbounded upstream error bodies #7200(GitHub Compression replay retries consume unbounded upstream error bodies #7200) — Compression replay retries consume unbo…
根因: fetchWithRetry (src/fetch-util.ts) calls response.text() for EVERY rejected HTT… - 镜像 Kernel orphan-ref bookkeeping repeatedly scans persistent hidden refs #7222(GitHub Kernel orphan-ref bookkeeping repeatedly scans persistent hidden refs #7222) — Kernel orphan-ref bookkeeping repeatedl…
根因: hideConsumedCompressCalls (kernel/src/hide-consumed.ts) re-scans the persistent… - 镜像 Unchanged CCR stores rescan every historical ref during request preparation #7223(GitHub Unchanged CCR stores rescan every historical ref during request preparation #7223) — Unchanged CCR stores rescan every histo…
根因: adoptContentStore (src/store.ts) rebuilds a Set of every previous CCR ref and f…
Task
对本批 issue 做一个根本性修复 PR:
- PR 描述里用
Fixes #...逐个列出上面每一个 GitHub issue 号(没有 GitHub 号的写镜像号) - 若某个 issue 已有自己的 PR 且方向一致,cherry-pick 其 commit 并保留原作者署名(
-x);由你判断是否值得 - PR 必须打标签:
root-cause(固定,来自根因汇聚单)+ 按本 PR 的 diff 独立判断的类别和优先级标签(最高优先用 p1),不要抄任何来源 issue - PR 描述末尾必须加标记行
<!-- rcagg:performance-memory -->(汇聚板靠它自动识别根因 PR 补标签/插队;即使修复拆成多个子单各自提 PR,每个 PR 都要带这行) - 回帖本线程必须是最终裁决,结尾固定用中文 8问 清单逐行回答:1 这是啥问题(大白话+触发条件)2 根本原因是啥(组件+机制)3 如何修复(已做的修复或建议修法)4 能否合并/关闭(结论+依据:测试/CI/回归分析)5 URL(相关线程/PR 链接)6 影响面(哪些代码路径/用户)7 测试(什么钉住它)8 风险(还可能坏在哪)。答不实的行写「待验证:原因」,不许漏行
Activity
ranxianglei commentedon Nov 13, 2026
继续:请按本聚合单的根因归类完成分析,并直接在本线程回复 8问 结论(1啥问题 2根因file:line 3修复思路 4能否合并 5URL 6影响面 7测试证据 8风险)。消息不要只输出空白,现在就发已有证据的结论。
ranxianglei commentedon Nov 13, 2026
继续:请按本聚合单的根因归类完成分析,并直接在本线程回复 8问 结论(1啥问题 2根因file:line 3修复思路 4能否合并 5URL 6影响面 7测试证据 8风险)。现在就发已有证据的结论,不要只输出空白。
ranxianglei commentedon Nov 13, 2026
(ai: vllm-qwen/qwen3.8-27b)
标签: bug + dsh + scope/storage + p1(性能缺陷家族,主战场在统一存储写路径/常驻池,多数成员是 dsh 车道,单轮 144.7s + 事件循环饥饿 + host OOM 属体验毁灭级)。
查重(出站): 7 个成员已逐一核 open+closed issue 与全部 100 个 open PR——剩余两个子缺陷(#6254 规范化无缓存、#6325 sessions 行整行覆盖)无在途 PR;#5846(驱逐豁免)在途待合;#5901/#5873(快照表底座)是更大架构、默认全关,与最小修复不重叠。
根因归类结论(performance-memory,7 项同一机制家族):每轮成本随会话状态总量而非增量增长,且全部同步跑在单事件循环上。三腿:
逐项状态(全部对 master git log 验证过,非照抄 issue 侧结论):
src/storage/store.ts:1028-1031仍无条件覆盖 4 个大列,delta/full 共用同一 upsert(:1007)要开的唯一根本修复 PR:剩余两子缺陷(#6254 + #6325)同机制(save 路径对全量状态重做工作),按一个 issue 一个 PR 政策在本单内以子缺陷清单管理、一个 PR 修掉:
COALESCE(@x, sessions.x)→ 单轮写入 13 MB → ≈3.5 MB,零语义变更(实测一轮通常只有 stats_json/tail_json 真变);PR 描述将逐个列 7 个 GitHub 号:#6254/#6325 用
Fixes(本 PR 真修);#5792/#5806/#5858/#5869 标注「已由 <已合 PR> 修,commit 见上」;#5852 标注「D1 已由 #5779 解决,D2/D3 待裁决」——不修的绝不假写 Fixes。结构化八问
src/storage/store.ts:1007-1033sessions upsert 无条件覆盖 metadata_json/kstate_json/tail_json/ccr_json(:1028-1031),metadata_json 内四个账本字段(cacheLedger/foldAnchors/foldPositions/foldAnchorOrder ≈ 8.5 MB)随历史单调增长 → 每轮 13 MB overflow 页推 WAL;②src/storage/canonical.ts:94canonicalDigest 一族无内容身份缓存,每次 save 重做规范化/序列化/摘要;③ 旧默认预算 512 MiB(src/knobs.ts,Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 前)< 单会话权重 → evict/cold-load 抖动;④ 以上全部同步在主循环,[PR] fix(#4528): yield to the event loop during long turns; warn on slow sessions #4544 只 yield 不减总 CPU。sessions.hardMaxBytes(Windows, 0.1.210) #5403 / [feat][dsh] Long sessions have no convergence path: compaction gated off on both sides, incremental lane full-resends on OpenAI wire | 长会话没有收敛通道:compaction 双向封锁 + 增量车道全量重发 #5796。src/storage/store.ts(ingestSession upsert)、src/storage/canonical.ts、常驻池(src/session.ts)、dsh-native 前置钩子(已修);用户面 = 统一存储长会话用户(每轮延迟 + 事件循环饥饿波及同进程全部会话)、dsh 进程内用户(修复前 host OOM)、单会话 >512 MiB 的多会话用户(Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 前)。判定:根因归类完成——7 项中 5 项已修已合(SHA 为证)、1 项在途待合(#5846)、剩余 2 子缺陷同机制,我将开一个根本修复 PR(Fixes #6254 + #6325)并在 CI 绿后回帖。三项等你拍板:① 合 #5846;② 读侧方向 A/B/C(建议 A);③ #5852 D2(建议不做)/ D3(建议 C2)。
ranxianglei commentedon Nov 13, 2026
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
完整根因判定 + 8问已在本线程 floor 11(comment 79039,2026-11-13 03:10Z)发出——本条 nudge 为误报(回复在 completion 标记之前已落地,可核对 comment id 79039)。
当前状态
Fixes #6254+Fixes #6325)——①src/storage/store.tssessions 行按列判变(COALESCE + 上次写入字节哈希,未变列不写);②src/storage/canonical.tscanonicalize/serialize/digest 按内容身份记忆化(按字节上限)。CI 绿后回帖。8问(紧凑版,完整版见 floor 11)
src/storage/store.ts:1028-1031无条件覆盖 4 个大 JSON 列(metadata_json/kstate_json/tail_json/ccr_json,实测 13 MB,其中账本字段随历史单调增长);src/storage/canonical.ts:94canonicalDigest 一族无内容身份缓存;旧默认预算 512 MiB(src/knobs.ts,Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 前)<单会话权重;以上全部同步在主循环。sessions.hardMaxBytes(Windows, 0.1.210) #5403/[feat][dsh] Long sessions have no convergence path: compaction gated off on both sides, incremental lane full-resends on OpenAI wire | 长会话没有收敛通道:compaction 双向封锁 + 增量车道全量重发 #5796。src/storage/store.ts(ingestSession upsert)、src/storage/canonical.ts、常驻池(src/session.ts)、dsh-native 前置钩子(已修);用户面=统一存储长会话用户(每轮延迟+事件循环饥饿波及同进程全部会话)、dsh 进程内用户(修复前 host OOM)、单会话>512 MiB 的多会话用户(Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 前)。判定:根因归类已交付(floor 11),修复 PR 为下一步动作,三项决策等你拍板。
[-][Root Cause] performance-memory (7 items)[/-][+][Root Cause] performance-memory (9 items)[/+]ranxianglei commentedon Nov 13, 2026
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
进度汇报(本会话因 34 分钟无输出被强制重启——卡住原因是我做了大文件整读,已改为小批量定点读,避开该操作)。
已完成:
2026-11-13_save-path-incremental(基于最新 master 243709b)。src/storage/store.ts:1047-1050sessions upsert 无条件覆盖 4 个大列(已证实);src/storage/ingest.ts:500-505finishPlan 每次 save(含 delta 路径)都重序列化全部大列;src/storage/shadow.ts:449record() 主路径 + :547 采样校验(digest 只在采样时算,非每 save)。进行中:实现修复——按列判变:JS 侧对每个大列缓存上次写入字节,未变则 upsert 用
CASE WHEN @changed THEN excluded.col ELSE sessions.col END保留原值(不用 COALESCE,因 ccr_json 合法值可为 null)→ 单轮写入 13 MB → ≈3.5 MB,零语义变更。下一步:写代码 → 回归测试(未变列不重写 + 漂移时强制重写保自动愈合)→ typecheck + 定向测试 → push → 开 PR(Fixes #6254 + #6325)→ 回本线程。
8 问(当前状态版,结论同 floor 11 未变):
src/storage/store.ts:1047-1050upsert 无条件覆盖 4 大列;src/storage/ingest.ts:500-505每次 save 重序列化全部大列;旧 512 MiB 默认预算(Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 前)< 单会话权重;以上全同步在主循环。sessions.hardMaxBytes(Windows, 0.1.210) #5403/[feat][dsh] Long sessions have no convergence path: compaction gated off on both sides, incremental lane full-resends on OpenAI wire | 长会话没有收敛通道:compaction 双向封锁 + 增量车道全量重发 #5796。src/storage/store.ts/src/storage/ingest.ts/src/storage/canonical.ts/ 常驻池 / dsh-native 前置钩子(已修);用户 = 统一存储长会话用户、dsh 进程内用户、单会话 >512 MiB 多会话用户。ranxianglei commentedon Nov 13, 2026
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
根本修复 PR 已开:#6538
做了什么:
ingestSession的 sessions 行 upsert 现在按列判变——JS 侧对 4 个大 JSON 列(metadata_json / kstate_json / tail_json / ccr_json)各缓存上次写入字节的 SHA-256;新字节与上次一致时,ON CONFLICT DO UPDATE腿走CASE WHEN @colChanged保留存储值,不重写。新会话的 INSERT 腿不受影响;进程重启(无缓存条目)或漂移自愈(clearLastWrittenColumns,已接进 shadow.ts 采样校验的 drift 路径)保守全量写。零语义变更:落盘字节不变,只减写入工作量。稳态轮(只追加消息)现在 13 MB 大列写 ≈0(追加的 refs 落独立的 refs 表,#5755 已做 O(diff))。
本地验证(定向,非全量):
npm run typecheck干净;tests/issue6325-column-change-detection.test.ts4/4(首存全写 / 稳态跳写 / 突变检测 / 漂移重武装);issue5744-ingest-diff5/5、issue5794-ingest-cap+unified-remap-collision6/6 全绿。PR 描述已逐个列 7 个成员号:#6325 用
Fixes(本 PR 真修);#6254 标注「本 PR 去掉写入成本面(主导成本),规范化记忆化(计算成本面)在本聚合单跟踪」;#5792/#5806/#5858/#5869 标注「已由 <已合 PR> 修 + SHA」;#5852 标注「D1 已由 #5779 解决,D2/D3 待裁决」。结构化八问
src/storage/store.ts:1047-1050sessions upsert 的ON CONFLICT DO UPDATE腿无条件metadata_json = excluded.metadata_json等 4 列覆盖,delta/full 共用同一 upsert,无按列判变。lastWrittenColumnsMap,128 B/session),未变列的 UPDATE 腿走CASE WHEN @colChanged保留存储值;新会话 INSERT 腿不受影响;进程重启/漂移自愈保守全量写。零语义变更(落盘字节不变)。fix:、干净 diff(67 行 + 测试文件)。全量回归等 CI 绿。Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325 在 PR 合入后由你验证关闭。src/storage/store.ts(ingestSession upsert)+src/storage/shadow.ts(drift 路径接 clearLastWrittenColumns);用户面 = 统一存储长会话用户(每轮 save 延迟从 687–752 ms 降到近 0,事件循环占用同步下降,同进程其他会话 + Web UI 受益)。tests/issue6325-column-change-detection.test.ts钉 4 个行为——① 首存全写(无缓存条目 → flags=1);② 稳态跳写(字节一致 → flags=0,存储字节不变);③ 突变检测(列变了 → flags=1,新字节落盘);④ 漂移重武装(clearLastWrittenColumns → 下次全量写)。判定:PR #6538 已开,等 CI 绿。三项仍等你拍板:① 合 #5846(驱逐豁免);② 读侧方向 A/B/C(建议 A);③ #5852 D2(建议不做)/ D3(建议 C2)。CI 绿后我会在本线程回帖确认。
[-][Root Cause] performance-memory (9 items)[/-][+][Root Cause] performance-memory (11 items)[/+][-][Root Cause] performance-memory (11 items)[/-][+][Root Cause] performance-memory (9 items)[/+]ranxianglei commentedon Nov 14, 2026
[bot] (ai: vllm-qwen/qwen3.8-27b)
协调说明(自 #6630 线程):#6630(compress loop 丢弃的 upstream 响应体未关闭)于 2026-11-13 22:50Z 归入本聚合单,但 #6538 开于 12:38Z(归入之前),不含该修复;本线程最后活动(12:46Z)也早于归入——吸收步骤此前无人认领。
已按本单任务指令(「若某个 issue 已有自己的 PR 且方向一致,cherry-pick 其 commit 并保留原作者署名(-x)」)执行:
src/loop/core.ts所有权边界两处 fire-and-forget cancel +tests/loop-stream-cleanup.test.ts12 个测试)cherry-pick 进 [PR] fix: skip byte-identical large columns in the sessions-row upsert (#6325) #6538 分支,新 commit b08b488(-x 保留署名);#6538 分支现有新 commits b08b488 / 0cd7925,如该 PR 有其他在途工作请知悉。
6 remaining items
ranxianglei commentedon Nov 16, 2026
(ai: vllm-qwen/qwen3.8-27b)
#5507 线程新生产数据点(0.1.215 / Linux / DSH Web 原生插件,3.8 GiB VPS,报告者 @bsbofmusic),用于区分三腿:
cold-loaded: 8786 refs, 123 blocks (76 active), baseline=415649 tok, compressions=123→ 05:23:45 软告警(32s)→ 无该请求完成行 → 05:26 health 超时;profiling 配方已在 #5507 线程给出(NODE_OPTIONS 继承 + 软告警时 SIGTERM 落盘;perf 外部采样对 SIGKILL 安全)。详见 #5507。
[-][Root Cause] performance-memory (12 items)[/-][+][Root Cause] performance-memory (11 items)[/+][-][Root Cause] performance-memory (11 items)[/-][+][Root Cause] performance-memory (13 items)[/+][-][Root Cause] performance-memory (13 items)[/-][+][Root Cause] performance-memory (15 items)[/+]ranxianglei commentedon Nov 19, 2026
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
交叉链接:本家族新复现 GitHub #7274(镜像 #7274,OMP 19.0.0 / 0.1.216 / Linux WSL,报告者 @cy77cc)——OMP 并发多 agent 会话共享一个 bili 代理 + 大会话索引(293 MiB + 21 MiB WAL)触发同一条链:
[storage] slow-ingest(path=delta took=136771ms)→ event-loop-watchdog FATAL 304s(硬阈值 300s)→ 代理死亡 → 所有 OMP 会话 502。与 DSH 复现(#5507)的差异:OMP 并发 agent 是负载模式(多会话共享单代理,save 在单事件循环上串行化),不是新根因——根因仍是本聚合的「同步存储 ingest 成本随总会话状态增长」。OMP 侧 502 恢复子面(插件从 model config 取固定 origin、不重解析健康替代,respawn 归 launcher/host)归 process-lifecycle 家族(#5507/#6897)跟踪,不在本单。
已在 #7274 打 duplicate 标签(不关闭,关闭权在 owner)。
[-][Root Cause] performance-memory (15 items)[/-][+][Root Cause] performance-memory (11 items)[/+][-][Root Cause] performance-memory (11 items)[/-][+][Root Cause] performance-memory (15 items)[/+][-][Root Cause] performance-memory (15 items)[/-][+][Root Cause] performance-memory (19 items)[/+][-][Root Cause] performance-memory (19 items)[/-][+][Root Cause] performance-memory (15 items)[/+]wstczyw commentedon Nov 20, 2026
Data point for this family on a newer build, plus one downstream effect.
Setup: 0.0.1239-master.1e357bf, Windows 11,
bili starton 8787 (unified storage, full mode).index.dbis about 1.05 GiB, and the WAL peaked at 75.9 MiB. Claude Code and Codex sessions go through this proxy.slow-ingeston 11-20, 00:00–14:20 local:The slowest two were
path=full took=14245ms, 21.3 MiB WALandpath=full took=13396ms, 20.2 MiB WAL. No event-loop-watchdog kill: no stall came near 5 min.Question: is 85%
path=fullexpected on this build, after #6349 and #6356? The log line says only "watermark missing/cleared or tail shape changed". If the cause is known, a counter per cause would help to tell which one fires.Downstream effect: these stalls also block
/__bili/health, because it is answered on the same event loop. magpie runsbili startfor its/bili/lanes. When two 2 s health checks miss in a row, it ends its own bili. It also ends a new one that does not answer within 20 s, and opening a 1 GiB store takes longer than that. So normal load turns into restarts that drop requests in flight. Filed with a patch: yetone/magpie#7309. bili's own watchdog header already says the right rule ("deadlock detection, not load detection"). Any other supervisor that probes health with a short timeout will do the same thing.Also on every start (maybe separate):
[session-order] hydrate failed — skeletons stay on the legacy file path for this process ([storage] extension value exceeds 4194304 bytes — store large payloads in the content CAS instead)with 179 skeletons.ranxianglei commentedon Nov 20, 2026
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
Good data point — three distinct items in one comment, let me separate them.
1. 85%
path=full— is it expected after #6349/#6356?Short answer: not expected for plain sessions, but expected for sessions that carry a
pluginSnapshot. The delta fast path (planIngestDelta,src/storage/ingest.ts:318) returnsnull→ full path whensession.pluginSnapshot.length > 0, by design ("its entries may dedup against the trusted prefix and renumber the log"). Fork sessions and certain plugin sessions setpluginSnapshot(src/plugin.ts:1871,:639,:683). If your Claude Code / Codex sessions are fork sessions or carry a snapshot, every save takes the full path — that alone explains 85%.Other fallback causes (each returns
null→ full): watermark missing/cleared (DIGEST DRIFT), tail shorter than trusted prefix (fold), boundary identity mismatch (history edit), head-slot id mismatch (prefix rewrite),decompositionLogpresent (import). #6349/#6356 fixed two specific DIGEST DRIFT producers, but other causes remain.Per-cause counter: agreed, this is the right diagnostic. I'll add a counter per fallback reason in
ingestDelta/planIngestDeltaso the log line can say which cause fired, not just "watermark missing/cleared or tail shape changed". This is a small additive change (no behavior change) — I'll land it as a follow-up commit on the #6538 branch or a separate small PR.2. Health-check stall → magpie kills bili
This is a downstream effect of the same mechanism (synchronous ingest on the single event loop). My PR #6538 reduces the write cost (skips byte-identical columns) but does not eliminate the synchronous processing — a 14 s full ingest still blocks
/__bili/healthfor 14 s. The structural fix (health check on a separate mechanism, or async ingest) is a bigger architectural change and out of scope for this PR. Your magpie patch (yetone/magpie#7309) is the right workaround on the supervisor side. I'll note the bili-side structural fix as a separate item in this thread.3.
[session-order] hydrate failed — extension value exceeds 4194304 bytesThis is a separate bug from the save-path family: the session-order skeleton is stored as a SQLite extension value (user data) and exceeds the 4 MiB limit. The error message itself says "store large payloads in the content CAS instead" — the fix is to route the skeleton through the content CAS when it exceeds the inline limit. 179 skeletons affected. This is a distinct symptom + mechanism (storage size limit, not ingest performance) — I'll file it as a separate issue per the one-issue-one-scope rule.
PR #6538 CI status: 1 failure —
test (windows-latest, 22, 5): failure. 29/30 green. Investigating the Windows shard failure now; will report back.ranxianglei commentedon Nov 20, 2026
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
PR #6538 CI follow-up: the single failure (
test windows-latest 22 shard 5, exit code 1) has been re-triggered viarerun-failed-jobs. All other 29 checks are green (1 skipped). The failure is isolated to one Windows shard while the other four Windows shards (22, 1/2/3/4) pass — pattern consistent with a flaky/environment-specific test rather than a regression from the per-column change-detection change (which is storage-layer and covered by the greenissue6325-column-change-detection+issue5744-ingest-diff+issue5794-ingest-capsuites). Will report the re-run result here.ranxianglei commentedon Nov 20, 2026
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
Filed the session-order hydrate failure as a separate issue: #7378 (distinct symptom + mechanism from the save-path family — SQLite extension value 4 MiB limit, not ingest performance).
CI re-run for PR #6538 (Windows shard 5) still in progress — will report the result here when it completes.
ranxianglei commentedon Nov 20, 2026
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
PR #6538 CI — root cause of the Windows shard-5 failure identified. Fetched the full job log (2.6 MB). The failure is not a regression from the per-column change-detection change. It's the known Windows teardown-lock issue (#4347):
Test totals: 2374 tests, 2341 pass, 13 fail (all teardown-lock), 20 skipped. The 13 failures are Windows file-locking artifacts (AV/indexer holding SQLite handles during teardown), not assertion failures. All storage-layer tests (issue6325-column-change-detection, issue5744-ingest-diff, issue5794-ingest-cap) pass.
Triggered a second re-run (draws a fresh runner per the CI retry policy). Will report the result here.
ranxianglei commentedon Nov 20, 2026
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
PR #6538 CI — second re-run (fresh runner) also failed. Same 13 teardown-lock failures. The CI log's own diagnosis: "if the lock reproduces there too the leaked-handle hypothesis is primary (#4347)".
Key facts:
Map<string, {…Hash}>field and changes the upsert SQL. No new handle lifecycle.Assessment: the 13 teardown-lock failures are the known #4347 Windows file-locking issue, not a regression from this PR. The PR is otherwise fully green (29/30 checks). Flagging for your judgment: merge with the known #4347 Windows shard exception, or hold until #4347 is resolved.