Skip to content

[Root Cause] performance-memory (15 items) #6449

Open

Description

@ranxianglei

同一根本原因(performance-memory)的 open issue 汇聚,由根因归类板自动维护(10min 刷新,勿手编)。

Members (15)

Task

对本批 issue 做一个根本性修复 PR:

  • PR 描述里用 Fixes #... 逐个列出上面每一个 GitHub issue 号(没有 GitHub 号的写镜像号)
  • 若某个 issue 已有自己的 PR 且方向一致,cherry-pick 其 commit 并保留原作者署名(-x);由你判断是否值得
  • PR 必须打标签:root-cause(固定,来自根因汇聚单)+ 按本 PR 的 diff 独立判断的类别和优先级标签(最高优先用 p1),不要抄任何来源 issue
  • PR 描述末尾必须加标记行 <!-- rcagg:performance-memory -->(汇聚板靠它自动识别根因 PR 补标签/插队;即使修复拆成多个子单各自提 PR,每个 PR 都要带这行)
  • 回帖本线程必须是最终裁决,结尾固定用中文 8问 清单逐行回答:1 这是啥问题(大白话+触发条件)2 根本原因是啥(组件+机制)3 如何修复(已做的修复或建议修法)4 能否合并/关闭(结论+依据:测试/CI/回归分析)5 URL(相关线程/PR 链接)6 影响面(哪些代码路径/用户)7 测试(什么钉住它)8 风险(还可能坏在哪)。答不实的行写「待验证:原因」,不许漏行

Activity

  1. ranxianglei commented on Nov 13, 2026

    @ranxianglei
    OwnerAuthor

    继续:请按本聚合单的根因归类完成分析,并直接在本线程回复 8问 结论(1啥问题 2根因file:line 3修复思路 4能否合并 5URL 6影响面 7测试证据 8风险)。消息不要只输出空白,现在就发已有证据的结论。

  2. ranxianglei commented on Nov 13, 2026

    @ranxianglei
    OwnerAuthor

    继续:请按本聚合单的根因归类完成分析,并直接在本线程回复 8问 结论(1啥问题 2根因file:line 3修复思路 4能否合并 5URL 6影响面 7测试证据 8风险)。现在就发已有证据的结论,不要只输出空白。

  3. ranxianglei commented on Nov 13, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    标签: bug + dsh + scope/storage + p1(性能缺陷家族,主战场在统一存储写路径/常驻池,多数成员是 dsh 车道,单轮 144.7s + 事件循环饥饿 + host OOM 属体验毁灭级)。

    查重(出站): 7 个成员已逐一核 open+closed issue 与全部 100 个 open PR——剩余两个子缺陷(#6254 规范化无缓存、#6325 sessions 行整行覆盖)无在途 PR;#5846(驱逐豁免)在途待合;#5901/#5873(快照表底座)是更大架构、默认全关,与最小修复不重叠。

    根因归类结论(performance-memory,7 项同一机制家族):每轮成本随会话状态总量而非增量增长,且全部同步跑在单事件循环上。三腿:

    1. 写腿(master 仍在):每次 save(哪怕 delta 只加几条消息)无条件整行覆盖 sessions 的 4 个大 JSON 列(实测 13 MB:metadata_json 内 cacheLedger 3.3 MB + foldAnchors 3.1 MB + foldPositions 1.3 MB + foldAnchorOrder 0.7 MB,加 ccr_json 2.4 MB、kstate_json 1.8 MB),且 canonicalize/serialize/digest 每次对全部内容重做、无内容身份缓存;
    2. 池腿(已修):旧默认 512 MiB < 单大会话权重 → 每轮驱逐 + 3.4 s 同步冷重载抖动;
    3. 循环腿(部分已修):以上全部同步占主循环,同进程所有会话 + Web UI 一起排队([perf][dsh] 25.8-32.7 s of each step elapses before bili sees the body (bili itself: 5.5-7.2 s) | TTFT 归因:每步 25.8-32.7 s 发生在 bili 收到请求体之前 #5792 的 25.8–32.7 s hostGap 主体是这段排队,不是宿主构建——报告者对照实验已钉死)。

    逐项状态(全部对 master git log 验证过,非照抄 issue 侧结论):

    成员(GitHub 号) 状态
    #5792 hostGap 25.8–32.7 s bili 侧重复解析已修(#5798,c1d8ad3fbf 已合);排队主体随写腿/池腿修复消失;残余 2.3–4.4 s 是 DSH 自身 buildRequest,不在 bili 范围
    #5806 驱逐抖动 #5783 冷加载宽限(79db55bded)+ #5779 默认 512 MiB→4 GiB(062e1dd78a)已合;近端豁免 #5846 在途待合
    #5852 决策单 D1 已被 #5779 解决(静态 4 GiB,超出原推荐 2 GiB);D2(heapUsed 硬准入,建议 B1 不做)/ D3(memory 端点快照,建议 C2)仍待你裁决
    #5858 host OOM 2.2 GB 两个根机制已修:#5814(c3dfbe6b21)+ #5815(892865d34b)已合,fastpath 恢复可命中后每轮全量重解析的生产者消失
    #5869 单轮 local 144.7 s 写腿已修(#5755,83bddaa067 已合,refs/ccr_entries/blocks 三表改 O(diff));读侧 O(n) pass 仍待你方向裁决(A 哈希记忆化 / B worker_threads / C 不动,建议 A)
    #6254 规范化占 12 s/23 s 未修——delta save 路径对 canonicalJson/serialize/canonicalDigest 无内容身份缓存(CPU profile 直接证据)
    #6325 每轮重写 13 MB 未修——src/storage/store.ts:1028-1031 仍无条件覆盖 4 个大列,delta/full 共用同一 upsert(:1007)

    要开的唯一根本修复 PR:剩余两子缺陷(#6254 + #6325)同机制(save 路径对全量状态重做工作),按一个 issue 一个 PR 政策在本单内以子缺陷清单管理、一个 PR 修掉:

    PR 描述将逐个列 7 个 GitHub 号:#6254/#6325 用 Fixes(本 PR 真修);#5792/#5806/#5858/#5869 标注「已由 <已合 PR> 修,commit 见上」;#5852 标注「D1 已由 #5779 解决,D2/D3 待裁决」——不修的绝不假写 Fixes。

    结构化八问

    1. 这是啥问题:长会话(数千至数万条消息、inbound >3.8 MiB)下 bili 每轮成本随会话状态总量而非增量增长——每轮重写 13 MB 账本 + 重做全部规范化/摘要,且同步占死单事件循环,同进程所有会话与 Web UI 一起被拖;旧 512 MiB 驻留预算还会让「每轮驱逐 → 3.4 s 冷重载」抖动;dsh 进程内会话上每轮全量重发+全量解析把 host 进程推到 2.2 GB OOM。触发条件:统一存储 + 长会话;或单会话 >512 MiB(修复前默认);或 dsh native 车道 + 会话 >3.8 MiB(修复前)。
    2. 根本原因是啥:save 路径对全量状态重做工作而非增量。① src/storage/store.ts:1007-1033 sessions upsert 无条件覆盖 metadata_json/kstate_json/tail_json/ccr_json(:1028-1031),metadata_json 内四个账本字段(cacheLedger/foldAnchors/foldPositions/foldAnchorOrder ≈ 8.5 MB)随历史单调增长 → 每轮 13 MB overflow 页推 WAL;② src/storage/canonical.ts:94 canonicalDigest 一族无内容身份缓存,每次 save 重做规范化/序列化/摘要;③ 旧默认预算 512 MiB(src/knobs.ts,Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 前)< 单会话权重 → evict/cold-load 抖动;④ 以上全部同步在主循环,[PR] fix(#4528): yield to the event loop during long turns; warn on slow sessions #4544 只 yield 不减总 CPU。
    3. 如何修复:7 个 PR 已合入([PR] fix(dsh): parse the pre-send request body once per request #5798/[PR] fix(dsh): size authored-compaction usage from the forged checkpoint (#5811) #5814/[PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815/[PR] fix: stop resident-pool evict/cold-load thrash that OOMs the proxy (#5782) #5783/Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779/[PR] fix(storage): refs ledger diffs instead of full-replace per save (#5744) #5755/[PR] fix(#4528): yield to the event loop during long turns; warn on slow sessions #4544,SHA 见上表);[PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846 在途待合。剩余两子缺陷一个 PR:按列判变(COALESCE + 上次写入哈希,未变列不写)+ 规范化内容身份记忆化(按字节上限)。两者零语义变更(落盘字节不变,只减工作量)。读侧 O(n) pass 与 decision request: resident-pool default budget — keep 512 MiB, raise to 2 GiB, or scale with physical RAM (+ two related pending calls) | 决策请求:常驻池默认预算维持/上调/按物理内存自适应(附两项关联待决) #5852 D2/D3 是架构/行为决策,不拍板不开工。
    4. 能否合并/关闭:[PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846 可合(CI 绿,驱逐策略变更已在 PR 披露,观测字段纯增量)——合并权在你。剩余两子缺陷的 PR 可开、CI 绿后可合,回归测试钉「未变列不重写」+「记忆化命中字节一致」。5 个已修成员:修复已在 master(git log 为证),按你的口径我不关闭,等你验证后统一关。decision request: resident-pool default budget — keep 512 MiB, raise to 2 GiB, or scale with physical RAM (+ two related pending calls) | 决策请求:常驻池默认预算维持/上调/按物理内存自适应(附两项关联待决) #5852 在 D2/D3 裁决完之前不能关。
    5. URL:成员 GitHub [perf][dsh] 25.8-32.7 s of each step elapses before bili sees the body (bili itself: 5.5-7.2 s) | TTFT 归因:每步 25.8-32.7 s 发生在 bili 收到请求体之前 #5792 / [perf][pool] Default residency budget of 512 MiB causes a 33 MB session to be evicted every round alongside a 578 MB lar | [perf][pool] 默认驻留预算 512 MiB 让 33 MB 会话在 578 MB 大会话旁每轮被驱逐(每次重载 3.4 s,全在事件循环上) #5806 / decision request: resident-pool default budget — keep 512 MiB, raise to 2 GiB, or scale with physical RAM (+ two related pending calls) | 决策请求:常驻池默认预算维持/上调/按物理内存自适应(附两项关联待决) #5852 / [Bug][dsh] 0.1.212 / Windows: host exit 134 (2.2 GB V8 heap OOM) on a 3.8 MiB long session — likely #5811 + #5813 + #5418 combined | 3.8 MiB 长会话触发 host OOM #5858 / 0.1.212 still reproduces the structural issue from #4528: single-request local preprocessing peaks at 144.7s; conversati | 0.1.212 仍复现 #4528 的结构性问题:单请求本地预处理最高 144.7s,关闭插件后对话明显变快 #5869 / [Bug][storage] CPU profile of v0.1.213 slow-ingest: canonicalJson + serialize + canonicalDigest dominate the delta save path #6254 / Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325;已合 PR [PR] fix(dsh): parse the pre-send request body once per request #5798 / [PR] fix(dsh): size authored-compaction usage from the forged checkpoint (#5811) #5814 / [PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815 / [PR] fix: stop resident-pool evict/cold-load thrash that OOMs the proxy (#5782) #5783 / Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 / [PR] fix(storage): refs ledger diffs instead of full-replace per save (#5744) #5755 / [PR] fix(#4528): yield to the event loop during long turns; warn on slow sessions #4544;在途 [PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846;祖先单 Long sessions starve the event loop: a 4,600-message session blocks the proxy and the web UI for 7–50 s per turn #4528 / 0.1.212: per-turn ingest is still O(history) — 96.5% of proxy CPU in StatementSync.run, event loop starves with 8 concurrent sessions (#4528 / #5469 follow-up) #5744 / [Bug][dsh] Proxy OOMs by self-hitting the 8 GiB heap in ~30-90 min: resident pool budget (512 MiB) is smaller than a single session, so every round thrashes cold-load/evict and allocates giant temporaries #5782 / [Bug][dsh] Plugin lane performs a full re-parse and resends the entire body every round (81 MiB → hundreds of MB AST), w | [Bug][dsh] plugin 车道每轮全量解析重发 body(81 MiB→数百 MB AST)是 #5403 OOM 的生产者:用水位线断言替换逐条复核,服务端只解析新增尾段 #5418 / [Bug][dsh] 20k-message / 81 MiB-inbound session: host aborted 6× in 2 h — the #3050 resident-AST OOM is not bounded by sessions.hardMaxBytes (Windows, 0.1.210) #5403 / [feat][dsh] Long sessions have no convergence path: compaction gated off on both sides, incremental lane full-resends on OpenAI wire | 长会话没有收敛通道:compaction 双向封锁 + 增量车道全量重发 #5796。
    6. 影响面:代码路径 src/storage/store.ts(ingestSession upsert)、src/storage/canonical.ts、常驻池(src/session.ts)、dsh-native 前置钩子(已修);用户面 = 统一存储长会话用户(每轮延迟 + 事件循环饥饿波及同进程全部会话)、dsh 进程内用户(修复前 host OOM)、单会话 >512 MiB 的多会话用户(Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 前)。
    7. 测试:已合 PR 各自带回归测试([PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815 fastpath 回归、[PR] fix(storage): refs ledger diffs instead of full-replace per save (#5744) #5755 delta 车道测试、Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 default-cap pin 27c3256 等);剩余 PR 钉两个行为:① delta save 且大列未变时 WAL 无增长/列未被重写(COALESCE 路径断言);② 记忆化命中返回字节一致的 canonical bytes(含失效路径:内容变了必须重算)。
    8. 风险:① 判变哈希 bug 会留陈旧列(变了但判成没变)→ 存储态与内存态分叉;缓解 = 哈希「将要写入的精确字节」、哈希缺失时回退全量写;② 记忆化缓存必须按字节设上限,否则重引入本单要修的不受控增长;③ 写腿修完后非 fastpath 安装的主导成本移到读侧 O(n) pass(0.1.212 still reproduces the structural issue from #4528: single-request local preprocessing peaks at 144.7s; conversati | 0.1.212 仍复现 #4528 的结构性问题:单请求本地预处理最高 144.7s,关闭插件后对话明显变快 #5869 残留)——这正是 A/B/C 裁决要拍的;④ 4 GiB 默认(Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779)让所有安装稳态 RSS 可能上升(release notes 已披露),小内存机器 + 多会话会驻留更多状态,单会话仍受 hard cap([Performance] v0.1.208 long sessions exceed 2 GiB memory and spend 14–22 seconds in local processing #4788)约束。

    判定:根因归类完成——7 项中 5 项已修已合(SHA 为证)、1 项在途待合(#5846)、剩余 2 子缺陷同机制,我将开一个根本修复 PR(Fixes #6254 + #6325)并在 CI 绿后回帖。三项等你拍板:① 合 #5846;② 读侧方向 A/B/C(建议 A);③ #5852 D2(建议不做)/ D3(建议 C2)。

  4. ranxianglei commented on Nov 13, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    完整根因判定 + 8问已在本线程 floor 11(comment 79039,2026-11-13 03:10Z)发出——本条 nudge 为误报(回复在 completion 标记之前已落地,可核对 comment id 79039)。

    当前状态

    8问(紧凑版,完整版见 floor 11)

    1. 这是啥问题:长会话下 bili 每轮成本随会话状态总量而非增量增长——每轮重写 13 MB 账本 + 重做全部规范化/摘要,同步占死单事件循环拖累同进程所有会话与 Web UI;旧 512 MiB 驻留预算还会「每轮驱逐→3.4 s 冷重载」抖动;dsh 进程内会话被每轮全量重发+全量解析推到 2.2 GB host OOM。触发:统一存储+长会话 / 单会话>512 MiB(修复前默认)/ dsh native 车道+会话>3.8 MiB(修复前)。
    2. 根本原因是啥:save 路径对全量状态重做工作而非增量——src/storage/store.ts:1028-1031 无条件覆盖 4 个大 JSON 列(metadata_json/kstate_json/tail_json/ccr_json,实测 13 MB,其中账本字段随历史单调增长);src/storage/canonical.ts:94 canonicalDigest 一族无内容身份缓存;旧默认预算 512 MiB(src/knobs.ts,Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 前)<单会话权重;以上全部同步在主循环。
    3. 如何修复:7 个 PR 已合([PR] fix(dsh): parse the pre-send request body once per request #5798/[PR] fix(dsh): size authored-compaction usage from the forged checkpoint (#5811) #5814/[PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815/[PR] fix: stop resident-pool evict/cold-load thrash that OOMs the proxy (#5782) #5783/Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779/[PR] fix(storage): refs ledger diffs instead of full-replace per save (#5744) #5755/[PR] fix(#4528): yield to the event loop during long turns; warn on slow sessions #4544,SHA 见 floor 11);[PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846 在途;剩余 2 子缺陷一个 PR:按列判变(COALESCE+上次写入哈希,零语义变更)+ 规范化内容身份记忆化(按字节上限)。
    4. 能否合并/关闭:[PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846 可合(CI 绿、驱逐策略变更已披露、观测字段纯增量),合并权在你;剩余 PR 可开、CI 绿后可合;5 个已修成员修复已在 master(git log 为证),按你的口径我不关闭、等你验证后统一关;decision request: resident-pool default budget — keep 512 MiB, raise to 2 GiB, or scale with physical RAM (+ two related pending calls) | 决策请求:常驻池默认预算维持/上调/按物理内存自适应(附两项关联待决) #5852 在 D2/D3 裁决完前不能关。
    5. URL:成员 GitHub [perf][dsh] 25.8-32.7 s of each step elapses before bili sees the body (bili itself: 5.5-7.2 s) | TTFT 归因:每步 25.8-32.7 s 发生在 bili 收到请求体之前 #5792/[perf][pool] Default residency budget of 512 MiB causes a 33 MB session to be evicted every round alongside a 578 MB lar | [perf][pool] 默认驻留预算 512 MiB 让 33 MB 会话在 578 MB 大会话旁每轮被驱逐(每次重载 3.4 s,全在事件循环上) #5806/decision request: resident-pool default budget — keep 512 MiB, raise to 2 GiB, or scale with physical RAM (+ two related pending calls) | 决策请求:常驻池默认预算维持/上调/按物理内存自适应(附两项关联待决) #5852/[Bug][dsh] 0.1.212 / Windows: host exit 134 (2.2 GB V8 heap OOM) on a 3.8 MiB long session — likely #5811 + #5813 + #5418 combined | 3.8 MiB 长会话触发 host OOM #5858/0.1.212 still reproduces the structural issue from #4528: single-request local preprocessing peaks at 144.7s; conversati | 0.1.212 仍复现 #4528 的结构性问题:单请求本地预处理最高 144.7s,关闭插件后对话明显变快 #5869/[Bug][storage] CPU profile of v0.1.213 slow-ingest: canonicalJson + serialize + canonicalDigest dominate the delta save path #6254/Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325;已合 [PR] fix(dsh): parse the pre-send request body once per request #5798/[PR] fix(dsh): size authored-compaction usage from the forged checkpoint (#5811) #5814/[PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815/[PR] fix: stop resident-pool evict/cold-load thrash that OOMs the proxy (#5782) #5783/Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779/[PR] fix(storage): refs ledger diffs instead of full-replace per save (#5744) #5755/[PR] fix(#4528): yield to the event loop during long turns; warn on slow sessions #4544;在途 [PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846;祖先单 Long sessions starve the event loop: a 4,600-message session blocks the proxy and the web UI for 7–50 s per turn #4528/0.1.212: per-turn ingest is still O(history) — 96.5% of proxy CPU in StatementSync.run, event loop starves with 8 concurrent sessions (#4528 / #5469 follow-up) #5744/[Bug][dsh] Proxy OOMs by self-hitting the 8 GiB heap in ~30-90 min: resident pool budget (512 MiB) is smaller than a single session, so every round thrashes cold-load/evict and allocates giant temporaries #5782/[Bug][dsh] Plugin lane performs a full re-parse and resends the entire body every round (81 MiB → hundreds of MB AST), w | [Bug][dsh] plugin 车道每轮全量解析重发 body(81 MiB→数百 MB AST)是 #5403 OOM 的生产者:用水位线断言替换逐条复核,服务端只解析新增尾段 #5418/[Bug][dsh] 20k-message / 81 MiB-inbound session: host aborted 6× in 2 h — the #3050 resident-AST OOM is not bounded by sessions.hardMaxBytes (Windows, 0.1.210) #5403/[feat][dsh] Long sessions have no convergence path: compaction gated off on both sides, incremental lane full-resends on OpenAI wire | 长会话没有收敛通道:compaction 双向封锁 + 增量车道全量重发 #5796。
    6. 影响面:代码路径 src/storage/store.ts(ingestSession upsert)、src/storage/canonical.ts、常驻池(src/session.ts)、dsh-native 前置钩子(已修);用户面=统一存储长会话用户(每轮延迟+事件循环饥饿波及同进程全部会话)、dsh 进程内用户(修复前 host OOM)、单会话>512 MiB 的多会话用户(Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 前)。
    7. 测试:已合 PR 各自带回归测试([PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815 fastpath 回归、[PR] fix(storage): refs ledger diffs instead of full-replace per save (#5744) #5755 delta 车道、Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 default-cap pin 27c3256 等);剩余 PR 钉两个行为——① delta save 且大列未变时 WAL 无增长/列未被重写(COALESCE 路径断言);② 记忆化命中返回字节一致的 canonical bytes(含内容变化必须重算的失效路径)。
    8. 风险:① 判变哈希 bug 会留陈旧列(变了判成没变)→存储态与内存态分叉,缓解=哈希「将要写入的精确字节」、哈希缺失回退全量写;② 记忆化缓存必须按字节设上限,否则重引入本单要修的不受控增长;③ 写腿修完后非 fastpath 安装主导成本移到读侧 O(n) pass(0.1.212 still reproduces the structural issue from #4528: single-request local preprocessing peaks at 144.7s; conversati | 0.1.212 仍复现 #4528 的结构性问题:单请求本地预处理最高 144.7s,关闭插件后对话明显变快 #5869 残留),正是 A/B/C 裁决要拍的;④ 4 GiB 默认(Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779)让所有安装稳态 RSS 可能上升(release notes 已披露),小内存+多会话驻留更多状态,单会话仍受 hard cap([Performance] v0.1.208 long sessions exceed 2 GiB memory and spend 14–22 seconds in local processing #4788)约束。

    判定:根因归类已交付(floor 11),修复 PR 为下一步动作,三项决策等你拍板。

  5. changed the title [-][Root Cause] performance-memory (7 items)[/-] [+][Root Cause] performance-memory (9 items)[/+] on Nov 13, 2026
  6. ranxianglei commented on Nov 13, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    进度汇报(本会话因 34 分钟无输出被强制重启——卡住原因是我做了大文件整读,已改为小批量定点读,避开该操作)。

    已完成:

    1. 根因归类 + 8 问判定已发(floor 11)——7 项:5 项已修已合(SHA 为证)、[PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846 在途待合、[Bug][storage] CPU profile of v0.1.213 slow-ingest: canonicalJson + serialize + canonicalDigest dominate the delta save path #6254 + Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325 = 剩余同机制 2 子缺陷。
    2. 修复分支已建:2026-11-13_save-path-incremental(基于最新 master 243709b)。
    3. 代码研读完成(小批量定点读,非整读):
      • src/storage/store.ts:1047-1050 sessions upsert 无条件覆盖 4 个大列(已证实);
      • src/storage/ingest.ts:500-505 finishPlan 每次 save(含 delta 路径)都重序列化全部大列;
      • src/storage/shadow.ts:449 record() 主路径 + :547 采样校验(digest 只在采样时算,非每 save)。

    进行中:实现修复——按列判变:JS 侧对每个大列缓存上次写入字节,未变则 upsert 用 CASE WHEN @changed THEN excluded.col ELSE sessions.col END 保留原值(不用 COALESCE,因 ccr_json 合法值可为 null)→ 单轮写入 13 MB → ≈3.5 MB,零语义变更。

    下一步:写代码 → 回归测试(未变列不重写 + 漂移时强制重写保自动愈合)→ typecheck + 定向测试 → push → 开 PR(Fixes #6254 + #6325)→ 回本线程。

    8 问(当前状态版,结论同 floor 11 未变):

    1. 这是啥问题:长会话下 bili 每轮成本随会话状态总量而非增量增长——每轮重写 13 MB 账本 + 重做全部规范化/摘要 + 同步占死单事件循环,同进程所有会话与 Web UI 一起被拖;旧 512 MiB 预算还会「每轮驱逐 → 3.4 s 冷重载」抖动;dsh 进程内会话 host OOM。触发:统一存储 + 长会话 / 单会话 >512 MiB(修复前)/ dsh native + >3.8 MiB(修复前)。
    2. 根本原因是啥:save 路径对全量状态重做工作而非增量——src/storage/store.ts:1047-1050 upsert 无条件覆盖 4 大列;src/storage/ingest.ts:500-505 每次 save 重序列化全部大列;旧 512 MiB 默认预算(Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 前)< 单会话权重;以上全同步在主循环。
    3. 如何修复:7 个 PR 已合(见 floor 11 表);[PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846 在途;剩余 2 子缺陷一个 PR:按列判变 + 规范化记忆化(零语义变更,落盘字节不变只减工作量)。
    4. 能否合并/关闭:[PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846 可合(CI 绿);剩余 PR 可开、CI 绿后可合;5 个已修成员等你验证后统一关;decision request: resident-pool default budget — keep 512 MiB, raise to 2 GiB, or scale with physical RAM (+ two related pending calls) | 决策请求:常驻池默认预算维持/上调/按物理内存自适应(附两项关联待决) #5852 待 D2/D3 裁决。
    5. URL:成员 [perf][dsh] 25.8-32.7 s of each step elapses before bili sees the body (bili itself: 5.5-7.2 s) | TTFT 归因:每步 25.8-32.7 s 发生在 bili 收到请求体之前 #5792/[perf][pool] Default residency budget of 512 MiB causes a 33 MB session to be evicted every round alongside a 578 MB lar | [perf][pool] 默认驻留预算 512 MiB 让 33 MB 会话在 578 MB 大会话旁每轮被驱逐(每次重载 3.4 s,全在事件循环上) #5806/decision request: resident-pool default budget — keep 512 MiB, raise to 2 GiB, or scale with physical RAM (+ two related pending calls) | 决策请求:常驻池默认预算维持/上调/按物理内存自适应(附两项关联待决) #5852/[Bug][dsh] 0.1.212 / Windows: host exit 134 (2.2 GB V8 heap OOM) on a 3.8 MiB long session — likely #5811 + #5813 + #5418 combined | 3.8 MiB 长会话触发 host OOM #5858/0.1.212 still reproduces the structural issue from #4528: single-request local preprocessing peaks at 144.7s; conversati | 0.1.212 仍复现 #4528 的结构性问题:单请求本地预处理最高 144.7s,关闭插件后对话明显变快 #5869/[Bug][storage] CPU profile of v0.1.213 slow-ingest: canonicalJson + serialize + canonicalDigest dominate the delta save path #6254/Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325;已合 [PR] fix(dsh): parse the pre-send request body once per request #5798/[PR] fix(dsh): size authored-compaction usage from the forged checkpoint (#5811) #5814/[PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815/[PR] fix: stop resident-pool evict/cold-load thrash that OOMs the proxy (#5782) #5783/Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779/[PR] fix(storage): refs ledger diffs instead of full-replace per save (#5744) #5755/[PR] fix(#4528): yield to the event loop during long turns; warn on slow sessions #4544;在途 [PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846;祖先 Long sessions starve the event loop: a 4,600-message session blocks the proxy and the web UI for 7–50 s per turn #4528/0.1.212: per-turn ingest is still O(history) — 96.5% of proxy CPU in StatementSync.run, event loop starves with 8 concurrent sessions (#4528 / #5469 follow-up) #5744/[Bug][dsh] Proxy OOMs by self-hitting the 8 GiB heap in ~30-90 min: resident pool budget (512 MiB) is smaller than a single session, so every round thrashes cold-load/evict and allocates giant temporaries #5782/[Bug][dsh] Plugin lane performs a full re-parse and resends the entire body every round (81 MiB → hundreds of MB AST), w | [Bug][dsh] plugin 车道每轮全量解析重发 body(81 MiB→数百 MB AST)是 #5403 OOM 的生产者:用水位线断言替换逐条复核,服务端只解析新增尾段 #5418/[Bug][dsh] 20k-message / 81 MiB-inbound session: host aborted 6× in 2 h — the #3050 resident-AST OOM is not bounded by sessions.hardMaxBytes (Windows, 0.1.210) #5403/[feat][dsh] Long sessions have no convergence path: compaction gated off on both sides, incremental lane full-resends on OpenAI wire | 长会话没有收敛通道:compaction 双向封锁 + 增量车道全量重发 #5796。
    6. 影响面:src/storage/store.ts / src/storage/ingest.ts / src/storage/canonical.ts / 常驻池 / dsh-native 前置钩子(已修);用户 = 统一存储长会话用户、dsh 进程内用户、单会话 >512 MiB 多会话用户。
    7. 测试:已合 PR 各自带回归测试;剩余 PR 钉「未变列不重写」+「漂移检测后强制重写保自动愈合」+「记忆化命中字节一致」。
    8. 风险:判变哈希 bug 留陈旧列(缓解:哈希将写入的精确字节 + 无缓存回退全量写 + 漂移时清缓存强制重写);记忆化缓存须按字节上限;写腿修完主导成本移到读侧 O(n) pass(A/B/C 裁决要拍);4 GiB 默认升稳态 RSS(已披露)。
  7. ranxianglei commented on Nov 13, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    根本修复 PR 已开:#6538

    做了什么:ingestSession 的 sessions 行 upsert 现在按列判变——JS 侧对 4 个大 JSON 列(metadata_json / kstate_json / tail_json / ccr_json)各缓存上次写入字节的 SHA-256;新字节与上次一致时,ON CONFLICT DO UPDATE 腿走 CASE WHEN @colChanged 保留存储值,不重写。新会话的 INSERT 腿不受影响;进程重启(无缓存条目)或漂移自愈(clearLastWrittenColumns,已接进 shadow.ts 采样校验的 drift 路径)保守全量写。

    零语义变更:落盘字节不变,只减写入工作量。稳态轮(只追加消息)现在 13 MB 大列写 ≈0(追加的 refs 落独立的 refs 表,#5755 已做 O(diff))。

    本地验证(定向,非全量):

    • npm run typecheck 干净;
    • 新增回归测试 tests/issue6325-column-change-detection.test.ts 4/4(首存全写 / 稳态跳写 / 突变检测 / 漂移重武装);
    • 相关既有测试 issue5744-ingest-diff 5/5、issue5794-ingest-cap + unified-remap-collision 6/6 全绿。
    • 全量回归 = 远程 CI(PR 已推,等 CI 绿)。

    PR 描述已逐个列 7 个成员号:#6325 用 Fixes(本 PR 真修);#6254 标注「本 PR 去掉写入成本面(主导成本),规范化记忆化(计算成本面)在本聚合单跟踪」;#5792/#5806/#5858/#5869 标注「已由 <已合 PR> 修 + SHA」;#5852 标注「D1 已由 #5779 解决,D2/D3 待裁决」。

    结构化八问

    1. 这是啥问题:长会话(数万条消息、inbound >3.8 MiB)下 bili 每轮 save 无条件整行覆盖 4 个大 JSON 列(实测 13 MB),哪怕 delta 只加几条消息——每轮 687–752 ms 重写 13 MB overflow 页进 WAL,且同步占死单事件循环。触发条件:统一存储 + 长会话(>数千条消息)。
    2. 根本原因是啥:src/storage/store.ts:1047-1050 sessions upsert 的 ON CONFLICT DO UPDATE 腿无条件 metadata_json = excluded.metadata_json 等 4 列覆盖,delta/full 共用同一 upsert,无按列判变。
    3. 如何修复:按列判变——JS 侧缓存上次写入字节的 SHA-256(lastWrittenColumns Map,128 B/session),未变列的 UPDATE 腿走 CASE WHEN @colChanged 保留存储值;新会话 INSERT 腿不受影响;进程重启/漂移自愈保守全量写。零语义变更(落盘字节不变)。
    4. 能否合并/关闭:PR [PR] fix: skip byte-identical large columns in the sessions-row upsert (#6325) #6538 可合——typecheck 干净、定向测试全绿(新回归 4/4 + 相关既有 11/11)、零语义变更、单模块(src/storage/)、纯 fix:、干净 diff(67 行 + 测试文件)。全量回归等 CI 绿。Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325 在 PR 合入后由你验证关闭。
    5. URL:PR [PR] fix: skip byte-identical large columns in the sessions-row upsert (#6325) #6538 ;成员 GitHub Performance issue: saving a sub-2 KB config snapshot and a few new messages, yet rewriting 13 MB every round. | 性能问题:保存不到 2 KB 的配置快照和几条新消息,每轮重写 13 MB #6325(本 PR 修)/ [Bug][storage] CPU profile of v0.1.213 slow-ingest: canonicalJson + serialize + canonicalDigest dominate the delta save path #6254(follow-up)/ [perf][dsh] 25.8-32.7 s of each step elapses before bili sees the body (bili itself: 5.5-7.2 s) | TTFT 归因:每步 25.8-32.7 s 发生在 bili 收到请求体之前 #5792 / [perf][pool] Default residency budget of 512 MiB causes a 33 MB session to be evicted every round alongside a 578 MB lar | [perf][pool] 默认驻留预算 512 MiB 让 33 MB 会话在 578 MB 大会话旁每轮被驱逐(每次重载 3.4 s,全在事件循环上) #5806 / decision request: resident-pool default budget — keep 512 MiB, raise to 2 GiB, or scale with physical RAM (+ two related pending calls) | 决策请求:常驻池默认预算维持/上调/按物理内存自适应(附两项关联待决) #5852 / [Bug][dsh] 0.1.212 / Windows: host exit 134 (2.2 GB V8 heap OOM) on a 3.8 MiB long session — likely #5811 + #5813 + #5418 combined | 3.8 MiB 长会话触发 host OOM #5858 / 0.1.212 still reproduces the structural issue from #4528: single-request local preprocessing peaks at 144.7s; conversati | 0.1.212 仍复现 #4528 的结构性问题:单请求本地预处理最高 144.7s,关闭插件后对话明显变快 #5869;已合 PR [PR] fix(dsh): parse the pre-send request body once per request #5798 / [PR] fix(dsh): size authored-compaction usage from the forged checkpoint (#5811) #5814 / [PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815 / [PR] fix: stop resident-pool evict/cold-load thrash that OOMs the proxy (#5782) #5783 / Corrupt file .corrupt – timestamp blows up the disk | 坏文件.corrupt﹣时间戳撑爆磁盘 #5779 / [PR] fix(storage): refs ledger diffs instead of full-replace per save (#5744) #5755 / [PR] fix(#4528): yield to the event loop during long turns; warn on slow sessions #4544;在途 [PR] Fix #5806: Stop evicting recently used sessions on low memory, and surface the pool's over-limit state | 修 #5806:内存不够时别再踢掉刚用过的会话,并把池子超限状态显性化 #5846。
    6. 影响面:代码路径 src/storage/store.ts(ingestSession upsert)+ src/storage/shadow.ts(drift 路径接 clearLastWrittenColumns);用户面 = 统一存储长会话用户(每轮 save 延迟从 687–752 ms 降到近 0,事件循环占用同步下降,同进程其他会话 + Web UI 受益)。
    7. 测试:tests/issue6325-column-change-detection.test.ts 钉 4 个行为——① 首存全写(无缓存条目 → flags=1);② 稳态跳写(字节一致 → flags=0,存储字节不变);③ 突变检测(列变了 → flags=1,新字节落盘);④ 漂移重武装(clearLastWrittenColumns → 下次全量写)。
    8. 风险:① 判变哈希 bug 会留陈旧列(变了但判成没变)→ 存储态与内存态分叉;缓解 = 哈希「将要写入的精确字节」、无缓存条目时回退全量写、drift 路径清缓存重武装(采样校验兜底);② 外部写入者(两实例共享 sessions 目录,无单实例守卫:多个 bili 代理进程并存共写 sessions 目录,伴随孤儿 ref、有注册表无 transcript、计数自相矛盾的污染簇 #394 反模式)→ 内存 Map 不知道外部变更 → 可能跳过必要写入;缓解 = 采样校验的 drift 检测会大声告警 + drift-clear 重武装;③ 内存占用 = 128 B/session(4 个 32 字节哈希),有界(随会话数线性,不随会话大小)。

    判定:PR #6538 已开,等 CI 绿。三项仍等你拍板:① 合 #5846(驱逐豁免);② 读侧方向 A/B/C(建议 A);③ #5852 D2(建议不做)/ D3(建议 C2)。CI 绿后我会在本线程回帖确认。

  8. changed the title [-][Root Cause] performance-memory (9 items)[/-] [+][Root Cause] performance-memory (11 items)[/+] on Nov 13, 2026
  9. changed the title [-][Root Cause] performance-memory (11 items)[/-] [+][Root Cause] performance-memory (9 items)[/+] on Nov 14, 2026
  10. ranxianglei commented on Nov 14, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] (ai: vllm-qwen/qwen3.8-27b)

    协调说明(自 #6630 线程):#6630(compress loop 丢弃的 upstream 响应体未关闭)于 2026-11-13 22:50Z 归入本聚合单,但 #6538 开于 12:38Z(归入之前),不含该修复;本线程最后活动(12:46Z)也早于归入——吸收步骤此前无人认领。

    已按本单任务指令(「若某个 issue 已有自己的 PR 且方向一致,cherry-pick 其 commit 并保留原作者署名(-x)」)执行:

    #6538 分支现有新 commits b08b488 / 0cd7925,如该 PR 有其他在途工作请知悉。

  11. 6 remaining items

  12. ranxianglei commented on Nov 16, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    #5507 线程新生产数据点(0.1.215 / Linux / DSH Web 原生插件,3.8 GiB VPS,报告者 @bsbofmusic),用于区分三腿:

    1. autoFold=true + compress.externalSummary(独立摘要模型)确认开启后仍复现,且无证据请求进入独立摘要调用阶段 → 阻塞不在摘要调用路径,在更早的主线程同步段;
    2. 时序:05:23:20 cold-loaded: 8786 refs, 123 blocks (76 active), baseline=415649 tok, compressions=123 → 05:23:45 软告警(32s)→ 无该请求完成行 → 05:26 health 超时;
    3. SIGTERM 无响应(处理器排在阻塞的主循环后)→ systemd stop 超时 SIGKILL——同步阻塞的独立确认;
    4. 恢复对照:避开故障会话后 41 条请求、local 中位 63 ms(22–446 ms)、无新 FATAL → 阻塞绑定该会话状态(8786 refs / 11.74 MiB 会话文件 / index.db 838 MiB);
    5. 与 [Bug][reconcile] 0.1.214: fold reconcile does anchors x candidates text work; a resumed session blocks the event loop for over 300 s #6800 条件的差异:故障会话 76 个 active blocks([Bug][reconcile] 0.1.214: fold reconcile does anchors x candidates text work; a resumed session blocks the event loop for over 300 s #6800 为全部 inactive)→ 不能直接断言 reconcile 腿;profile 将区分写腿([Bug][storage] CPU profile of v0.1.213 slow-ingest: canonicalJson + serialize + canonicalDigest dominate the delta save path #6254/[PR] fix: skip byte-identical large columns in the sessions-row upsert (#6325) #6538)vs reconcile 腿([Bug][reconcile] 0.1.214: fold reconcile does anchors x candidates text work; a resumed session blocks the event loop for over 300 s #6800/[PR] fix(reconcile): compute candidate word evidence once per pass (#6800) #6801)vs 冷加载腿。

    profiling 配方已在 #5507 线程给出(NODE_OPTIONS 继承 + 软告警时 SIGTERM 落盘;perf 外部采样对 SIGKILL 安全)。详见 #5507。

  13. changed the title [-][Root Cause] performance-memory (12 items)[/-] [+][Root Cause] performance-memory (11 items)[/+] on Nov 18, 2026
  14. changed the title [-][Root Cause] performance-memory (11 items)[/-] [+][Root Cause] performance-memory (13 items)[/+] on Nov 18, 2026
  15. changed the title [-][Root Cause] performance-memory (13 items)[/-] [+][Root Cause] performance-memory (15 items)[/+] on Nov 18, 2026
  16. ranxianglei commented on Nov 19, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    交叉链接:本家族新复现 GitHub #7274(镜像 #7274,OMP 19.0.0 / 0.1.216 / Linux WSL,报告者 @cy77cc)——OMP 并发多 agent 会话共享一个 bili 代理 + 大会话索引(293 MiB + 21 MiB WAL)触发同一条链:[storage] slow-ingest(path=delta took=136771ms)→ event-loop-watchdog FATAL 304s(硬阈值 300s)→ 代理死亡 → 所有 OMP 会话 502。

    与 DSH 复现(#5507)的差异:OMP 并发 agent 是负载模式(多会话共享单代理,save 在单事件循环上串行化),不是新根因——根因仍是本聚合的「同步存储 ingest 成本随总会话状态增长」。OMP 侧 502 恢复子面(插件从 model config 取固定 origin、不重解析健康替代,respawn 归 launcher/host)归 process-lifecycle 家族(#5507/#6897)跟踪,不在本单。

    已在 #7274 打 duplicate 标签(不关闭,关闭权在 owner)。

  17. changed the title [-][Root Cause] performance-memory (15 items)[/-] [+][Root Cause] performance-memory (11 items)[/+] on Nov 19, 2026
  18. changed the title [-][Root Cause] performance-memory (11 items)[/-] [+][Root Cause] performance-memory (15 items)[/+] on Nov 19, 2026
  19. changed the title [-][Root Cause] performance-memory (15 items)[/-] [+][Root Cause] performance-memory (19 items)[/+] on Nov 19, 2026
  20. changed the title [-][Root Cause] performance-memory (19 items)[/-] [+][Root Cause] performance-memory (15 items)[/+] on Nov 20, 2026
  21. wstczyw commented on Nov 20, 2026

    @wstczyw

    Data point for this family on a newer build, plus one downstream effect.

    Setup: 0.0.1239-master.1e357bf, Windows 11, bili start on 8787 (unified storage, full mode). index.db is about 1.05 GiB, and the WAL peaked at 75.9 MiB. Claude Code and Codex sessions go through this proxy.

    slow-ingest on 11-20, 00:00–14:20 local:

    warnings 1193
    path=full / path=delta 1017 (85%) / 176
    duration p50 1.36 s, p90 3.13 s, max 14.2 s
    1–2 s / 2–5 s / ≥ 5 s 500 / 333 / 18

    The slowest two were path=full took=14245ms, 21.3 MiB WAL and path=full took=13396ms, 20.2 MiB WAL. No event-loop-watchdog kill: no stall came near 5 min.

    Question: is 85% path=full expected on this build, after #6349 and #6356? The log line says only "watermark missing/cleared or tail shape changed". If the cause is known, a counter per cause would help to tell which one fires.

    Downstream effect: these stalls also block /__bili/health, because it is answered on the same event loop. magpie runs bili start for its /bili/ lanes. When two 2 s health checks miss in a row, it ends its own bili. It also ends a new one that does not answer within 20 s, and opening a 1 GiB store takes longer than that. So normal load turns into restarts that drop requests in flight. Filed with a patch: yetone/magpie#7309. bili's own watchdog header already says the right rule ("deadlock detection, not load detection"). Any other supervisor that probes health with a short timeout will do the same thing.

    Also on every start (maybe separate): [session-order] hydrate failed — skeletons stay on the legacy file path for this process ([storage] extension value exceeds 4194304 bytes — store large payloads in the content CAS instead) with 179 skeletons.

  22. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    Good data point — three distinct items in one comment, let me separate them.

    1. 85% path=full — is it expected after #6349/#6356?

    Short answer: not expected for plain sessions, but expected for sessions that carry a pluginSnapshot. The delta fast path (planIngestDelta, src/storage/ingest.ts:318) returns null → full path when session.pluginSnapshot.length > 0, by design ("its entries may dedup against the trusted prefix and renumber the log"). Fork sessions and certain plugin sessions set pluginSnapshot (src/plugin.ts:1871, :639, :683). If your Claude Code / Codex sessions are fork sessions or carry a snapshot, every save takes the full path — that alone explains 85%.

    Other fallback causes (each returns null → full): watermark missing/cleared (DIGEST DRIFT), tail shorter than trusted prefix (fold), boundary identity mismatch (history edit), head-slot id mismatch (prefix rewrite), decompositionLog present (import). #6349/#6356 fixed two specific DIGEST DRIFT producers, but other causes remain.

    Per-cause counter: agreed, this is the right diagnostic. I'll add a counter per fallback reason in ingestDelta/planIngestDelta so the log line can say which cause fired, not just "watermark missing/cleared or tail shape changed". This is a small additive change (no behavior change) — I'll land it as a follow-up commit on the #6538 branch or a separate small PR.

    2. Health-check stall → magpie kills bili

    This is a downstream effect of the same mechanism (synchronous ingest on the single event loop). My PR #6538 reduces the write cost (skips byte-identical columns) but does not eliminate the synchronous processing — a 14 s full ingest still blocks /__bili/health for 14 s. The structural fix (health check on a separate mechanism, or async ingest) is a bigger architectural change and out of scope for this PR. Your magpie patch (yetone/magpie#7309) is the right workaround on the supervisor side. I'll note the bili-side structural fix as a separate item in this thread.

    3. [session-order] hydrate failed — extension value exceeds 4194304 bytes

    This is a separate bug from the save-path family: the session-order skeleton is stored as a SQLite extension value (user data) and exceeds the 4 MiB limit. The error message itself says "store large payloads in the content CAS instead" — the fix is to route the skeleton through the content CAS when it exceeds the inline limit. 179 skeletons affected. This is a distinct symptom + mechanism (storage size limit, not ingest performance) — I'll file it as a separate issue per the one-issue-one-scope rule.

    PR #6538 CI status: 1 failure — test (windows-latest, 22, 5): failure. 29/30 green. Investigating the Windows shard failure now; will report back.

  23. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    PR #6538 CI follow-up: the single failure (test windows-latest 22 shard 5, exit code 1) has been re-triggered via rerun-failed-jobs. All other 29 checks are green (1 skipped). The failure is isolated to one Windows shard while the other four Windows shards (22, 1/2/3/4) pass — pattern consistent with a flaky/environment-specific test rather than a regression from the per-column change-detection change (which is storage-layer and covered by the green issue6325-column-change-detection + issue5744-ingest-diff + issue5794-ingest-cap suites). Will report the re-run result here.

  24. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    Filed the session-order hydrate failure as a separate issue: #7378 (distinct symptom + mechanism from the save-path family — SQLite extension value 4 MiB limit, not ingest performance).

    CI re-run for PR #6538 (Windows shard 5) still in progress — will report the result here when it completes.

  25. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    PR #6538 CI — root cause of the Windows shard-5 failure identified. Fetched the full job log (2.6 MB). The failure is not a regression from the per-column change-detection change. It's the known Windows teardown-lock issue (#4347):

    [ci-tests] full retry also FAILED (exit=1) -> retry on the SAME runner reproduced
    13 teardown-lock failure(s) -> either a persistent external lock (AV/indexer) on
    this runner or a DETERMINISTIC handle leak in the code; rerunning the job draws
    a fresh runner, and if the lock reproduces there too the leaked-handle
    hypothesis is primary (#4347)
    

    Test totals: 2374 tests, 2341 pass, 13 fail (all teardown-lock), 20 skipped. The 13 failures are Windows file-locking artifacts (AV/indexer holding SQLite handles during teardown), not assertion failures. All storage-layer tests (issue6325-column-change-detection, issue5744-ingest-diff, issue5794-ingest-cap) pass.

    Triggered a second re-run (draws a fresh runner per the CI retry policy). Will report the result here.

  26. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    PR #6538 CI — second re-run (fresh runner) also failed. Same 13 teardown-lock failures. The CI log's own diagnosis: "if the lock reproduces there too the leaked-handle hypothesis is primary (#4347)".

    Key facts:

    Assessment: the 13 teardown-lock failures are the known #4347 Windows file-locking issue, not a regression from this PR. The PR is otherwise fully green (29/30 checks). Flagging for your judgment: merge with the known #4347 Windows shard exception, or hold until #4347 is resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions