Skip to content

[PR] fix: bili export <id> no longer decodes every persisted session (OOM on multi-session machines) - #5084

Open
ranxianglei wants to merge 13 commits into
masterfrom
2026-10-29_export-single-session-no-loadall
Open

ranxianglei wants to merge 13 commits into
masterfrom
2026-10-29_export-single-session-no-loadall

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #5062

What & why

bili export <id|label> started with a full loadAll() walk, decoding EVERY persisted session before it could render ONE. On a machine holding several long-lived sessions that OOMed the V8 heap (exit 134) before the documented bad-session → handoff workaround ran (#5026: single 1184-request session, ~138.5M input tokens, 8GB machine).

Selection now resolves from the unified summary rows (one SQL — the same authority listSessions uses, #2960 R2.1; id + label is all matching needs) and hydrates only the winner via the existing loadSync path (web panel / fork adoption). Fallbacks keep historical states byte-for-byte:

Behavior changes (disclosed)

  1. Unified (full) mode, selector: resolution authority moves from "decoded file set" to "unified index rows". A session whose JSON file exists but whose row the index never recorded (sidecar-degraded write) becomes unselectable — consistent with listSessions, which already served rows as authority in full mode (it was never LISTED either). Healthy full-mode stores are unaffected (organic import on every load/write keeps rows whole).
  2. Unified mode, performance: exporting or listing one session no longer decodes the rest of the store (the fix itself); rendered output for healthy sessions is unchanged.
  3. mysql (network backend) + full mode: unifiedSessionSummaries() now returns null (unavailable → file walk) instead of [] (read as "zero sessions"), which blanked the session list on healthy mysql installs. Second commit — same-mechanism side finding from the Exporting ONE session decodes EVERY persisted session (loadAll) — multi-long-session machines OOM the documented handoff workaround | 导出单个会话会解码存储里的全部会话(loadAll)——多长寿会话机器上官方交接手段 OOM 崩溃 #5062 analysis.
  4. New stderr notice (additive log line) when a selected row will not decode and the export falls back to the full scan.

Legacy/shadow mode: byte-for-byte unchanged (pinned by regression test).

Tests

New tests/issue5062-export-single-session.test.ts (both gate directions, #3617):

  • full mode: select by id/label + listing succeed AND loadAll provably never walked (prototype spy, 0 calls);
  • error surface byte-parity (no-match, ambiguous-prefix) resolved from rows alone;
  • fresh full-mode store (index exists, zero rows) still reports the exact empty-store message via the walk;
  • legacy mode: walk still taken (exactly 1 call), one corrupt envelope cannot block the healthy ones;
  • async-backend guard: null not [] (and sync backends keep serving their rows).

Local targeted runs green: new file 6/6, existing export suites (cli-export, export-jsonl, export-block-summaries, export-uncompressed) 16/16, typecheck clean. Full-suite verdict = CI on this head.

No config surface changes. No wire/message-shape changes. Kernel untouched.

ci-check added 2 commits October 29, 2026 09:37
…f decoding every persisted session (#5062)

bili export <id> started with a full loadAll() walk regardless of selector,
decoding EVERY persisted session before it could render one. On a machine
holding several long-lived sessions that OOMed the V8 heap (exit 134) before
the documented bad-session -> handoff workaround ran (#5026).

Selection now resolves from the unified summary rows (one SQL - the same
authority listSessions uses, #2960 R2.1; id + label is all matching needs)
and hydrates only the winner via the existing loadSync path. Rows absent
(legacy/shadow mode, degraded sidecar), empty (fresh mode flip, deleted
index.db), or a winner that will not decode fall through to the historical
full walk, which keeps those states byte-for-byte. Error surfaces
(empty-store message, no-match, ambiguous-prefix) are unchanged.
…instead of an empty row list (#5062)

Same-mechanism side finding from the #5062 analysis: on a mysql (network)
backend the sync facade's sessionSummaries() miss value is [], which callers
read as 'zero sessions' - listSessions blanked the session list on healthy
mysql + full installs. The contract is null = unavailable = file walk, the
same guard webSummaryBaseline already carries for this surface.
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-29_export-single-session-no-loadall (2dfda0f)

Latest build — install this exact version

npm install -g billion-context@0.0.9404-pr.5084.092774f

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
2dfda0f billion-context@0.0.9404-pr.5084.092774f build
4c170d9 billion-context@0.0.7407-pr.5084.b28a34d build
3c9cf58 billion-context@0.0.7286-pr.5084.01c9560 build
8105d74 billion-context@0.0.6856-pr.5084.0d82482 build
86d9d53 billion-context@0.0.6816-pr.5084.96f6418 build
929ce8d billion-context@0.0.6691-pr.5084.2a3d6ff build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5084.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: bili export <id> no longer decodes every persisted session (OOM on multi-session machines) [PR] fix: bili export <id> no longer decodes every persisted session (OOM on multi-session machines) Oct 29, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。全仓检索(open+closed,issue 与 PR,按失败签名「export OOM / loadAll 全量解码」)无其他条目;本 PR 是 #5062 的唯一在途修复(#5062 自身查重亦为「无」)。

标签:bug dsh scope/storage p2——缺陷类;受影响机器为 dsh 宿主(#5062/#5026 事故机);存储/导出路径;有绕行(#5026 handoff 工作流)且触发条件为多长会话机器,与 #5062 分诊一致定 p2。

PR #5084 审核进行中:已通读 diff(src/export.ts、src/persist.ts、新回归测试),正在本地跑定向测试 + 读镜像 CI 状态,结论按八问体例随后给出。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

PR #5084 审核完成。独立核验(未复用 issue 侧结论):对真实 merge base(5594c1315)取 diff,恰好 3 个文件——src/export.ts、src/persist.ts(+5)、新测试,无无关改动;本地定向测试 22/22 过(新文件 6 + 既有导出套件 cli-export / export-jsonl / export-block-summaries / export-uncompressed 共 16);镜像 CI 在 head 929ce8d7d 上 30/30 completed、0 failed(ubuntu/windows × node22/24 全分片、static-gates、mysql-contract、gate 全绿,gated e2e 车道按文件清单正常跳过)。

关键声明逐条回源码验证过:

  • 选择器只需 id+label:内核 matchSession(kernel/src/handoff.ts:117)确实只消费这两个维度,行摘要候选足够;
  • 「命中行但解不开 → stderr 提示 + 全量扫描」成立:full 模式下 loadSync 的索引读走 shadow.viewFor(src/storage/shadow.ts:518),整体 try/catch、任何错误返回 null 不抛,随后无信封即 null,导出侧兜底逻辑正好接住;
  • 第二 commit 的 isAsyncBackend → null 守卫与既有 webSummaryBaseline 守卫(Storage backend seam (step 1): async operation-level contract + sqlite re-wired through it — MySQL stays a registered stub (#3388) #3931 step 3)同形同机制;unifiedSessionSummaries() 全仓仅两个调用方(listSessions 与本次新增的 exportSession),两者都把 null 当「不可用 → 文件走查」处理,同步后端(sqlite)照常出行,mysql 从「空列表」回到走查——修复而非破坏;
  • 三条用户可见文案(空库提示 / no-match / 前缀歧义)新旧逐字节一致(含新抽出的 renderSessionListing 行映射)。

结构化八问

  1. 是什么问题:按 id/label 导出单个会话时,bili 先把所有已持久化会话全部解码一遍才能挑出一个;多长会话机器上直接 OOM(V8 堆爆、exit 134),文档化的坏会话绕行流程还没机会执行。
  2. 什么原因导致的:导出把「解码后的文件全集」当作选择权威——一个只需要 id+label 的元数据查找被实现成全量 loadAll() 走查;而统一索引里本来就有同一权威的单条 SQL(Identity pipeline: key sessions by (agent, conversation id) in the unified store — decouple fold ledgers from protocol/host origin (#2938 / #2847 / #2671 defect-1) #2960 R2.1,与 listSessions 同源)。
  3. 影响范围与严重程度:统一存储 full 模式 + 多长会话机器上 bili export <selector> 进程崩溃、命令不可用;有文档化绕行([Bug][dsh-native] turn fails with bare TypeError (reading 'length') host-side before the request reaches the proxy - very long-lived session, intermittent #5026 handoff),小存储机器无感。p2,与 Exporting ONE session decodes EVERY persisted session (loadAll) — multi-long-session machines OOM the documented handoff workaround | 导出单个会话会解码存储里的全部会话(loadAll)——多长寿会话机器上官方交接手段 OOM 崩溃 #5062 分诊一致。
  4. 怎么修复的:从单条 SQL 的行摘要解析选择器,只对命中的那一个走既有 loadSync 水合;行不可用/为空/命中行解不开时一律退回历史全量走查——降级不猜,错误文案逐字节保持。
  5. 是否有回归问题:审过的既有行为逐项——legacy/shadow 模式导出路径字节级不变(回归测试钉死:走查恰好 1 次、坏信封不挡健康会话);三条错误/列文案字节一致;jsonl/markdown/handoff 渲染路径除「用解析出的 target」外未动;unifiedSessionSummaries 两个调用方均 null-safe;同步后端出行不受守卫影响(测试钉住)。有意变更两处均已披露:① full 模式选择权威从「解码文件集」移到「索引行」——索引没记到行的幽灵文件从此不可选(它本来也从未出现在 listSessions 里,属一致性收敛而非丢失);② mysql full 模式的列出/导出从「显示空」变回文件走查(修复)。无 fail-fast 变吞错、无阈值/时序漂移;新增一行 stderr 提示为纯增量日志。
  6. 是否破坏缓存:不涉及——离线 CLI 导出路径,不碰模型流量、前缀缓存、重试与去重键。
  7. 是否改了设计方向:不涉及——复用既有权威(行摘要 + loadSync),无新数据流。
  8. 是否引入了新的配置:不涉及。

测试有效性(质量关):双方向齐全(#3617)——正向行选择成功且原型 spy 证明 loadAll 零调用;负向空库走查、坏行降级、legacy 走查恰好一次、异步后端 null-not-[] 各有断言咬合点(精确文案正则 + spy 计数),非恒真断言。

判定:可合。CI 全绿于推送 head、本地定向验证通过、diff 收敛到目的本身、未发现回归;两处行为变更均已在 PR 披露。合并按规矩留给人工:#5084

@ranxianglei ranxianglei added bug Something isn't working dsh p2 scope/storage labels Oct 29, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync done: origin/master merged into head twice (master moved again while the first sync sat in the CI runner queue); both merges conflict-free with no overlap on this PR's files; head now 8105d74, local typecheck clean and 6/6 export regression tests green; merge state after push: behind -> blocked (required checks queued on runners, 0 failures so far).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Merge-guard "behind" verdict was stale: the PR base is already the current master tip (09a5b5b7, compare ahead_by=4 / behind_by=0) — nothing to sync, nothing pushed. The blocked merge state came from the main CI run being cancelled ~2 min after start (Oct 29 17:17Z, mid duplicate-trigger storm); I've triggered a full rerun (run 40919523330, attempt 2 now queued). Caveat: master itself is currently red (inherited test flake family, tracked in #4599) — if any shard fails with that signature it is inherited, not from this diff.

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync done (rounds 3+4: master kept advancing while the previous sync sat in the CI runner queue): merged origin/master two more times, both conflict-free (src/persist.ts auto-merged cleanly over master's storage/sidecar rework); head now 97561b9, which contains master tip 7c9e65b, so the branch is no longer behind; local typecheck clean and 6/6 export regression tests green; checks on the new head: 22 queued, 0 failures.

ci-check added 2 commits October 31, 2026 11:50
…-no-loadall' into 2026-10-29_export-single-session-no-loadall
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync done (round 6): a concurrent worker had pushed an earlier-master sync to this branch, so I merged that remote head plus the remaining master commits into my verified head, all conflict-free; head now f464be0 which contains current master tip 491ac1d, so the branch is no longer behind; local typecheck clean and 6/6 export regression tests green on this head; state after push: mergeable, checks computing.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

Synced & pushed (correcting my earlier line, which cited a stale master tip): merged master e5e210865→e19386182 (15 commits: #4313 message-timestamps / unified schema v10 + kernel 0.0.173, #5140 cold-restore test pin) into the branch head — clean merge, no conflicts (src/persist.ts touched by both sides in disjoint regions; export path unaffected by the ts hydration). Local: typecheck clean + tests/issue5062-export-single-session.test.ts 6/6. New head 4c170d9d6; base now == master tip e19386182, so the behind state is cleared — 11 fresh CI runs queued. Note: master has been moving every round today (fleet actively merging); if the guard re-fires it's simply master advancing again. Prior red shards were the two documented inherited flake families (#4982 ladder race, claude-native other-port attach race), zero PR-specific failures.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

Merge conflict resolved and pushed. Here is what I did this run.

Conflict resolution

  • Merged origin/master (2235 commits ahead of the branch) into 2026-10-29_export-single-session-no-loadall.
  • Only src/export.ts conflicted (2 regions); src/persist.ts and the rest auto-merged cleanly.
  • Region 1 (module-level helpers): kept BOTH the PR's new EMPTY_STORE_MESSAGE + renderSessionListing AND master's new renderSelectedSessionJsonl extraction — they are independent and both needed.
  • Region 2 (jsonl render inside exportSession): master extracted the inline jsonl block into renderSelectedSessionJsonl(store, session, full); the PR had reworked session selection to a target variable (replacing matches[0]!). Resolved by calling master's extracted function with the PR's target: renderSelectedSessionJsonl(store, target, opts.full ?? false). The function body is byte-identical to the PR's inline block, so no semantic change.

Verification (local, targeted)

  • npm run typecheck: clean.
  • tests/issue5062-export-single-session.test.ts: 6/6 pass.
  • Existing export suites (cli-export, export-jsonl, export-block-summaries, export-uncompressed): 17/17 pass.

Push

  • Fast-forward push (the remote head was unchanged, so no force-push was needed). New head 2dfda0f181, which contains the current master tip 259cd87674 — the branch is no longer behind.

CI state (mirror cache, verified live this run)

  • The mirror cache was stale on first read (fetched 2026-10-31, old sha 4c170d9d69) and refreshed to the new head within a couple of minutes.
  • On head 2dfda0f181: mergeable_state=blocked — i.e. required checks still in progress, NOT a conflict/behind state. 30 checks: 8 success, 10 in progress, 12 skipped (gated e2e lanes, correctly not triggered by this diff), 0 failures so far.
  • The full-suite verdict is the CI test lanes on this pushed head; per the local-testing discipline I am not chasing it locally.

结构化八问

  1. 是什么问题:PR 与 master 冲突(merge state dirty),且落后 master 2235 个提交,无法合并。
  2. 什么原因导致的:PR 分支自 10-31 后未再同步,master 期间大量推进(含 Request two capabilities: compressed block "retirement" and summary "external rewrite". | 请求两个能力:压缩块「退役」与摘要「外部重写」 #5765 block ledger、Extend the storage layer to support additional backends beyond SQLite (e.g., MySQL, Greenplum), while maintaining extens | 存储层增加支持除了sqlite之外的其他存储,比如mysql,greenplum等,并保持扩展性 #3388 step 2 jsonl 抽取等),在 src/export.ts 上与本 PR 的改动产生文本冲突。
  3. 影响范围与严重程度:仅阻塞本 PR 合并,不影响任何已发布行为;PR 本身修复 Exporting ONE session decodes EVERY persisted session (loadAll) — multi-long-session machines OOM the documented handoff workaround | 导出单个会话会解码存储里的全部会话(loadAll)——多长寿会话机器上官方交接手段 OOM 崩溃 #5062(bili export <id> 单会话 OOM)。
  4. 怎么修复的:把 origin/master 合并进 PR 分支;src/export.ts 两处冲突按「保留双方新增 + 用 master 抽取的 renderSelectedSessionJsonl 接 PR 的 target 选择变量」解决;src/persist.ts 自动合并。
  5. 是否有回归问题:本地 typecheck 干净;PR 回归测试 6/6、既有导出套件 17/17 全绿;冲突解决未改动任何既有语义(只保留双方新增、把 matches[0]! 换成等价的 target)。全量裁决以推送 head 的 CI test 车道为准(当前 0 失败)。
  6. 是否破坏缓存:不涉及——离线 CLI 导出路径,不碰模型流量/前缀缓存/重试/去重键。
  7. 是否改了设计方向:不涉及——纯冲突解决,复用双方既有结构。
  8. 是否引入了新的配置:不涉及。

判定:冲突已解、分支不再 behind、CI 运行中且 0 失败。等 test 车道全绿即可合(合并按规矩留给人工)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] (ai: vllm-qwen/qwen3.8-27b)

进度汇报(进程被强制重启 3 次后恢复):06:31 / 16:04 / 16:44 三次因 35 分钟无输出被重启。本轮已避开长挂起操作(所有网络命令加 timeout),独立审核已完成,结论如下。

PR #5084 独立审核(未复用 issue 侧结论) — 对当前 head 2dfda0f181 逐条回源码核验:

结构化八问

  1. 是什么问题:按 id/label 导出单个会话时,bili 先把所有已持久化会话全部解码一遍才能挑出一个;多长会话机器上 V8 堆 OOM(exit 134),文档化的坏会话绕行流程没机会执行。
  2. 什么原因导致的:导出把「解码后的文件全集」当作选择权威——一个只需 id+label 的元数据查找被实现成全量 loadAll() 走查;而统一索引里本就有同一权威的单条 SQL(Identity pipeline: key sessions by (agent, conversation id) in the unified store — decouple fold ledgers from protocol/host origin (#2938 / #2847 / #2671 defect-1) #2960 R2.1,与 listSessions 同源)。
  3. 影响范围与严重程度:统一存储 full 模式 + 多长会话机器上 bili export <selector> 进程崩溃、命令不可用;有文档化绕行([Bug][dsh-native] turn fails with bare TypeError (reading 'length') host-side before the request reaches the proxy - very long-lived session, intermittent #5026 handoff),小存储机器无感。p2,与 Exporting ONE session decodes EVERY persisted session (loadAll) — multi-long-session machines OOM the documented handoff workaround | 导出单个会话会解码存储里的全部会话(loadAll)——多长寿会话机器上官方交接手段 OOM 崩溃 #5062 分诊一致。
  4. 怎么修复的:从单条 SQL 的行摘要解析选择器,只对命中的那一个走既有 loadSync 水合;行不可用/为空/命中行解不开时一律退回历史全量走查——降级不猜,错误文案逐字节保持。
  5. 是否有回归问题:逐项审过——legacy/shadow 导出路径字节级不变(测试钉死走查恰好 1 次、坏信封不挡健康会话);三条错误/列文案字节一致;unifiedSessionSummaries 两个调用方(listSessions 既有 + exportSession 新增)均 null-safe;isAsyncBackend 守卫对 mysql 是修复(空列表→文件走查)非破坏,sqlite 不受影响;renderSelectedSessionJsonl 冲突解决无语义变更。无 fail-fast 变吞错(坏行走 stderr 提示 + 回退全扫,诚实不吞)、无阈值/时序/输出格式漂移。有意变更两处均已披露:① full 模式选择权威移到索引行(索引没记到行的幽灵文件不可选,与 listSessions 一致);② mysql 列出/导出从「显示空」变回文件走查(修复)。
  6. 是否破坏缓存:不涉及——离线 CLI 导出路径,不碰模型流量/前缀缓存/重试/去重键。
  7. 是否改了设计方向:不涉及——复用既有权威(行摘要 + loadSync),无新数据流。
  8. 是否引入了新的配置:不涉及。

⚠️ 当前状态:PR 落后 master 50 个提交(审核期间 master 推进)

master 从 259cd87674(11-12 22:16Z 同步点)推进到 99c22d283f(11-13 15:46Z),新增 50 提交(announcement policy / notices / BILI_PLUGIN_BYPASS 等)。这 50 提交无一触碰本 PR 的 src/export.ts / src/persist.ts,故 re-sync 应为无冲突干净合并。GitHub 当前 mergeable_state=behind、mergeable=true(无冲突)。上一同步 head 2dfda0f181 的 CI 全绿(30 checks:17 success / 13 skipped / 0 failed,镜像缓存 22:31Z 终态)。

判定:代码层面可合——逻辑正确、既有行为逐项保持、测试有牙、上一同步 head CI 全绿。合并前置:merge-guard 需 re-sync 这 50 个 master 提交(预期无冲突)+ CI 重跑全绿,然后人工合并。re-sync 属 merge-guard 车道(本 issue 历史上多轮同步均由其处理),我不越界代做。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p2 scope/storage

Projects

None yet

1 participant