Skip to content

fix(web): stop /__bili dashboard cold scans from freezing the instance (#3199) - #3214

Merged
ranxianglei merged 3 commits into
masterfrom
2026-10-15_fix-3199-overview-cold-scan
Oct 15, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
2026-10-15_fix-3199-overview-cold-scan

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Closes #3199

Rebased onto current master (c43bdf01a). Note: the issue's suggestion 3a (WAL growth) landed independently via #3250/#3255 — open/close TRUNCATE checkpoints + periodic PASSIVE every 20 ingests + configurable storage.walJournalSizeLimitBytes cap — so this PR no longer carries its own checkpoint code.

What was wrong

On a cold instance with a large session corpus in unified storage (storage.unified=shadow or full), the first /__bili/overview (and /__bili/sessions) hit froze the whole instance for minutes — 78–180 s measured on a 2.3 GB corpus. Two producers, one failure domain (main-thread cold-corpus cost):

  1. List surfaces. Shadow mode deliberately bypassed the index fast path (Identity pipeline: key sessions by (agent, conversation id) in the unified store — decouple fold ledgers from protocol/host origin (#2938 / #2847 / #2671 defect-1) #2960 R2.2: rows may lag unrecorded files, so the JSON tree is the truth base), so the first list hit decoded the entire corpus on the main event loop. Verified still present on latest master (src/persist.ts unifiedWebSummaries() gate !this.shadow?.readsUnified → null; src/web/sessions-data.ts unchanged since v0.1.197).
  2. Boot. bootBounded parsed the entire corpus before server.listen() ran — so even health probes couldn't get through during boot. While the instance was frozen, host-side connection handling migrated traffic to a freshly spawned cold instance, which paid the same scan → freeze → migrate again (the autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044 churn family; each migration also left an orphan behind). Also verified byte-identical on latest master.

What changed (3 commits)

  1. fix(web) — shadow list surfaces now read the index as a baseline and reconcile against the file tree inside refreshIndex: fresh rows are adopted without decoding (file mtime within 5 min of row updated_at; normal writes always land after their row because ingest precedes the JSON flush), files newer than their row or never recorded are decoded individually, ghost rows drop (only files produce entries). Cold-start cost drops from "decode the whole corpus" to one SQL + one stat per file + O(delta) decodes. Full mode stays authoritative (vanished-file sessions remain listed — R2.1 behavior untouched); legacy keeps the plain walk verbatim.
  2. Same commit — boot in unified modes parses only a bounded window of the newest files (mtime desc) before listen; label aliases are seeded synchronously from index rows (new SessionSummaryRow.updatedAt column); remaining alias/canonical bookkeeping finishes in the background (backgroundScanComplete()). Survivor arbitration still uses the original savedAt-desc trim, so the 重启伪造时间戳:buildSession 用 Date.now() 覆盖 lastSeen,面板"最后活动"全员开机时刻、fallback=latest 在 ~245 路平局里随机挂会话 #404 freshness contract holds. Legacy mode keeps the synchronous full parse byte-for-byte.
  3. fix(tests) — small labeled in-passing fix: the pre-existing feat: 支持 search_context 显式查询历史 session #841 cold-load case assumed every top-level entry under the sessions dir is a protocol dir and died with ENOTDIR next to index.db (verified failing on pristine master with the node:sqlite builtin engine). Guarded the descent with isDirectory().
  4. test: — new tests/issue3199-overview-cold-scan.test.ts pins all paths, including a structural assert that decode count equals the delta count only.

Behavior changes (old → new, disclosed per owner policy)

Surface Old New Why
shadow-mode web lists full-corpus decode on first hit (minutes) index baseline + tree delta (milliseconds); may lag bypass-written files by ≤5 min, display-only (detail views always read the file; any activity re-ingests) kills the reported freeze
unified-mode boot readiness listener starts after parsing the WHOLE corpus listener starts after the warm-window parse; label aliases for UNRECORDED files fill post-boot in the background (recorded labels seeded synchronously from the index) kills the churn fuel

Steady-state values are identical — same data sources, same summary pipeline (summaryOf unchanged). No cache/retry/incremental-cursor semantics touched; the disk-index (mtime,size) cache keys are preserved (row-adopted entries carry real file mtime/size).

Config surface

None. Reuses existing knobs (sessions.warmMax / sessions.warmMaxBytes, BILI_SESSION_WARM_MAX[_BYTES]).

kernel-change

N/A — no changes under kernel/.

Modes reasoned

Plugin/proxy wire modes unaffected (admin/web surface only); both compression modes considered — no wire payload, rebuild, or injection path touched.

Tests

New suite pins T1 (shadow reconcile incl. ghost-drop + delta-only decode count), T2 (bounded boot + #404 survivor semantics + background alias completion), T3 (legacy control: synchronous full parse). Targeted local matrix green on the rebased head (131 tests: web-sessions, storage-unified, store, persist-zstd, persist-encryption, bounded-storage(+scale), issue404-restore-timestamps, fix-401-storage, identity-registry, search-context-cross-session); typecheck clean. Full-suite verdict via remote CI on this head.

Out of scope (named follow-ups)

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-15_fix-3199-overview-cold-scan (22dfa12)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3782-pr.3214.8ee359e

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3214.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 本 PR 对应原始 bug issue #3199(冷实例大语料下 /__bili/overview 首扫冻死),与在途其他 PR(#3213 nudge-decision、#3202 byte-budgets、#3210 attach-diag、#3195/#3196 dead-refs)的症状与机制均不同。CI 侧的 shard 红与 #3197(live-suite zone-port 带,待合并)同属一条线,但不归本 PR 管。本 PR 自身范围无重复实现。

先给结论:代码质量好、方向对、修法打在根因上,本 PR 自己的影响面本地全绿;当前 CI test lane 的红是仓库级 live-suite 端口碰撞抖动风暴造成的,不是这个 PR 引起(证据见下)。建议 re-run 失败的 check(抖动常能自愈),或在 #3197 落地后再合。

方向关(① 审原 issue)

读 #3199:冷实例 + unified storage(shadow/full)+ 大语料时,首次 /__bili/overview 在主线程对整个语料做 JSON.decode(2.3GB 时冻结 78–180s),boot(bootBounded)也是解析完整个语料才 listen();冻结再喂给 host 的 respawn churn(#3044 族);放大器是僵尸实例快照挡住 SQLite 自动 checkpoint → index.db 664MB + WAL 2.5GB。作者三招都打到"主线程冷扫描"这个根因,不是把症状藏起来:

  1. shadow 列表改为「index 基线 + 目录树 delta」——新鲜行直接采用不 decode,只有比行更新/从未记录的文件才 decode;
  2. boot 只同步解析 warm window(按 mtime 取最近 maxSessions/maxBytes),其余后台补扫;
  3. sidecar 打开时对 >8MB 的 WAL 做一次 PASSIVE checkpoint 回收。
    疏而非堵,成立。

回归关(② 既有行为逐项清点)

代码关(③)/ 测试关(④)

  • 命名达意、注释讲 why、错误处理完整(try/catch 非吞错)、DRY(parseBootPayload 抽取)、无 as any/@ts-ignore、魔法数字用具名常量(SHADOW_ROW_STALE_WINDOW_MS=5min、WAL_CHECKPOINT_MIN_BYTES=8MB)。整洁度达标。
  • 新测试有效且 hermetic/deterministic:T1 用结构性断言 decodedTotal===2(不是恒真,delta-only decode 回归时抓得住);T2 钉住"SQL 同步种子别名 vs 后台补注册未记录文件"的时序差;T3 legacy 对照。in-passing 修了 search-context-cross-session feat: 支持 search_context 显式查询历史 session #841 的 ENOTDIR(unified 顶层 index.db/WAL 被当目录遍历),独立标注 commit,合理。
  • 本地验证:typecheck exit 0;本 PR 影响面 15 个文件定向跑 146 pass / 0 fail。

diff 整洁度(⑤)

干净,每行都服务于目的。两处纯空白对齐(SQL FROM sessions s 缩进、一处 JSDoc 缩进)属可忽略噪声;statSync(...)?.size ?? 0 里的 ?. 是无效防御(statSync 缺失会 throw 不会返回 undefined,try/catch 已处理)——均为 cosmetic,不值得为此动分支。

CI 现状(当前唯一拦路项,非本 PR 之过)

本 head 上 sharded test lane 在 shard 2&3(ubuntu/windows × node22/24,共 6 lane)红。runner 有 #3029 的 swallowed-failure 自动重试(transient 会自愈到绿),这些没转绿说明是确定性或持续抖动。关键证据:多个互不相关的在途 PR 呈现同一组 shard 2/3 红——#3215(hermetic-zone)、#3213(nudge-decision,连 shard1 也红)、#3210(attach-diag)、#3202(byte-budgets);而 #3197("ci: per-file zone-port bands for live suites",待合并)是全绿的。即 live-suite 进程起停的端口碰撞抖动风暴,修复在 #3197 尚未合入 master,所以所有基于 master 的 head 都继承这条红。本 PR 与此无关。

结构化八问

  1. 是什么问题:冷实例大语料下,dashboard /__bili/overview 首扫和 boot 都在主线程整语料 JSON.decode,冻结实例数十秒到上百秒,并诱发 host 反复重启。
  2. 什么原因导致的:unified 存储下 web 列表 / overview / boot 三条路径都退化成"全语料同步 decode/解析",没有利用 index.db 已有的汇总基线,也没把冷数据移出启动关键路径;副作用是快照堆积挡住 SQLite checkpoint、放大磁盘压力。
  3. 影响范围与严重程度:仅 unified storage(shadow/full)+ 大语料的冷实例;legacy 模式不受影响。严重度=特定条件下实例不可用(冻结期 dashboard 无响应、触发 host respawn),稳态值不变。
  4. 怎么修复的:把三处冷扫描改成"基线 + 增量"——shadow 列表用 index 行做基线只对 delta decode;boot 只同步解析 warm window、其余后台补;sidecar 打开时被动回收过大 WAL。根因消除而非遮症状。
  5. 是否有回归问题:审过 full 列表 / legacy boot / 重启伪造时间戳:buildSession 用 Date.now() 覆盖 lastSeen,面板"最后活动"全员开机时刻、fallback=latest 在 ~245 路平局里随机挂会话 #404 契约 / 缓存键 / 两条 wire 模式,均保持或已披露;唯一边角是 warm 候选池 mtime 有界化在大语料极端情形下丢预热槽位(on-demand 兜底、无数据丢失),低危。
  6. 是否破坏缓存:否。(mtime,size) 磁盘索引键、增量游标、重试策略均未动。
  7. 是否改了设计方向:否。仍是 unified-first + legacy fallback 的数据流,只是把冷路径从主线程挪出/降为增量,边界未转向。
  8. 是否引入新配置:否。复用既有 sessions.warmMax/warmMaxBytes(及 BILI_SESSION_WARM_MAX[_BYTES] env),无新增配置面。

判定

可合入质量达标(代码、测试、披露到位,本 PR 影响面本地全绿 + typecheck 绿)。当前唯一拦路的是仓库级 live-suite 端口抖动风暴(非本 PR 引起,修复在待合并的 #3197)。建议 re-run 失败的 test lane 看能否自愈,或在 #3197 合入后合入本 PR。合并动作请 owner 手动执行(agent 不合 PR)。

awork added 3 commits October 15, 2026 18:34
…arsing in unified storage modes (#3199)

/__bili/overview (and /__bili/sessions) froze the event loop for minutes on
cold instances with large session corpora in unified storage: shadow mode
deliberately bypassed the index fast path (rows may lag unrecorded files),
so the first list hit decoded the entire corpus on the main thread, and
boot parsed the whole corpus before the listener started — feeding the
host-side respawn churn (#3044 family).

Two mechanisms, one failure domain (main-thread cold-corpus cost):

1. Shadow list surfaces now read the index as a BASELINE and reconcile
   against the file tree inside refreshIndex: fresh rows are adopted
   without decoding (file mtime within 5 min of row updated_at — normal
   writes always land after their row), files newer than their row or never
   recorded are decoded individually, ghost rows drop because only files
   produce entries. Full mode stays authoritative (vanished-file sessions
   remain listed); legacy keeps the plain walk. Cold-start cost drops from
   "decode the whole corpus" to one SQL + one stat per file + O(delta)
   decodes.

2. Boot in unified modes (shadow or full) parses only a bounded window of
   the newest files (mtime desc) before listen; label aliases are seeded
   synchronously from index rows and the remaining alias/canonical
   bookkeeping finishes in the background (backgroundScanComplete()).
   Survivor arbitration still uses the original savedAt-desc trim, so the
   #404 freshness semantics hold. Legacy mode keeps the synchronous full
   parse verbatim.

WAL reclamation (issue suggestion 3a) landed independently on master via
#3250/#3255 (open/close TRUNCATE checkpoints + periodic PASSIVE +
configurable cap) — this PR no longer carries its own checkpoint code.

Behavior changes (disclosed per owner policy):
- shadow-mode web lists may lag bypass-written files by up to 5 minutes,
  display-only (detail views always read the file; any activity re-ingests);
- unified-mode boot readiness = warm-window parse only; label aliases for
  UNRECORDED files fill post-boot in the background (recorded labels are
  seeded synchronously from the index).

No new config surface (reuses sessions.warmMax / sessions.warmMaxBytes).
Kernel untouched. Plugin/proxy wire modes unaffected (admin/web surface).

Tests: new tests/issue3199-overview-cold-scan.test.ts pins all paths;
targeted suites green locally (web-sessions, storage-unified, store,
persist-zstd, persist-encryption, bounded-storage, issue404,
identity-registry, search-context-cross-session); full verdict via remote CI.
…OTDIR under unified storage)

Small in-passing fix found while testing #3199, its own labeled commit.
Pre-existing failure verified on pristine master with the node:sqlite
builtin engine: the #841 cold-load case assumed every top-level entry under
the sessions dir is a protocol directory and readdir'd each one — unified
storage puts index.db (+WAL files) at top level, so the snapshot scan died
with ENOTDIR. Guard the descent with isDirectory(); unrelated to the #3199
change itself.
…egacy control)

- T1: shadow list = index baseline + tree delta — fresh row adopted with
  zero decodes; externally rewritten file decoded and wins; unrecorded file
  listed; ghost row dropped; live session overrides. Structural assert that
  decode count equals the delta count only (no full-corpus decode).
- T2: unified boot parses only the newest window before listen; survivors
  come from the savedAt-desc trim within the window (#404 contract kept);
  label aliases seeded synchronously from index rows; remainder scan
  completes post-boot and covers unrecorded labels.
- T3: legacy control — synchronous full parse, aliases complete immediately
  after boot (zero regression for non-unified installs).
@ranxianglei
ranxianglei force-pushed the 2026-10-15_fix-3199-overview-cold-scan branch from db8e7b0 to 22dfa12 Compare October 15, 2026 10:38
@ranxianglei
ranxianglei merged commit 844af99 into master Oct 15, 2026
40 of 66 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.447-master.844af99 (rolling master tag).

npm install -g billion-context@0.0.447-master.844af99  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Oct 16, 2026
…; output-restore floor respects proven budgets (#3406 items 2+4)

Item 2 — the #3286 ownership fast path judged envelope ownership by a
builtAtMs+1.5s grace window, so a sibling writing inside that window (legacy
mode, sidecar disabled — never touches index.db) was adopted as ours and its
envelope clobbered on the next save. The token now records the stat taken
right after OUR write landed (size+mtime); the fast path fires only when the
file still matches that stamp byte-for-byte. Unstamped tokens (debounced
kernel-internal saves, restore-replica seeds, crash before stamping) always
take the full read. Residual window shrinks from 1.5s to ~µs.

Item 4 — restoreOutputBudget's no-ceiling branch raised a session's recorded
high-water to OUTPUT_RESTORE_ABSOLUTE_FLOOR (32768); providers hard-capping
maxOutputTokens below that reject the raised value outright (400). A high-water
at/above the new trust line (4096, aligned with OUTPUT_CLAMP_MIN_OUTPUT) is
the provider-accepted budget and is restored as-is, never raised above;
missing/below-line high-waters keep the #3071 floor (born-starved, fork
rattle, death-spiral decay chains).

Item 3 (label resume vs background scan race, #3214) — NOT changed: no
low-risk mutex exists without a persistence-format/index change (details in
the PR body); deliberately stopped per the risk gate.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Web UI first paint stalls 78-180s+: /__bili/overview cold scan blocks the event loop, and the freeze itself feeds the respawn churn

1 participant