Repository navigation
fix(web): stop /__bili dashboard cold scans from freezing the instance (#3199) - #3214
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.3782-pr.3214.8ee359eThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3214.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复。 本 PR 对应原始 bug issue #3199(冷实例大语料下 先给结论:代码质量好、方向对、修法打在根因上,本 PR 自己的影响面本地全绿;当前 CI test lane 的红是仓库级 live-suite 端口碰撞抖动风暴造成的,不是这个 PR 引起(证据见下)。建议 re-run 失败的 check(抖动常能自愈),或在 #3197 落地后再合。 方向关(① 审原 issue)读 #3199:冷实例 + unified storage(shadow/full)+ 大语料时,首次
回归关(② 既有行为逐项清点)
代码关(③)/ 测试关(④)
diff 整洁度(⑤)干净,每行都服务于目的。两处纯空白对齐(SQL CI 现状(当前唯一拦路项,非本 PR 之过)本 head 上 sharded test lane 在 shard 2&3(ubuntu/windows × node22/24,共 6 lane)红。runner 有 #3029 的 swallowed-failure 自动重试(transient 会自愈到绿),这些没转绿说明是确定性或持续抖动。关键证据:多个互不相关的在途 PR 呈现同一组 shard 2/3 红——#3215(hermetic-zone)、#3213(nudge-decision,连 shard1 也红)、#3210(attach-diag)、#3202(byte-budgets);而 #3197("ci: per-file zone-port bands for live suites",待合并)是全绿的。即 live-suite 进程起停的端口碰撞抖动风暴,修复在 #3197 尚未合入 master,所以所有基于 master 的 head 都继承这条红。本 PR 与此无关。 结构化八问
判定可合入质量达标(代码、测试、披露到位,本 PR 影响面本地全绿 + typecheck 绿)。当前唯一拦路的是仓库级 live-suite 端口抖动风暴(非本 PR 引起,修复在待合并的 #3197)。建议 re-run 失败的 test lane 看能否自愈,或在 #3197 合入后合入本 PR。合并动作请 owner 手动执行(agent 不合 PR)。 |
…arsing in unified storage modes (#3199) /__bili/overview (and /__bili/sessions) froze the event loop for minutes on cold instances with large session corpora in unified storage: shadow mode deliberately bypassed the index fast path (rows may lag unrecorded files), so the first list hit decoded the entire corpus on the main thread, and boot parsed the whole corpus before the listener started — feeding the host-side respawn churn (#3044 family). Two mechanisms, one failure domain (main-thread cold-corpus cost): 1. Shadow list surfaces now read the index as a BASELINE and reconcile against the file tree inside refreshIndex: fresh rows are adopted without decoding (file mtime within 5 min of row updated_at — normal writes always land after their row), files newer than their row or never recorded are decoded individually, ghost rows drop because only files produce entries. Full mode stays authoritative (vanished-file sessions remain listed); legacy keeps the plain walk. Cold-start cost drops from "decode the whole corpus" to one SQL + one stat per file + O(delta) decodes. 2. Boot in unified modes (shadow or full) parses only a bounded window of the newest files (mtime desc) before listen; label aliases are seeded synchronously from index rows and the remaining alias/canonical bookkeeping finishes in the background (backgroundScanComplete()). Survivor arbitration still uses the original savedAt-desc trim, so the #404 freshness semantics hold. Legacy mode keeps the synchronous full parse verbatim. WAL reclamation (issue suggestion 3a) landed independently on master via #3250/#3255 (open/close TRUNCATE checkpoints + periodic PASSIVE + configurable cap) — this PR no longer carries its own checkpoint code. Behavior changes (disclosed per owner policy): - shadow-mode web lists may lag bypass-written files by up to 5 minutes, display-only (detail views always read the file; any activity re-ingests); - unified-mode boot readiness = warm-window parse only; label aliases for UNRECORDED files fill post-boot in the background (recorded labels are seeded synchronously from the index). No new config surface (reuses sessions.warmMax / sessions.warmMaxBytes). Kernel untouched. Plugin/proxy wire modes unaffected (admin/web surface). Tests: new tests/issue3199-overview-cold-scan.test.ts pins all paths; targeted suites green locally (web-sessions, storage-unified, store, persist-zstd, persist-encryption, bounded-storage, issue404, identity-registry, search-context-cross-session); full verdict via remote CI.
…OTDIR under unified storage) Small in-passing fix found while testing #3199, its own labeled commit. Pre-existing failure verified on pristine master with the node:sqlite builtin engine: the #841 cold-load case assumed every top-level entry under the sessions dir is a protocol directory and readdir'd each one — unified storage puts index.db (+WAL files) at top level, so the snapshot scan died with ENOTDIR. Guard the descent with isDirectory(); unrelated to the #3199 change itself.
…egacy control) - T1: shadow list = index baseline + tree delta — fresh row adopted with zero decodes; externally rewritten file decoded and wins; unrecorded file listed; ghost row dropped; live session overrides. Structural assert that decode count equals the delta count only (no full-corpus decode). - T2: unified boot parses only the newest window before listen; survivors come from the savedAt-desc trim within the window (#404 contract kept); label aliases seeded synchronously from index rows; remainder scan completes post-boot and covers unrecorded labels. - T3: legacy control — synchronous full parse, aliases complete immediately after boot (zero regression for non-unified installs).
db8e7b0 to
22dfa12
Compare
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.447-master.844af99 # this exact build
npm install -g billion-context@master # always the newest merged build
|
…; output-restore floor respects proven budgets (#3406 items 2+4) Item 2 — the #3286 ownership fast path judged envelope ownership by a builtAtMs+1.5s grace window, so a sibling writing inside that window (legacy mode, sidecar disabled — never touches index.db) was adopted as ours and its envelope clobbered on the next save. The token now records the stat taken right after OUR write landed (size+mtime); the fast path fires only when the file still matches that stamp byte-for-byte. Unstamped tokens (debounced kernel-internal saves, restore-replica seeds, crash before stamping) always take the full read. Residual window shrinks from 1.5s to ~µs. Item 4 — restoreOutputBudget's no-ceiling branch raised a session's recorded high-water to OUTPUT_RESTORE_ABSOLUTE_FLOOR (32768); providers hard-capping maxOutputTokens below that reject the raised value outright (400). A high-water at/above the new trust line (4096, aligned with OUTPUT_CLAMP_MIN_OUTPUT) is the provider-accepted budget and is restored as-is, never raised above; missing/below-line high-waters keep the #3071 floor (born-starved, fork rattle, death-spiral decay chains). Item 3 (label resume vs background scan race, #3214) — NOT changed: no low-risk mutex exists without a persistence-format/index change (details in the PR body); deliberately stopped per the risk gate.
Closes #3199
Rebased onto current master (
c43bdf01a). Note: the issue's suggestion 3a (WAL growth) landed independently via #3250/#3255 — open/close TRUNCATE checkpoints + periodic PASSIVE every 20 ingests + configurablestorage.walJournalSizeLimitBytescap — so this PR no longer carries its own checkpoint code.What was wrong
On a cold instance with a large session corpus in unified storage (
storage.unified=shadoworfull), the first/__bili/overview(and/__bili/sessions) hit froze the whole instance for minutes — 78–180 s measured on a 2.3 GB corpus. Two producers, one failure domain (main-thread cold-corpus cost):src/persist.tsunifiedWebSummaries()gate!this.shadow?.readsUnified → null;src/web/sessions-data.tsunchanged since v0.1.197).bootBoundedparsed the entire corpus beforeserver.listen()ran — so even health probes couldn't get through during boot. While the instance was frozen, host-side connection handling migrated traffic to a freshly spawned cold instance, which paid the same scan → freeze → migrate again (the autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044 churn family; each migration also left an orphan behind). Also verified byte-identical on latest master.What changed (3 commits)
fix(web)— shadow list surfaces now read the index as a baseline and reconcile against the file tree insiderefreshIndex: fresh rows are adopted without decoding (file mtime within 5 min of rowupdated_at; normal writes always land after their row because ingest precedes the JSON flush), files newer than their row or never recorded are decoded individually, ghost rows drop (only files produce entries). Cold-start cost drops from "decode the whole corpus" to one SQL + one stat per file + O(delta) decodes. Full mode stays authoritative (vanished-file sessions remain listed — R2.1 behavior untouched); legacy keeps the plain walk verbatim.SessionSummaryRow.updatedAtcolumn); remaining alias/canonical bookkeeping finishes in the background (backgroundScanComplete()). Survivor arbitration still uses the original savedAt-desc trim, so the 重启伪造时间戳:buildSession 用 Date.now() 覆盖 lastSeen,面板"最后活动"全员开机时刻、fallback=latest 在 ~245 路平局里随机挂会话 #404 freshness contract holds. Legacy mode keeps the synchronous full parse byte-for-byte.fix(tests)— small labeled in-passing fix: the pre-existing feat: 支持search_context显式查询历史 session #841 cold-load case assumed every top-level entry under the sessions dir is a protocol dir and died with ENOTDIR next toindex.db(verified failing on pristine master with the node:sqlite builtin engine). Guarded the descent withisDirectory().test:— newtests/issue3199-overview-cold-scan.test.tspins all paths, including a structural assert that decode count equals the delta count only.Behavior changes (old → new, disclosed per owner policy)
Steady-state values are identical — same data sources, same summary pipeline (
summaryOfunchanged). No cache/retry/incremental-cursor semantics touched; the disk-index(mtime,size)cache keys are preserved (row-adopted entries carry real file mtime/size).Config surface
None. Reuses existing knobs (
sessions.warmMax/sessions.warmMaxBytes,BILI_SESSION_WARM_MAX[_BYTES]).kernel-change
N/A — no changes under
kernel/.Modes reasoned
Plugin/proxy wire modes unaffected (admin/web surface only); both compression modes considered — no wire payload, rebuild, or injection path touched.
Tests
New suite pins T1 (shadow reconcile incl. ghost-drop + delta-only decode count), T2 (bounded boot + #404 survivor semantics + background alias completion), T3 (legacy control: synchronous full parse). Targeted local matrix green on the rebased head (131 tests: web-sessions, storage-unified, store, persist-zstd, persist-encryption, bounded-storage(+scale), issue404-restore-timestamps, fix-401-storage, identity-registry, search-context-cross-session); typecheck clean. Full-suite verdict via remote CI on this head.
Out of scope (named follow-ups)