Repository navigation
fix: bound web admin session loading (paged list, summary index, lazy detail) - #1939
Conversation
…tmp-") parity Review of #1939: walkSessionFiles used name.includes(".tmp-"), which is broader than the kernel StateStore.walkJsonFiles skip (startsWith). A canonical filename may legitimately contain ".tmp-" mid-string when the upstream host carries such a label (hostLabel keeps dots, e.g. api.tmp-x.example_<hash>.json): the kernel loads that session at boot but the old check hid it from the web list/overview and 404ed its detail. Encrypted temps (.tmp-enc-N-N) carry no .json suffix, so the endsWith gate above already drops them. Adds a regression test pinning the parity.
…switch data Review of #1939: the original PR removed assert.ok(!plainHtml.includes( "8.0K")) from the #1535 regression test without explanation. Re-running with the assertion restored proves it still holds deterministically on the new summary-index render path (data-level field-omission checks at lines 502-503 do not cover the rendered HTML), so the guard is kept.
[bot] 🏷 Review 结论:方向正确、实现扎实;审核中发现 1 个真实缺陷 + 1 处未解释的测试弱化,已直接在分支上修复并本地验证(2 commits,fast-forward 推送 结构化七问1. 是什么问题:web admin 的 2. 什么原因导致的:根因是数据流设计——"整语料进内存",内存占用 = f(语料总字节),与页面大小无关;5s 轮询只是让 OOM 从"迟早"变成"必然"。修频率是堵,修数据流才是疏。 3. 怎么修复的:改为有界摘要索引——walk 只 stat 不 decode;逐文件顺序 decode 一次(文件间 yield 给 GC)生成轻量 summary 记录后丢弃原文;list/overview 从索引出(分页 4. 是否有回归问题(逐项审过既有行为):
5. 是否破坏缓存:旧 5s TTL memo 被 mtime+size 失效取代——语义从"每 5s 必重扫"变为"文件变化才重扫"(warm scan 零 decode,有结构性测试证明)。这正是 OOM 修复的核心;无其他缓存/重试/去重机制受影响。session 磁盘格式、persist 层均未动。 6. 是否改了设计方向:是——web session 数据路径从"整语料驻留内存"转为"有界摘要索引 + 惰性单文件解码"。属 src/web + persist 边界内的数据流调整,与 #1937 floor 3 提出的方案一致,实现与方案吻合。 7. 是否引入了新的配置:无。未新增 env/config/CLI 面。 行为变更披露(old → new,PR 已披露,审核复核确认)
本地验证证据(typecheck + 定向测试;全量留给 CI)
CI 状态(如实报告)两个 head sha( 收工判定:代码可合并(审核问题已修复并本地验证);合并操作 human-only,请 owner 在 CI 恢复并转绿后自行 merge: #1939 |
|
这个有冲突 解决下吧 |
…tmp-") parity Review of #1939: walkSessionFiles used name.includes(".tmp-"), which is broader than the kernel StateStore.walkJsonFiles skip (startsWith). A canonical filename may legitimately contain ".tmp-" mid-string when the upstream host carries such a label (hostLabel keeps dots, e.g. api.tmp-x.example_<hash>.json): the kernel loads that session at boot but the old check hid it from the web list/overview and 404ed its detail. Encrypted temps (.tmp-enc-N-N) carry no .json suffix, so the endsWith gate above already drops them. Adds a regression test pinning the parity.
…switch data Review of #1939: the original PR removed assert.ok(!plainHtml.includes( "8.0K")) from the #1535 regression test without explanation. Re-running with the assertion restored proves it still holds deterministically on the new summary-index render path (data-level field-omission checks at lines 502-503 do not cover the rendered HTML), so the guard is kept.
45b722b to
b366a0f
Compare
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.2222-pr.1939.3991c9bThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr1939.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 冲突已解决,rebase 到最新 master( 两处冲突:
验证(rebase 后全新跑过):
CI 状态: 仓库级调度停摆(#1941 跟踪的那个)已自行恢复,新 head 上全套已在跑——real-image billing / fake-codex / opencode-native / dsh-profile 四条 e2e 已绿,test 矩阵(ubuntu/windows × node 22/24)+ registry e2e 进行中。矩阵全绿后即可合并,我随后回帖确认。 |
[bot] 🏷 CI 全绿,收工确认:
判定:可合并。 分支为 |
…y detail The admin page's /__bili/overview and /__bili/sessions both funneled through a 5s-TTL memo of SessionStore.loadAll(), while the UI polls every 5s — so every poll fully decoded and re-built EVERY persisted session file and kept all full Session objects (blockContents maps, messages, state) resident. Peak memory ≈ 2x the parsed corpus; a ~127MiB history under a 512MiB container limit OOM-killed the proxy on each overview request (#1937). - sessions-data.ts: replace the loadAll() memo with a bounded summary index — stat-only directory walk, per-file summaries cached by (mtime,size), sequential decode of changed/new files only (parsed record discarded right after extraction, GC checkpoint between files), live-pool files skipped, corrupt/renamed/content-store files logged and skipped. List/overview cost is now O(#sessions x ~1KB) resident regardless of total corpus bytes. - detail: lazy single-file decode via new read-only SessionStore.loadSessionFromFile (filename<->id guard mirrors kernel loadSync), per-id single-flight; no more full scan for one id. - /__bili/sessions: optional ?q=/?page=/?pageSize= (clamped <=200) switch to server-side filtered paging; no-param response shape unchanged (+ additive total). Unreadable sessions dir -> HTTP 500 instead of silent empty, with the last known index served when one exists. - frontend: 50 rows/page + load-more, debounced server-side search, 20s fetch timeout, persistent error strip, poll backoff 5s -> 60s cap, in-flight guard, background refresh never clobbers a paged view.
…tmp-") parity Review of #1939: walkSessionFiles used name.includes(".tmp-"), which is broader than the kernel StateStore.walkJsonFiles skip (startsWith). A canonical filename may legitimately contain ".tmp-" mid-string when the upstream host carries such a label (hostLabel keeps dots, e.g. api.tmp-x.example_<hash>.json): the kernel loads that session at boot but the old check hid it from the web list/overview and 404ed its detail. Encrypted temps (.tmp-enc-N-N) carry no .json suffix, so the endsWith gate above already drops them. Adds a regression test pinning the parity.
…switch data Review of #1939: the original PR removed assert.ok(!plainHtml.includes( "8.0K")) from the #1535 regression test without explanation. Re-running with the assertion restored proves it still holds deterministically on the new summary-index render path (data-level field-omission checks at lines 502-503 do not cover the rendered HTML), so the guard is kept.
b366a0f to
b9bcd86
Compare
b9bcd86 to
7b21444
Compare
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.43-master.cd6cbb9 # this exact build
npm install -g billion-context@master # always the newest merged build
|
…dex (#2152) #1939's stale-snapshot/500 semantics turned a not-yet-created sessions directory into a hard failure: readdir's ENOENT propagated out of walkSessionFiles, so /__bili/overview and /__bili/sessions answered 500 on a fresh install, and the advisory-endpoint-parity CI lane went red whenever no sibling test had populated the default dir first. ENOENT at the walk root now reads as zero disk sessions — it is indistinguishable from a vanished directory anyway, and the index self-heals on the next refresh once the store recreates the directory. Every other walk error keeps the existing serve-stale-snapshot-else-500 behavior. Tests: the #1937 breakage test is rewritten to the new semantics (vanished dir reads empty, then self-heals after the store rewrites into it; ENOTDIR still rejects loudly) and a regression pins buildOverview on a missing dir.
…dex (#2152) #1939's stale-snapshot/500 semantics turned a not-yet-created sessions directory into a hard failure: readdir's ENOENT propagated out of walkSessionFiles, so /__bili/overview and /__bili/sessions answered 500 on a fresh install, and the advisory-endpoint-parity CI lane went red whenever no sibling test had populated the default dir first. ENOENT at the walk root now reads as zero disk sessions — it is indistinguishable from a vanished directory anyway, and the index self-heals on the next refresh once the store recreates the directory. Every other walk error keeps the existing serve-stale-snapshot-else-500 behavior. Tests: the #1937 breakage test is rewritten to the new semantics (vanished dir reads empty, then self-heals after the store rewrites into it; ENOTDIR still rejects loudly) and a regression pins buildOverview on a missing dir.
… CI red since #2153) The test injects a DISABLED SessionStore, so its own server boot never creates the real sessions dir, and /__bili/overview's walk deliberately throws on a missing top-level dir (#1939) — 500 instead of 200. Passing therefore hinged on a concurrently-running sibling file (e.g. anthropic-cache-control.test.ts, the next name alphabetically, whose un-injected boot mkdirs $HOME/.local/share/billion-context/sessions) winning a cross-file race. PR runs got lucky; every master push run since 4e3ba1e failed all four test lanes. Pin BILI_SESSIONS_DIR to an owned empty tempdir instead.
Closes #1937
What changed
The admin page's
GET /__bili/overviewandGET /__bili/sessionsboth funneled through a 5s-TTL memo ofSessionStore.loadAll(), while the UI polls every 5s. Every poll therefore fully decoded and re-built EVERY persisted session file (JSON parse +buildSessionwith blockContents maps / messages / state) and kept all fullSessionobjects resident until the next scan replaced them. Peak memory ≈ 2× the parsed corpus; with ~250 files / ~127MiB of history under a 512MiB container limit (~259MiB V8 heap), each overview request OOM-killed the proxy (Reached heap limit Allocation failed - JavaScript heap out of memory).This PR makes admin-data reads bounded:
src/web/sessions-data.ts): stat-only directory walk; per-file summaries cached by(mtime,size); only new/changed files are decoded, sequentially, one at a time, with a GC checkpoint between files and the parsed record discarded immediately after summary extraction. Live-pool files are skipped entirely (their live summaries win anyway); corrupt, renamed (filename≠body id), and CCR content-store files are logged and skipped. List/overview resident cost is now O(#sessions × ~1KB) regardless of total corpus bytes; a single request's peak is one decoded file.SessionStore.loadSessionFromFile(path, expectId)(filename↔id guard mirrors kernelloadSync; no 代理压缩不上报改写量,宿主上下文会计塌缩至 ~0(omps recordAnchoredHistoryRewrite 零调用);且持久化 lastInputTokens 恢复无钳制,旧版负值升级后复活 #408 clamp-rewrite side effect), with per-id single-flight for concurrent requests.GET /__bili/sessions?q=&page=&pageSize=(pageSize clamped ≤200). Without params the response shape is unchanged ({sessions, hiddenEmpty}+ additivetotal).client.ts/page.ts/i18n.ts): 50 rows/page with a Load-more button, debounced server-side search, hard 20s fetch timeout, a persistent visible error strip on data failure, poll backoff 5s→10s→…→60s cap, an in-flight guard so slow responses never stack, and background refresh that never clobbers a paged/searched view.persist.ts(relPathFor,isValidRecord,fileNameMatchesId,SessionStore.dir,loadSessionFromFile),sessions-data.ts(buildSessionPage,_diskScanStatsForTest). No config/env/CLI surface added.Behavior changes (old → new)
/__bili/sessionswithout params: gains additive fieldtotal; everything else identical — older UIs unaffected./__bili/sessionswith?q=/?page=/?pageSize=: NEW query parameters (previously ignored because the route matched exact URL).pageSizeclamps to ≤200; invalid values fall back to 50/page, page 1./__bili/overviewand/__bili/sessionscan now return HTTP 500{"error": ...}when the sessions dir is unreadable AND no earlier successful scan exists (old: silently returned an empty list — the "no data" state the issue saw before the crash). When a prior index exists, stale data is served instead of erroring.Version attribution note
The issue was observed on a customized v0.1.180 Docker image. The unbounded mechanism itself (TTL-5s memo of
loadAll()+ 5s frontend poll + retained full sessions) exists in stock master, so stock builds at the same data scale are equally exposed; the custom image may have amplified but does not create the defect.Pre-flight
npm run typecheck— cleantests/web-sessions.test.ts(12, incl. 5 new 管理页面全量加载历史会话时触发堆 OOM:需要有界分页、摘要统计与详情懒加载 #1937 regression tests: warm-scan zero decodes, detail single-file decode proof, corrupt/content-store resilience, stale-snapshot vs loud-fail on dir breakage, endpoint paging/search/clamping),tests/web-i18n.test.ts(catalog lint incl. the two new buttons), plus all 9 persist-related suites (101 pass total)npm run build— cleansrc/server.ts, soci-e2e.ymlwill run)