Skip to content

Cold-load diagnostics: a session state that loads with refs but no blocks now warns loudly instead of replaying raw in silence (#3573) - #3591

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-17_cold-load-diagnostics
Oct 17, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-17_cold-load-diagnostics

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

(ai: vllm-qwen/qwen3.8-27b)

What

The v0.1.198 incident (#3573) left a folded session re-entering as raw history — 1820 messages, usage 0%, ref=0 — with zero fold-reconcile logs, so three candidate causes (state never loaded / loaded refs-without-blocks / request keyed onto a ghost fork id) were indistinguishable in the field.

Code audit pinned why it was silent: reconcileFoldCoverage early-exits without logging when the loaded state has no active blocks (nothing to reconcile), and the lazy cold-load path (SessionStore.loadSync, warmed-0 default) logged nothing about what it actually restored from disk. This PR makes that moment observable:

  • every on-disk cold load logs one info line: source (unified store vs legacy file), carried refs, blocks (active), baseline tokens, savedAt age;
  • the incident signature specifically — refs mapped, zero persisted blocks — escalates to a warning naming the consequence (folded history will replay raw, preflight rebuilds from scratch).

No behavior change beyond the two log lines: no thresholds, wire shape, persistence format, or defaults touched. Log-only observability on a hot path is one Object.keys + two filters per cold load (a rare event by design).

Relationship to the other #3573 halves

Note for transparency: an earlier draft of my work on this issue also included a preflight "cold-start deformation" deterministic-digest fast path. That half was dropped as superseded by #3590's wall-clock admission, which reaches the same acceptance criterion (first turn finishes within client patience) while keeping model-quality summaries and adding no new regime.

Verification

  • New tests/issue3573-cold-load-diagnostics.test.ts: 3 cases (incident signature → WARN + counts; healthy load → counts, no warning; empty state → no false positive). Runs under the default unified-shadow storage mode (what the incident machine ran). Mutation-checked: 0/3 pass with the persist.ts change reverted, 3/3 with it.
  • Kindred tests/issue3157-persist-durability.test.ts green locally; typecheck clean. Full-suite verdict = remote CI on this head.

查重: 不重复 — #3586 / #3590 同属本 issue 但各自修「折叠后鬼表」与「马拉松超耐心」两个机制;本 PR 是 OP 修复方向 ① 点名的载入侧诊断,三者互不覆盖。

Refs #3573


Review supplement (2026-10-17, agent review): commit 3e38248 adds savedTok (stats.tokensSaved) to both log lines. The original refs-without-blocks warn also fires for sessions that were NEVER folded (any session with >=1 persisted turn and zero folds), so ordinary short sessions would log the same warning as a lost-fold incident after every restart. /NO persisted blocks/ + savedTok>0 = folds existed and are missing (investigate); savedTok=0 = likely never folded (raw replay is normal). Warn condition deliberately stays broad so a loss mode erasing stats cannot go silent again. Tests pin both classes. Note: the first CI run had one flaky red lane (test windows-latest 22 shard 1: lane-decisions-matrix event-loop hang, unrelated to this diff — that test runs SessionStore disabled and passes 22/22 locally; master green at this base); re-run triggered.

… silent

The v0.1.198 incident session re-entered as raw history with zero
fold-reconcile logs: reconcileFoldCoverage early-exits silently when the
loaded state carries no active blocks, and nothing logged what the lazy
cold load actually restored. Every on-disk load now reports its carried
refs/blocks/baseline at info level, and the incident signature (refs
mapped, zero persisted blocks) escalates to a warning naming the
consequence, so the next occurrence is a one-line diagnosis.
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-17_cold-load-diagnostics (3e38248)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4248-pr.3591.4dcaa62

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3591.tgz
npm install -g package

Install instructions are refreshed on each push.

Review found the refs-without-blocks warn also fires for sessions that were
NEVER folded (any session with >=1 persisted turn and zero folds): after a
restart, revisiting ordinary short sessions would log the same warning as a
lost-fold incident, diluting the signal this PR exists to create. Both lines
now carry stats.tokensSaved: >0 proves compression ran in the session's
lifetime (folds existed and are now missing — investigate); 0 means likely
never folded (raw replay is its normal shape). The warn condition stays
deliberately broad so a loss mode that also erases stats cannot go silent
again. Tests pin both classes.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — #3586(折叠后鬼表 meter)、#3590(马拉松超耐心)与本 PR 同属 #3573 但机制互不覆盖;本 PR 是 OP 修复方向 ①("至少 log")点名的载入侧诊断。另查了 open PR:无其他 PR 触碰 persist 冷载入日志路径。

审核结论(八问)

  1. 是什么问题:v0.1.198 事故(v0.1.198 升级后老会话丢失压缩状态:197 折叠过的会话以 117 万 token 原文重放,预压缩超过客户端耐心,老会话看起来「炸了」 #3573)里一个已折叠会话以原始历史身份回来(1820 条、usage 0%、ref=0),却一条 fold-reconcile 日志都没有——"状态没加载 / 加载了 refs 没 blocks / 请求打到了 ghost fork id"三个候选原因在现网无法区分。
  2. 什么原因导致的:不是行为 bug,是观测缺口——reconcileFoldCoverage 在无 active block 时静默早退,懒冷载路径(SessionStore.loadSync)对实际从磁盘恢复了什么零日志。状态在"重新进入内存"这一刻从未留下痕迹。
  3. 影响范围与严重程度:纯诊断缺口,无外部行为影响;严重度 = 事故不可诊断(v0.1.198 升级后老会话丢失压缩状态:197 折叠过的会话以 117 万 token 原文重放,预压缩超过客户端耐心,老会话看起来「炸了」 #3573 根因至今卡在等现场证据,下次发生仍会重演这次的多日排查)。
  4. 怎么修复的:log-only 可观测性。每次磁盘冷载打一条 info(source / refs / blocks / baseline / savedAt 龄期);事故签名(refs>0 且零持久化 blocks)升级为 warn 并写明后果。独立核对了覆盖性主张:所有持久化状态重新进入内存池的路径都过 loadSync(bootBounded warm 循环 + getSession miss),buildForLoadAll 只喂 export/迁移枚举,不进常驻池——成立。审核补充(commit 3e38248,已推分支):评审发现原 warn 条件对"从未折叠过的会话"也会触发(任何 ≥1 个持久化回合、零折叠的短会话都是 refs>0/blocks=0),重启后回访普通短会话会打出和丢折叠事故一模一样的警告,稀释信号。两条日志现在都带 savedTok(stats.tokensSaved):>0 = 该会话生命周期内压缩发生过(blocks 曾存在、现在没了 → 值得查);=0 = 大概率从未折叠(raw replay 是正常形态)。warn 条件刻意保持宽——一个连 stats 一起抹掉的丢失模式不能再让我们回到沉默。测试钉住两类。
  5. 是否有回归问题:审了 loadSync 三个分支(unified-first / unified 内 legacy-fallback / plain-legacy)——控制流未动,仅新增私有方法 + 三处单行调用;reconcile/export/迁移/更新路径未触碰;无线形、阈值、持久化格式、默认值变化(符合 PR 声明)。本地定向验证:typecheck 干净;新测试 3/3、kindred issue3157 2/2;mutation 复核:还原 persist.ts → 0/3,恢复 → 3/3(作者声称的 mutation-check 我独立复跑确认)。CI 首轮唯一红 lane 是 test (windows-latest, 22, 1):lane-decisions-matrix.test.ts 在 Windows 上 event-loop 时序挂起("Promise resolution is still pending but the event loop has already resolved",另两个兄弟 case 被 cancelledByParent 连带)——与本 diff 无关(该测试的 SessionStore 是 enabled:false,loadSync 直接早退,本 PR 代码路径根本没走到;本地 22/22 绿;master 在同一 base b10c525 两次 CI 全绿)。已触发 flaky 重跑,新 commit 又触发了全新 CI。
  6. 是否破坏缓存:不涉及——不碰任何缓存/重试/增量游标/去重键。
  7. 是否改了设计方向:不涉及——纯附加观测,架构与数据流不变。已知残留:full-mode("unified store")调用点无专属测试(三个 case 跑在默认 shadow 模式、走 legacy-label 分支);该处是与受测 helper 的单行镜像,风险低,记录在此不阻塞。
  8. 是否引入了新的配置:不涉及。

判定:代码与测试质量达标,方向与 OP ① 一致,回归面干净;待 3e38248 的 CI 全绿后可合(合并动作归你)。PR 描述已补 review supplement 段。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 确认:head 3e38248 主 CI(run 39046631518)全绿——含此前 flaky 的 test (windows-latest, 22, 1)(lane-decisions-matrix 这次直接过,坐实上一轮是瞬态挂起,已把复发数据点补到 #3574);全部 e2e 车道(fake-codex / real codex / dsh / pi / opencode / persona / local-registry / windows-codex-overlay)+ PR Build Artifact 也全绿。PR 当前 open、mergeable=true(merge_state: blocked 来自每次 push 都幽灵失败的 ci-gate-plugins.yml,已有 #3532/#3574 跟踪,与本 PR 无关)。

审核收工:代码与测试达标、回归面干净、补充 commit 已并入分支,可合(合并归你)。

@ranxianglei ranxianglei mentioned this pull request Oct 17, 2026
6 tasks
@ranxianglei
ranxianglei merged commit fd3232f into master Oct 17, 2026
42 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.547-master.fd3232f (rolling master tag).

npm install -g billion-context@0.0.547-master.fd3232f  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant