Skip to content

[PR] fix: escalate structural digest drift to alarm + envelope freeze, load the fuller side (#4323) - #4332

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-23_drift-escalation-freeze
Oct 23, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-23_drift-escalation-freeze

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Closes #4323

Problem

Multiple bili instances sharing one sessions dir (#394 anti-pattern) ping-pong the JSON envelope and the SQLite index into permanent divergence ([diverged: state] never converging). Every restart/eviction cold load hit the drift latch and served the STALE envelope unconditionally — measured incident: envelope 12 active blocks vs index 16, ~240K already-folded tokens silently returned to context (#4323; both recurrences same mechanism per owner analysis).

Root cause confirmed by owner on #4323; this PR implements the two owner-specified code fixes. The ops layer (single-instance discipline, zombie-lane cleanup) stays with the operator; #4177 second-instance detection already lands post-0.1.204.

Changes

① Drift escalation + envelope-write freeze (src/storage/shadow.ts)

② Drift-latched cold load takes the fuller fold coverage (src/persist.ts)

  • loadSync's drift fallback previously assumed "the last file write is fresher" — false under multi-writer ping-pong. Now: when the unified store STRICTLY carries more active blocks than the envelope, serve the index (WARN with both counts). Tie or unreadable other side → historical envelope preference, unchanged. Unlatched loads untouched (unified-first default).

Observability (src/storage/runtime-status.ts)

  • Additive frozenEnvelopeSessions count on the /__bili/status + overview storage wire.

Behavior changes (old → new)

Surface Old New
Drift-latched cold load always served the legacy envelope while present serves the unified store when it strictly has MORE active blocks; tie/unreadable keeps old behavior
Envelope writes after 3 consecutive non-converging samples kept churning the shared file frozen for this process until a clean sample

No prefix-cache / retry / watermark / dedup-key semantics touched. No config surface added (internal constant only). No kernel/ or pi-subagents/ changes.

Verification

  • New suite tests/issue4323-drift-escalation.test.ts (6 tests): escalation threshold positive+negative at each step, clean-sample re-admission, latched-load selection matrix (index-richer→index / envelope-richer→envelope / tie→envelope / unlatched negative gate), wire field.
  • Neighbors green locally: storage-full-flip, issue3919-shadow-drift-forensics, web-storage-banner, storage-cantopen-hint (26/26 combined). typecheck clean. Full regression = CI.

Review note (rebase, 2026-10-23)

Rebased onto current master (aa198fe): head f567967 → 5bfcc0f. Single conflicted file src/storage/runtime-status.ts (4 hunks) — master's #4175 durability-trio fields (integrity/secondProcess/backup) and this PR's additive frozenEnvelopeSessions landed in the same four spots; resolved as the union of both (purely additive). Post-rebase: typecheck clean; targeted suites green locally (issue4323 6/6, storage neighbors 21/21). The red test shards + mysql-contract on the rebased head are pre-existing master-side defects since #4177 merged — tracked in #4333 (fix: #4334 in flight) and #4326-B respectively, NOT caused by this PR; no CI fixes were bundled here.

@ranxianglei ranxianglei changed the title fix: escalate structural digest drift to alarm + envelope freeze, load the fuller side (#4323) [PR] fix: escalate structural digest drift to alarm + envelope freeze, load the fuller side (#4323) Oct 22, 2026
…d the fuller side (#4323)

Multi-writer shared sessions dir (#394) ping-pongs the JSON envelope and the SQLite index into permanent divergence; every drift-latched cold load then served the stale envelope unconditionally, putting active fold blocks back into context (~240K tokens, #4323 incident).

- shadow.ts: count consecutive sampled drifts; at DRIFT_ESCALATE_STREAK (3) log ONE loud WARN naming both sides' active block counts and freeze THIS process's JSON envelope writes for the session; a clean sample re-admits.
- persist.ts: all three save routes go index-only for frozen sessions (same path as writerOff); loadSync's drift latch serves the unified store when it strictly carries MORE active blocks than the envelope (tie or unreadable other side keeps the historical envelope preference).
- runtime-status.ts: additive frozenEnvelopeSessions on the status wire.
- tests/issue4323-drift-escalation.test.ts: escalation threshold both directions, re-admission, latched-load selection (index-richer / envelope-richer / tie / unlatched negative gate), wire field.
@ranxianglei
ranxianglei force-pushed the 2026-10-23_drift-escalation-freeze branch from f567967 to 5bfcc0f Compare October 22, 2026 17:05
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-23_drift-escalation-freeze (5bfcc0f)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.5145-pr.4332.aeec752

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4332.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 本 PR 实现 #4323 中 owner 指定的两个代码修复(漂移升级+冻结、latched 冷加载取更全一侧),仓库内无针对同一缺陷的并行修复 PR。审核中发现的 CI 红项(LIVE_FILES 绊线、mysql-contract)是 master 侧既有缺陷,已由 #4333/#4334/#4326 跟踪,与本 PR 不同机制——未把任何 CI 修复合进本 PR。

标签: bug / scope/storage / p1(缓存泄露类存储层缺陷:折叠覆盖在冷加载丢失;沿用原 issue #4323 分类)。

审核结论(独立审核,非复读 issue 结论)

① 方向(原 issue): #4323 的根因链(owner floor 4 钉死):多实例互写 → digest 漂移永不收敛 → drift latch 下无条件 serve 旧 envelope → 丢 N 个 active fold → ~240K 已折叠 token 回灌上下文。本 PR 与 owner 指定的两条修法一一对应,无偏离无越界:①连续 3 次采样漂移不收敛 → 一条响亮的 WARN(点名两侧 active block 数)+ 冻结本进程对该会话的 envelope 写入,直到干净样本再准入;②latched 冷加载从「无条件 envelope」改为「仅当 index 严格更多 active blocks 时取 index」。运营层(单实例纪律)与 #4177 第二实例检测被正确地留在 PR 之外。方向成立。

② 回归审查(既有行为清点): 逐项读了改动前代码并判定:

  • sampled drift 的 error 行前缀([storage] unified shadow DIGEST DRIFT …)字节不变——既有 grep/告警不受影响;
  • 漂移后的 watermark-clear 自愈动作(clearIngestWatermark)未动;
  • wire 上 driftedSessions 计数语义不变,新增 frozenEnvelopeSessions 纯增量;
  • 无 latch 的加载保持 unified-first(B4 负向门钉住);latch 下 tie/对侧不可读保持历史 envelope 偏好(B2/B3 钉住);
  • 三条 save 路由的 writerOff index-only 路径只加了一个 OR 条件,下游(sidecar ingest、orphan-contents 安全阀)走同一路径;
  • mysql/网络后端车道被显式排除(envelopeFrozen 仅在 sync!==null 生效),async 路径的 driftedSessions 语义不变——边界自洽;
  • 高危模式逐项过:无 fail-fast 变吞错(try/catch 只包日志计数增强,不碰 heal 路径);无阈值/时序/输出格式漂移;无配置面新增。
    未发现回归。

③ 代码: 命名达意(driftStreaks/frozenEnvelopeIds/envelopeFrozen/activeBlockCount/countActiveBlocks);状态为 per-process 内存(与 ingestCounts 同纪律)、刻意不持久化且注释说明了理由;countActiveBlocks 对畸形文件防御性返回 0(不抛进 load 路径);activeBlockCount 返回 null 而非猜测,调用方契约有文档;升级只在 streak === 3 触发一次(不是 >=),干净样本同时重置 streak 与 freeze。一处无害小瑕疵(不值得改):envelope 侧 truthy b.active 与 index 侧 b.active === true 风格不一致,良构行上等价。

④ 测试有效性: 6 个测试全是真断言、有牙:A 逐步钉 verifyDrift=1/2/3 配 frozen=false/false/true(阈值每步正+负门),修复后再准入,并钉「恰好 1 条升级告警 + 恰好 3 条逐样本 error 行」;B1–B4 覆盖选择矩阵全四象限(index 多→index / envelope 多→envelope / 平手→envelope / 无 latch→unified-first 负向门)。B1 在旧代码(永远 serve envelope)上必失败——真回归钉。小缺口:对侧不可读(activeBlockCount→null)分支落到历史行为,由 B2/B3 间接钉住,可接受。

⑤ diff 整洁度: 4 文件 +490/-3,每行都与目的相关,无空白噪音、无锁文件/生成物。已 rebase 到当前 master(aa198fe):唯一冲突文件 runtime-status.ts 共 4 处,全是「master #4175 加的 durability 三字段 vs 本 PR 加的 frozenEnvelopeSessions 字段」落在同一位置,按双方并集解(纯增量字段)。rebase 后 typecheck clean,新套件 6/6 + 邻近 storage 套件 21/21 本地绿(全量回归归 CI)。新 head: 5bfcc0f(PR 描述已附 rebase note)。

⑤b 代码整洁硬标准: 全项过——命名/死代码/重复/错误处理上下文/魔法数字(DRIFT_ESCALATE_STREAK 具名常量)/注释讲为什么/风格一致/零新增依赖。

行为变更披露(old → new)

面 旧 新
drift latch 下的冷加载 legacy envelope 存在即无条件 serve index 严格更多 active blocks 时 serve index(WARN 点名两侧计数);tie/不可读保持旧行为
连续 3 次采样漂移不收敛后 继续写共享文件(互写 churn 持续) 本进程对该会话冻结 envelope 写入直到干净样本再准入;unified store 照常收每次保存

prefix-cache/retry/watermark/dedup-key 语义均未触碰;无新配置面(仅内部常量)。

八问

  1. 是什么问题: 多实例互写下 JSON envelope 与 SQLite index 永久分叉,drift latch 下每次冷加载都无条件 serve 旧 envelope,已折叠内容(事故中 ~240K token)回灌上下文。
  2. 什么原因导致的: 多个 bili 实例共享一个 sessions dir(无单实例守卫:多个 bili 代理进程并存共写 sessions 目录,伴随孤儿 ref、有注册表无 transcript、计数自相矛盾的污染簇 #394 反模式)互写两份视图使其永不收敛;旧 fallback 假设「最后一次文件写入更新」,互写下该假设不成立。
  3. 影响范围与严重程度: 仅共享目录的多写部署会触发(单实例部署不走此路径);一旦触发,每次重启/驱逐冷加载都丢折叠覆盖、上下文膨胀成本上升——缓存泄露类,p1。
  4. 怎么修复的: 按次数升级(连续 3 次采样漂移 = 结构性分叉签名)→ 单条响亮告警 + 冻结本进程 envelope 写入以止住互写 churn 源头;latched 加载改「取折叠更多的一侧」,以折叠进度为新鲜度代理。
  5. 是否有回归问题: 未发现(逐项清点见 ②);error 行字节不变、watermark 自愈、无 latch 加载、writerOff 路径等既有行为全部保持。
  6. 是否破坏缓存: 否——此修复正是止住已折叠内容回灌;prefix-cache/retry/watermark 机制未变。
  7. 是否改了设计方向: 否——冻结是 per-process 内存态(刻意不持久化),根治仍在运营层单实例纪律 + [PR] storage: durability trio — open-time quick_check + online VACUUM INTO backup + second-process lockfile (#4175) #4177 拓扑检测;本 PR 只做可观测与止损。
  8. 是否引入新配置: 否——仅内部常量 DRIFT_ESCALATE_STREAK=3,无 env/config/CLI 面。

CI 状态与合并判定

@ranxianglei
ranxianglei merged commit 3221859 into master Oct 23, 2026
74 of 118 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.751-master.3221859 (rolling master tag).

npm install -g billion-context@0.0.751-master.3221859  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p1 scope/storage

Projects

None yet

1 participant