Skip to content

[PR] fix: auto-quarantine a corrupt index.db instead of degrading to legacy-only forever - #3965

Open
ranxianglei wants to merge 3 commits into
masterfrom
2026-10-20_corrupt-index-selfheal
Open

ranxianglei wants to merge 3 commits into
masterfrom
2026-10-20_corrupt-index-selfheal

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Closes #3959
Family: #3788 (new sub-mechanism entry: corrupt index.db file — engine healthy)

Behavior change (old -> new)

OLD: when index.db failed to open with a genuine corruption error (database disk image is malformed / file is not a database), the sidecar degraded to legacy-only on every boot until an operator manually deleted the file ("delete the file to reset the index"). Field report #3959: one corrupt db pinned the machine to legacy-only across repeated restarts.

NEW: on a corruption-class open failure the dead file is QUARANTINED (renamed in place to index.db.corrupt-<timestamp>, never deleted, kept for forensics) and a fresh index opens immediately. The legacy JSON session files remain the source of truth in both shadow and full mode (full keeps writing them), so sessions re-ingest as they are saved/loaded — nothing is lost. The warn log now names the quarantine path.

Unchanged by design:

  • Non-corruption failures (CANTOPEN/AV lock/OneDrive/integrity label SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, win32-x64) #3823, BUSY, permission, no-engine) keep today's EXACT degrade path and log wording byte-for-byte — there the file may be perfectly fine behind an OS lock, so quarantining would move a healthy file aside.
  • The mid-run circuit breaker (3 consecutive record() failures disables the sidecar for the process) is untouched; after this fix the next boot self-heals instead of re-failing.

Mechanism

No config surface change. No kernel change. No wire-shape change (log wording only).

Tests

New hermetic suite tests/storage-corrupt-index-selfheal.test.ts (both directions pinned, #3617 discipline):

  • non-SQLite bytes in index.db -> quarantined (original bytes preserved verbatim), fresh index opens, wire reports healthy shadow
  • truncated real db (malformed) -> same recovery
  • CANTOPEN (directory named index.db) -> degrades, NO quarantine side effect
  • no-engine failure -> degrades, NO quarantine side effect

Existing test updated (intentional, per the behavior change above): tests/storage-review-fixes.test.ts "review 3" (#2771) pinned the OLD contract for corruption-class errors — permanent legacy-only degrade until manual delete. It now pins the self-heal contract (exactly one quarantined copy, fresh index present, sidecar available, no re-quarantine on the next process); its original safety assertions (SessionStore construction must not throw, legacy load still succeeds) are kept verbatim. The non-corruption degrade path stays pinned by storage-cantopen-hint + the negative cases above.

Local targeted runs green on the rebased head: the new suite + storage-review-fixes + the whole storage-subsystem set (146 tests) + adjacent suites; typecheck clean. Full-suite verdict = CI on this head.

Rebased onto current master after the MySQL-backend step-3 merges (#3955/#3966) landed in shadow.ts; conflict resolved by re-landing the four surgical changes on the new structure (no semantic interaction — the mysql async path does not touch the sqlite open branch).

@ranxianglei ranxianglei changed the title fix: auto-quarantine a corrupt index.db instead of degrading to legacy-only forever [PR] fix: auto-quarantine a corrupt index.db instead of degrading to legacy-only forever Oct 20, 2026
@ranxianglei
ranxianglei force-pushed the 2026-10-20_corrupt-index-selfheal branch from 91ef989 to 89d8159 Compare October 20, 2026 12:40
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_corrupt-index-selfheal (1158918)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4814-pr.3965.ad5885f

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3965.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

Rebased onto current master (docs-sync red was the pre-#3969 master state; the rebased head carries the regen). Zero conflicts; tsc clean; corruption-quarantine + storage-review-fixes + storage-unified suites green locally.

@ranxianglei
ranxianglei force-pushed the 2026-10-20_corrupt-index-selfheal branch from 89d8159 to e3d2701 Compare October 20, 2026 13:43
awork added 2 commits October 21, 2026 00:32
…no longer degrades permanently)

The pre-existing #2771 review test asserted the OLD contract for
corruption-class open failures: permanent legacy-only degrade until manual
file deletion. #3959 deliberately changes that class to auto-quarantine +
fresh reopen, so the expectation flips from replicasAvailable()==false to
true, plus quarantine-file assertions. The original safety property
(SessionStore construction must NOT throw / legacy load still works) is
kept verbatim.
@ranxianglei
ranxianglei force-pushed the 2026-10-20_corrupt-index-selfheal branch from e3d2701 to 11fae9e Compare October 20, 2026 16:36
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解,分支已 rebase 到当前 master 并 force-push:e3d270112 → 11fae9e67,GitHub 侧 mergeable 已恢复。

八问:

  1. 是什么问题:master 前进后本 PR 分支与目标分支出现合并冲突(dirty),需要把新 master 并进 PR 分支。
  2. 什么原因导致的:自 merge-base(74090ac)以来 master 前进了 45 个提交;其中唯一触碰 src/storage/ 的是 dist/index.js crashes on startup: mysql2 is statically inlined into the bundle, so deployment modes without node_modules | dist/index.js 启动即崩:mysql2 被静态打进 bundle,不带 node_modules 依赖的安装形态(dsh e2e、dist-only 嵌入)直接挂 #4008(mysql2 驱动改懒加载,动 backend.ts/mysql.ts)。与 PR 改的文件(driver.ts/shadow.ts/两个测试文件)无文本交叠,所以 rebase 零冲突一次通过——之前的 dirty 状态是 master 移动造成的位置性冲突,非语义冲突。
  3. 影响范围与严重程度:仅 PR 分支自身的可合并性,不涉及任何代码行为变化;对已发布版本零影响。
  4. 怎么修复的:标准冲突唤醒流程——rebase 当前 master(66a719f)→ 验证无语义交叠 → 本地门禁 → force-with-lease push。
  5. 是否有回归问题:无。逐项核过:①PR diff 在 rebase 后原样保留(4 文件,+254/-14,与推送前一致);②dist/index.js crashes on startup: mysql2 is statically inlined into the bundle, so deployment modes without node_modules | dist/index.js 启动即崩:mysql2 被静态打进 bundle,不带 node_modules 依赖的安装形态(dsh e2e、dist-only 嵌入)直接挂 #4008 与本 PR 触碰文件不相交,无语义叠加风险;③rebase 后本地 typecheck 干净;④定向套件全绿:storage-corrupt-index-selfheal(4/4)+ storage-cantopen-hint / acp-status-storage-degraded / storage-unified / storage-diagnostics-lines / storage-mode-label(26 pass / 1 skip)。全量回归以 CI 在本 head 的结果为准。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:冲突闭环,PR 回到可合并状态,等 CI 对新 head 跑绿即可合入。

CI auto-fix (issue #3965): two pre-existing pins asserted the OLD
corrupt-index behavior (permanent legacy-only degrade) that this PR
intentionally replaced with quarantine + fresh-open self-heal:

- web-storage-banner (#3040): now asserts effective=full, degraded=false,
  live engine, and the .corrupt-<ts> quarantine artifact with the exact
  corrupt bytes preserved.
- storage-engine-proofs (#2771): re-pointed the #2781 handle-release
  roundtrip to AFTER closeQuietly — under self-heal the store legitimately
  holds a live index.db handle until close, and Windows refuses to move a
  directory containing open handles (the old mid-test placement threw
  there, skipping cleanup and surfacing as a masked EBUSY in rmrf).
  Adds the forensics assertions (one quarantine copy, bytes verbatim).
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI auto-fix done (attempt 1/2): pushed 1158918 to the PR branch — two stale test pins that this PR's intentional behavior change broke are now realigned to the new contract. Both pass on CI on every lane they ran, including Windows.

What was actually red: your rebase already fixed docs-sync. The remaining failures split into two groups:

  1. In-scope (fixed): two old tests still asserted the old "corrupt index → degrade to legacy forever" contract:
  2. Out of scope (pre-existing, not from this PR): the #3769 pair still flakes on 4/6 lanes despite [PR] fix: bare-id loadSync hydrates pooled envelopes without the unified index (master CI flake, #4014) #4015 being in the base — identical red existed on master before this PR's rebase (master@e98ccda, all shards), and the logs show TLS-handshake/circuit-breaker churn right before the failing assertions under shard load. That's the owner-active [PR] fix: bare-id loadSync hydrates pooled envelopes without the unified index (master CI flake, #4014) #4015 thread; not a small auto-fix-lane item. One-off flakes also observed, each green elsewhere: plugin-fork hang (previous head only), responses-ws-http-fallback file kill (this head, win22-s2 only, WS-timing hang, first occurrence), 任何一轮没有拿到 usage 报告的请求,都会把上下文分母放大 12–20×(nudge 读到 >1000% usage、假 EMERGENCY、基线冻结数分钟) #1839 G3 (old head only, green since).

八问

  1. 是什么问题:PR CI 红——两个旧测试还钉着本 PR 有意替换掉的"索引损坏→永久降级 legacy"契约(其中 [Bug][storage] corrupt index.db bricks SessionStore construction in shadow/full mode (violates the sidecar never-breaks-legacy policy) #2771 那个在 Windows 上是句柄生命周期问题被清理报错掩盖);另有 master 既有的 [Bug] MCP failed compression record rotation prematurely rewrites the history prefix, triggering expensive cache recompu | [Bug] MCP 失败压缩记录轮换提前改写历史前缀,触发大额缓存重算 #3769 flake。
  2. 什么原因导致的:① 作者本地定向跑漏了这两个文件,pin 没跟着新契约更新;② [Bug][storage] corrupt index.db bricks SessionStore construction in shadow/full mode (violates the sidecar never-breaks-legacy policy) #2771 pin 隐含了旧路径的假设(降级时根本没有打开的句柄),self-heal 后 store 持有活句柄直到 close,Windows 不允许移动含打开句柄的目录;③ [Bug] MCP failed compression record rotation prematurely rewrites the history prefix, triggering expensive cache recompu | [Bug] MCP 失败压缩记录轮换提前改写历史前缀,触发大额缓存重算 #3769 = CI 负载下的 TLS/重试竞态,master 上早已存在([PR] fix: bare-id loadSync hydrates pooled envelopes without the unified index (master CI flake, #4014) #4015 已并入 base 但负载下仍偶发)。
  3. 影响范围与严重程度:无用户影响——本次提交只动两个测试文件,产品代码零改动;严重度仅限 CI 门禁。
  4. 怎么修复的:把两个 pin 对齐到新契约(断言 + 测试编排),并补隔离取证断言(恰好一份 .corrupt-* 副本、字节逐字保留、旁边是全新 index.db);不碰任何 src。
  5. 是否有回归问题:审过既有行为清单——wire 契约不变、非损坏类失败的原降级路径不变(其 pin 在 CI 全绿)、self-heal 套件全绿;本次 diff 仅 2 个测试文件,无行为面变更。
  6. 是否破坏缓存:不涉及(未触碰缓存/重试/游标逻辑)。
  7. 是否改了设计方向:不涉及(PR 原设计"日志措辞级变更"保持不变)。
  8. 是否引入新配置:无。

判定:lane 内可修项已全部修完并在 CI 验证(ubuntu-22-1、windows-22-1 两条此前红的 lane 现已全绿);PR 整体转绿取决于 master 侧的 #3769 flake 收敛(#4015 线程进行中)或红 lane 重跑运气。按 auto-fix lane 约定停在这里,不再消耗第 2 次尝试;合并仍归人。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant