Skip to content

feat: one-shot EPERM diagnostic alert for Windows + AV-exclusion docs (#367) - #373

Merged
ranxianglei merged 1 commit into
masterfrom
2026-08-31_persist-eperm-diagnostic-alert
Sep 5, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-08-31_persist-eperm-diagnostic-alert

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Summary

Implements the two remaining layers of the #362 root-cause fix (Layer 1 resilience already landed in acp-kernel PR #172):

  • Layer 2 — diagnostic alert (main): when persist fails N consecutive times (default 5, BILI_PERSIST_EPERM_ALERT_THRESHOLD) with a file-lock error (EPERM/EBUSY/EACCES) on the same session id, emit one actionable warning naming the sessions dir and pointing at the Windows Defender exclusion (+ OneDrive/sync warning). One-shot by default; optional low-frequency repeat via BILI_PERSIST_EPERM_ALERT_REPEAT_MS. Windows-only + lock-codes-only, so non-Windows / non-lock failures never alert.
  • Layer 3 — docs: Windows Defender exclusion steps + OneDrive warning in README/CONFIGURATION (EN + zh-CN).

How detection works

The kernel's debounced scheduleSave path swallows write errors, so its [persist] (write|flushSync|shutdown flush) failed for <id>: <err> log lines are the only signal for the hot path. SessionStore now wraps the StateStore log callback with a PersistEpermAlert detector (src/persist-eperm.ts) that counts per-id consecutive lock failures and emits the alert. This works against the pinned acp-kernel 0.0.46 — no cross-repo change needed.

⚠️ Heads-up: Layer 1 is not yet in the pinned kernel

The pinned acp-kernel 0.0.46 does not contain PR #172 (spill-to-<name>.fb.json / error rate-limiting) — verified against the installed dist. PR #172 is only in acp-kernel master / prerelease 0.0.46-pr.172.23. So until acp-kernel ships a stable with PR #172 and this repo bumps the pin, published billion-context still reproduces the #362 data-loss + log-flooding. This PR's Layer 2 is independent and works on 0.0.46 (it makes the lock diagnosable), but the data-loss resilience still needs the kernel bump. I'll track that as a separate issue.

Tests

  • New tests/persist-eperm.test.ts — 11 unit tests: alert fires exactly at the Nth consecutive EPERM; below-threshold / non-lock / non-error-level / non-Windows → no alert; one-shot; rate-limit cadence; per-id counting; all 3 kernel failure kinds; no false match on unrelated lines; alert message content.
  • tests/proxy-persist.test.ts — integration: a real SessionStore routes a write failure through the detector with no false alert on a non-lock error.

Pre-flight

  • npm run typecheck — clean
  • npm test — 862/862 pass
  • npm run build — success

Closes #367 (Layer 2 + Layer 3).

@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-08-31_persist-eperm-diagnostic-alert (7bc107f)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-373

Each push to this PR publishes a new version under the pr-373 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr373.tgz
npm install -g package

This comment is automatically updated on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · vllm-qwen/qwen3.8-27b

[bot] 已 checkout PR 分支 2026-08-31_persist-eperm-diagnostic-alert (2a2122f) 并独立复核了 pre-flight 与检测机制(非转述 PR 声明)。结论:LGTM,可合并(合并由人工操作,我不合并)。

独立验证

检测机制核对(对已安装 acp-kernel 0.0.46 dist 逐行验证)

  • kernel persist 共 4 种失败日志:write failed for / flushSync failed for / shutdown flush failed for(写/锁失败)+ builder failed for(builder 抛异常,非锁)。检测器正则 /^\[persist\] (?:write|flushSync|shutdown flush) failed for (.+?): (.+)$/ 精确匹配前 3 种、正确排除 builder。
  • 热路径确认:scheduleSave → debounce → writeNow → writeInner;writeInner 在 catch 里先 log("error", "[persist] write failed for <id>: <err>") 再 throw,而 scheduleSave 的 .catch(() => {}) 吞掉的是 rejection——所以该日志行一定被 observe 看到。核心假设成立。
  • errText(e) 返回 e.message(即 EPERM: operation not permitted, rename ...),LOCK_CODES(\b(EPERM|EBUSY|EACCES)\b)能命中;renameWithRetry 对锁码重试 3 次后 throw 最后错误 → 终态错误带锁码。✓

代码质量(对 AGENTS.md)

  • 无 as any / @ts-ignore;告警走 loggerLog("warn", ...)(非 console.error)✓
  • 可测设计:platform/now/onAlert 可注入。11 单测覆盖验收标准全部(第 N 次才告警 / 低于阈值 / 非锁 ENOENT / 非 error 级 / 非 win32 / 一次性 / 限流节奏 / 按 id 计数 / 三种失败类型 / 无关日志不误匹配 / 文案内容)+ 1 集成测试(真实 SessionStore 把写失败路由过检测器,非锁错误不误告警)✓

Layer 1 提示属实(重要)

已核对安装的 0.0.46:writeInner 最终失败仍 fsp.unlink(tmp)(丢数据)、无 spill 到 <name>.fb.json、无 error 限流。即 PR #172 韧性层确不在 pinned 0.0.46,已发布的 billion-context 仍会复现 #362 的丢数据+刷屏,直到 acp-kernel 发 stable + 本仓 bump pin。本 PR 的 Layer 2 独立于 0.0.46(让锁「可诊断」),披露准确,跨仓依赖已另开 #374 跟踪。✓

非阻塞小 nit(不挡合并,供后续)

  1. 计数语义:counts Map 按 id 单调递增、成功写不重置 → 实为「累计」而非严格「连续」。默认一次性告警下无实际影响;仅 "failed N consecutive writes" 措辞略有出入。
  2. counts Map 无上界:只记录命中过锁码的 id(罕见),长驻进程下理论上无界,实际可忽略。
  3. 集成测试跨平台脆弱性:tests/proxy-persist.test.ts 用「在 sessions 目录路径写一个文件」制造写失败(linux 下 ENOTDIR)。若在 Windows 上跑,同一操作可能产生 EPERM(锁码)→ 会触发检测器并让「非锁不误告警」断言失败。CI 是 ubuntu,故仅为潜在脆弱性。

结论:PR 完整实现 #367 的 Layer 2 + Layer 3,检测机制经对内核逐行核对成立,pre-flight 全绿,代码符合规范,Layer 1 跨仓依赖披露准确且已跟踪。可合并(rebase 到最新 master 后)。合并请人工操作:#373

@ranxianglei
ranxianglei force-pushed the 2026-08-31_persist-eperm-diagnostic-alert branch from 2a2122f to 89b4bba Compare August 31, 2026 02:35
@ranxianglei

Copy link
Copy Markdown
Owner Author

Conflict-resolved against master (kernel 0.0.53 + #559) — branch force-updated, head 3c25a9e.

What was resolved

  1. CONFIGURATION.md — union: kept master's BILI_TUNNEL_ALLOWED_HOSTS row alongside the two new BILI_PERSIST_EPERM_ALERT_* rows (grouped after BILI_PERSIST_DEBOUNCE_MS).
  2. tests/proxy-persist.test.ts — kept master's #499 pfa-migration test and this PR's detector wire test.

Compatibility fix for acp-kernel 0.0.53 (just released, pin bumped in #560)

  • recordFailure still emits the exact [persist] write failed for <id> (total <N>x): <err> shape the detector parses — regex verified against node_modules/acp-kernel@0.0.53. ✅
  • Bug found & fixed while resolving: the wire test's await settle(50) is too short now — 0.0.53 retries the whole write cycle on transient ENOTDIR (~1.5s backoff ladder) before the failure line is ever logged, so the test would fail deterministically. Rewrote to the waitFor poll (5s deadline) pattern from test: poll instead of fixed sleeps for debounced persist writes (Windows CI flake) #559.

Pre-flight

  • npm run typecheck ✓
  • npm test 1058/1060 (2 known env-dependent fails in plugin-agent.test.ts, unrelated)
  • npm run build ✓

Ready to merge — closes #367 (Layer 2 + Layer 3; Layer 1 shipped in acp-kernel 0.0.47→0.0.53).

- CONFIGURATION.md: keep both the EPERM alert knobs and master's
  BILI_TUNNEL_ALLOWED_HOSTS row.
- tests/proxy-persist.test.ts: keep master's #499 pfa migration test and
  the PR's wire test; the wire test now polls (waitFor, 5s) for the
  kernel failure line because acp-kernel 0.0.53 retries the whole write
  cycle on transient ENOTDIR (~1.5s) before logging it — a fixed
  settle(50) would fail.

Verified against pinned acp-kernel 0.0.53: recordFailure emits the same
'[persist] write failed for <id> (total <N>x): <err>' shape the detector
regex parses; EPERM/EBUSY/EACCES lock codes remain non-transient enough
to count toward the alert.

Pre-flight: typecheck ✓, tests 1058/1060 (2 known env fails in
plugin-agent.test.ts), build ✓.
@ranxianglei
ranxianglei force-pushed the 2026-08-31_persist-eperm-diagnostic-alert branch from 3c25a9e to 7bc107f Compare September 5, 2026 15:07
@ranxianglei

Copy link
Copy Markdown
Owner Author

上一条推送用了 merge 方式解冲突,导致 GitHub Files changed 显示 ~115 个文件(把 master 这一周的演进也显示进去了,非本 PR 内容)。已改为 rebase 到 master(7bc107f,线性单提交),现在 Files changed 就是本 PR 真实的 8 个文件 +321/-1,内容与上一条评论的解决方案完全一致(含 0.0.53 waitFor 兼容修复)。34/34 targeted tests ✓ typecheck ✓ build ✓

@ranxianglei
ranxianglei merged commit 4ef3444 into master Sep 5, 2026
6 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] Post-merge verification done. Heads up first: this PR is already merged into master (4ef3444) and shipped as v0.1.86 (release PR #561), so I treated this as verifying the shipped content rather than gating a merge. There's nothing for me to block or merge.

Verdict: LGTM ✅ — the 0.0.53-compatible detector is correct.

The only real risk the rebase onto kernel-0.0.53 master introduced was whether the alert still parses the changed kernel log line and keeps a stable session id for counting. I checked the actual acp-kernel@0.0.53 dist (dist/persist/index.js, recordFailure) rather than relying on the "regex verified ✅":

  • Kernel now emits [persist] write failed for <id> (total <N>x): <err>; …spillover…. The id is printed clean before (total, and the regex's non-greedy id group stops there → the captured id is byte-stable across lines, so per-id accumulation works. ✅
  • It now reads the kernel's own consecutive counter (the (total N x) group) instead of self-counting — which also resolves my earlier round-1 note about cumulative-vs-consecutive counting. The kernel resets the count on success, so it's genuinely "consecutive". ✅
  • Accepting both error and warn levels is load-bearing and correct: the kernel logs only count=1 at error; every later line (including the threshold-crossing one) is warn. Gating on error alone would have made it silently never fire after the first line. ✅
  • Lock-code gate (EPERM|EBUSY|EACCES) and Windows-only are retained; one-shot/rate-limit moved to a global hasAlerted/alertedAt. With the default threshold of 5 and the kernel logging at powers-of-two, the alert fires at the first reported count ≥ 5, i.e. 8. Fine.

Pre-flight (run independently on the merged content, byte-identical to master):

  • npm run typecheck — clean
  • npm run build — success (dist/index.js 2.63 MB, acp-kernel inlined, zero runtime deps)
  • targeted tests/persist-eperm.test.ts + tests/proxy-persist.test.ts — 34/34 pass, including the rewritten wire test driving the real 0.0.53 StateStore → SessionStore → detector through the ENOTDIR retry ladder (confirms routing + no false Defender alert on a non-lock error)
  • full npm test — 1059 pass / 1 fail

The single failure is unrelated to this PR and is environment-dependent:
not ok 451 - resolveClientCommand: codex/claude resolve to themselves (tests/launcher.test.ts:1028, logic at src/launcher.ts:1608). It calls resolveClientCommand("codex", { PATH: "/usr/bin" }) expecting the bare name codex, but got /usr/bin/codex — because a codex binary happens to be present under /usr/bin in this environment, so resolveOnPath finds it and returns the absolute path. On a clean runner without /usr/bin/codex it returns the bare name and passes. This PR doesn't touch launcher.ts at all. (Your run instead flagged 2 env-dependent fails in plugin-agent.test.ts; the specific failing set just varies by what's installed locally — none are in this PR's 8 files.) Nothing to do on this PR; happy to make that launcher test env-agnostic in a separate change if you'd like.

Two minor doc nits (non-blocking, report-only):

  1. BILI_PERSIST_EPERM_ALERT_REPEAT_MS — the table row says "in ms" but the prose says "every M minutes" (CONFIGURATION.md:666, CONFIGURATION.zh-CN.md:659). The variable is milliseconds.
  2. The "default 5" wording — due to the kernel's power-of-two rate-limiting it actually fires at the first reported count ≥ 5 (= 8), not exactly 5. Cosmetic.

I can open a tiny docs-fix follow-up PR for these two if you want them cleaned up.

Cross-repo status: Layer 1 (spill-to-.fb.json + rate-limiting) shipped in acp-kernel 0.0.47 → 0.0.53, and the billion-context pin bump to 0.0.53 already landed as #560 (in master). So the kernel-bump dependency you were tracking (#374) looks satisfied — worth closing #374. Net: #367 is fully closed by #373 (L2+L3) + #560 (L1 pin).

Nothing here blocks anything (already released). Let me know if you want the two doc nits fixed up.

@ranxianglei ranxianglei mentioned this pull request Sep 5, 2026
ranxianglei pushed a commit that referenced this pull request Sep 7, 2026
…session state

Dual-instance rollback guard (#405 fix #4, tracked in #598): when two bili
processes share BILI_SESSIONS_DIR, the instance holding an older in-memory
copy must not overwrite the fresher on-disk state. Monotonic signals
(stats.requests, state.nextBlockId) prove staleness; a stale save is turned
into a content-identical rewrite of the disk payload (keeping the newer
state) with a throttled one-minute warn.

Rebased onto master (v0.1.92 era): loadEnvelope() probe extracted from
loadSync keeps the #408 negative-clamp reload path; constructor unifies
this.log with the #373 PersistEpermAlert baseLog.

Fixes #598
@ranxianglei
ranxianglei deleted the 2026-08-31_persist-eperm-diagnostic-alert branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

persist EPERM 根治(#362 follow-up):可诊断告警 + Windows AV 排除文档

1 participant