Skip to content

[PR] fix(logger): log file open failure now warns + falls back to stderr instead of total silent loss - #3858

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-19_logger-silent-loss
Oct 20, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-19_logger-silent-loss

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: vllm-qwen/qwen3.8-27b)

Closes #3857

What & why

Discovered while working #3823 (win32 unified-storage degrade with ZERO [storage] lines in the log). The tee logger's fail-open design had two holes that combine into total silent log loss for launcher-spawned children:

  1. createWriteStream opens asynchronously. A failed first open surfaced as an 'error' event that only destroyed the stream — the one-shot reopen-failure warning ([warn] log file ... unavailable ... continuing with stderr-only logging) lived only in getStream()'s try/catch and was unreachable for async failures.
  2. isFileLoggingActive() returned true for a PENDING stream whose fd had not opened yet. The [noise] 观测噪音三件:启动横幅同秒重复 24 次 / reasoning-pair-violated 每请求重报(单会话 276 条) / %TEMP% 端口日志无轮转(5 天 31.7MB) #2864 launcher mute consulted it right after configureLogger(), so it muted stderr before the open result was known. On open failure: file leg dead, stderr leg muted → every subsequent line (storage warnings, engine bootstrap, everything) vanished from every sink.

Fix

Behavior change (disclosed)

  • Launched children (BILI_LAUNCH_TOKEN set): stderr mute timing moves from "startup, stream object exists" to "log file actually opened". If the first open fails: OLD = silent forever (both sinks dead); NEW = one [warn] line on stderr + all logging continues stderr-only (which is where it's redirected to %TEMP%/bili-proxy-<port>.log).
  • Manual starts: byte-identical (no mute path).
  • Rotation/clobber recovery: unchanged semantics, same single warn per episode.

Tests

  • NEW tests/logger-silent-loss.test.ts: dir-as-log-path = deterministic cross-platform ASYNC open failure (EISDIR/EPERM); worst-case pre-mute simulation; asserts one warn per episode, late lines reach restored stderr, isFileLoggingActive()===false; hook exactly-once / never-on-failure / immediate-on-late-registration.
  • tests/logger-stderr-mute.test.ts test 1 waits for a proven open before asserting activity (old assertion encoded the racy pending-stream semantics).
  • Local targeted: all six logger suites 17/17 green; launcher suites 332/332 green; typecheck clean. Full-suite verdict = remote CI.

@ranxianglei ranxianglei changed the title fix(logger): log file open failure now warns + falls back to stderr instead of total silent loss [PR] fix(logger): log file open failure now warns + falls back to stderr instead of total silent loss Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-19_logger-silent-loss (e5ed953)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4582-pr.3858.1ba8e94

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3858.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b)

Rebased onto current master (48d45e7) — the branch was ~30 merges behind, including the #3818/#3819 storage-mode flip and #3790 bootstrap diagnostics. Rebase was conflict-free; logger suites green locally (logger-silent-loss + logger-stderr-mute + log-mask, 25/25) and typecheck clean on the new base.

Two labeled drive-by commits added on top (no behavior change), because master itself is currently red on two gates this PR's CI also runs — reproduced locally on pure master, so this is inherited redness, not from this diff:

  1. docs(config) — the committed CONFIGURATION.md / .zh-CN.md still carry two superseded storage table rows each; tools/gen-config-docs.mjs generate deletes exactly those 4 lines (the master-red docs-sync gate).
  2. fix(log-mask) — maskUrlForPublish is exported but every reference sits inside src/log-mask.ts ([security][web] Diagnostics blob prints private relay hosts verbatim in the config section — the "secrets masked" header only masks values, not hosts | 诊断包 config 段原样输出内网中转地址:标题说"密钥已遮蔽",但只遮值不遮主机 #3811); dropping the leading export restores the master-red unused-exports gate.

No new issue filed for these (few-line fixes riding an in-flight PR); flagging so the same two commits can ride whichever in-flight PR lands first.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 检索 open+closed issue 与近期 PR(按失败签名:logger 静默丢失 / 异步 open 失败 / stderr mute 竞态),无既有覆盖;相邻工作 #2864(launcher 子进程静音 stderr)、#1233(EPIPE 风暴降级)、#1718(日志脱敏)机制均不同——#3857 线程已做过同样比对,复核一致。本 PR 是 #3857 的修复 PR,一 issue 一 PR,无并列重复修复。

标签: pr / bug / scope/other / p2 —— 缺陷类;logger 属宿主基础设施(kernel/wiring/storage/ui/ci/docs 均不属);与 #3857 定级一致(触发时整条日志腿静默全灭、可用性不受损 → p2);不打客户端标签(跨所有 launcher 车道)。

开始审核:拉取 pr-3858(head 400afa6)独立读 diff、核对 #3857 的三条修法是否落地、跑定向测试(logger/launcher 套件 + typecheck),CI 状态稍后补。PR 当前 behind master,会先 rebase 到最新 master 再下结论。

@ranxianglei ranxianglei added bug Something isn't working p2 scope/other labels Oct 19, 2026
awork added 5 commits October 20, 2026 02:38
…lent loss (#3857)

Two fail-open holes in the tee logger's degradation path:

1. createWriteStream opens asynchronously; a failed first open surfaced as an
   'error' event that only destroyed the stream — no warn was ever emitted and
   the one-shot reopen-failure warning stayed unreachable for the most common
   failure class (the log directory not being creatable at startup).

2. isFileLoggingActive() returned true for a PENDING stream whose fd had not
   opened yet. The launcher child mute (#2864) consulted it right after
   configureLogger(), so it muted stderr before the open result was known; on
   open failure both sinks were dead and every subsequent line — including the
   storage degradation warnings — vanished with no trace in any file.

Fix: route stream 'error' events through the existing one-shot warning
(stderr-only fallback + unmute), and move the mute decision behind a new
onFileLegLive() hook that fires once per PROVEN open (fd captured). The
reopenWarned reset moves from stream creation to the proven-open point so one
degradation episode warns exactly once. isFileLoggingActive() now requires a
captured fd.

Behavior change (disclosed): launched children (BILI_LAUNCH_TOKEN set) now
mute stderr only after the log file has actually opened; if the first open
fails, the child logs one [warn] to stderr and continues stderr-only instead
of going silent. Manual starts are byte-identical.

Tests: new tests/logger-silent-loss.test.ts (dir-as-log-path = deterministic
cross-platform async open failure; worst-case pre-mute; one warn per episode;
late lines reach restored stderr; hook exactly-once / never-on-failure /
immediate-on-late-registration); tests/logger-stderr-mute.test.ts test 1 now
waits for a proven open before asserting activity (the old assertion encoded
the racy pending-stream semantics).
…ed-exports gate)

Drive-by fix, labeled: master 48d45e7 fails the unused-exports gate on this
symbol (INTERNAL_ONLY — every reference sits inside src/log-mask.ts: its own
doc comment and the maskUrlsInText call added by #3811). Dropping the leading
export restores the gate; no behavior change.
…ows (master-red docs-sync gate)

Drive-by fix, labeled: master 48d45e7 fails docs-sync because the committed
CONFIGURATION.md / .zh-CN.md still carry two superseded table rows each (the
old storage.unified 'unset (shadow)' row and an old storage.sqliteAutoDownload
row without the #3788 cooldown text). tools/gen-config-docs.mjs generate
deletes exactly those 4 rows; no other content changes.
…en event (#3857)

The windows-latest node-22 CI shard reproduced the #3857 field shape
(win32 hands out a valid fd for paths whose writes then always fail —
e.g. a DIRECTORY): the shipped version keyed the degradation-episode
reset and the onFileLegLive hook on the 'open' event, so every retry
cycle re-armed the one-shot warn (warn storm) and the hook fired over a
never-delivering leg — which would have muted stderr (#2864) over a dead
file leg.

Liveness is now proven by the first DELIVERED write:
- 'open' only records the fd (orphan-inode detection); it no longer
  resets reopenWarned or fires the hook;
- noteProvenWrite() ends the degradation episode and fires the hook once;
- isFileLoggingActive() / onFileLegLive() immediate-fire require the
  proven state; error/rotation/close paths reset it.

Behavior change (disclosed): launcher-spawned children engage the
stderr mute after the first byte actually lands in bili.log instead of
after the first open completes — strictly safer (never mutes over a
non-deliverable leg); manual starts are unaffected. If no line ever
reaches the file, stderr stays unmuted (duplicate temp-log noise in a
degenerate zero-line case, never silent loss).

Tests: the dir-as-log-path fixture now pins the contract on DELIVERY on
every platform (POSIX fails the open EISDIR; win32/node-22 opens fine
and only writes fail); mute-suite test 1 seeds a write before asserting
liveness. Local: 17/17 logger suites green, typecheck clean.
…iveness waits

The three #3857/#2864 tests waited on isFileLoggingActive() (fd established
on the 'open' event) before asserting delivery-based state (hook fired /
leg live). 'open' precedes the first flushed write, so the fd can be visible
while no byte has been delivered yet - measured 22% miss rate at tight
polling. The 20ms test polling masked the window locally but it is not
deterministic under load. Wait on actual delivery instead: hook fire count
(hook registered pre-seed) or a byte on disk (existsSync-guarded stat),
which is monotonic and race-free.
@ranxianglei
ranxianglei force-pushed the 2026-10-19_logger-silent-loss branch from 400afa6 to e5ed953 Compare October 19, 2026 18:50
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核完成(查重结论见首条回复:不重复)。本轮为完整审核结论。

独立审核动作(未照抄 PR 描述):

  1. PR 原分支落后 master 12 个提交——已 rebase 到当前 master(干净无冲突,含 src/server.ts 热文件)并推回 PR 分支,head 现为 e5ed953e2;
  2. 逐行读了 src/logger.ts 全文(397 行)、src/server.ts 静音块、两个 bundled gate commit(log-mask 私有化、config docs 去重)、新测试与改动测试;全仓 grep 确认被改 API(isFileLoggingActive / onFileLegLive / maskUrlForPublish)的调用面;
  3. 本地定向验证:logger 六套件 17/17、launcher 五套件 403/403、typecheck 与 build 干净;全量回归以远程 CI 为准;
  4. 发现并直接修掉一个测试确定性缺陷(独立标注 commit e5ed953e2,见下);
  5. 远程 CI(e5ed953):全部 48 个 check run success/skipped,含 static-gates 与 ubuntu/windows × node22/24 六个 test 分片。

发现的问题(已修):三个 #3857/#2864 测试用 isFileLoggingActive()(fd 在 'open' 事件建立)当等待条件,却断言交付侧状态(hook 触发/腿可用)。Node 的 'open' 事件先于首个缓冲写入落盘——我在 Linux 上实测该窗口真实存在:1ms 轮询下 100 次里 22 次 fd 可见但一个字节都还没写出去(20ms 轮询 0/100;测试自身 20ms 轮询把窗口盖住了,负载更高的 CI 上不保证)。修法:等待条件换成单调不可逆的交付证据(hook 计数 / 文件出现且 size>0),修后 6/6 × 连续 6 轮全绿。PR 原状在繁忙 CI 上有偶发红的可能,属 §7.2「测试必须确定性」,故直接修进 PR 分支而非只报告。

bundled commits 独立核查:① maskUrlForPublish 私有化——仅模块内一处调用,安全;② config docs 去重——每份文档删的两行均为过期重复行(缺 #3788 冷却文案 / pre-#3818 旧默认值),保留行是超集且为当前事实。两者都是 master-red 门禁的有标注顺手修,符合规范,不算跑题。

八问:

  1. 是什么问题:日志文件打不开时(权限拒绝、杀软/完整性锁、沙箱),代理的日志整条静默丢失——没有任何一行 warn 说明发生了什么,排障零线索。
  2. 什么原因导致的:两个独立缺陷叠加——①一次性降级告警只挂在同步异常通道上,异步 open 失败走 stream 'error' 事件而原处理只销毁流不告警;②launcher 子进程的 stderr 静音决策([noise] 观测噪音三件:启动横幅同秒重复 24 次 / reasoning-pair-violated 每请求重报(单会话 276 条) / %TEMP% 端口日志无轮转(5 天 31.7MB) #2864)把「流对象已创建且可写」(pending)当成「文件腿可用」,在 open 结果出来之前就静了 stderr。两腿同时死 → 全静默。
  3. 影响范围与严重程度:launcher 派生的子进程(BILI_LAUNCH_TOKEN)最重——日志能力整体失效且零痕迹(可用性不受影响);手动前台启动仅丢那一条降级 warn。p2。
  4. 怎么修复的:stream error 事件路由进既有一次性告警(+解除静音);活性判定从「open/fd 成功」升级为「首个字节真正写入成功」(win32 node 22 存在 open 出有效 fd 但写入必败的路径,open 本身不可信);静音决策挂到新钩子 onFileLegLive 之后;一次性告警的重挂点从建流移到确认写入处,避免每次注定失败的重试重新武装成刷屏。
  5. 是否有回归问题:逐项过既有行为——手动启动(无静音路径)byte-identical;轮转/orphan-inode 检测语义不变(轮转路径经 getStream 重置 writeProven,旧流迟到回调被 stream!==s 挡住);bytesWritten 在写失败时仍自增 = 既有行为未动;isFileLoggingActive 语义收紧后全仓唯一调用方就是 server.ts 原静音点(已迁到钩子),无其他调用方;文件腿死亡到 error 事件到达之间的极小窗口内日志可能双丢——可接受,测试里已注释为已知边界。未发现回归。
  6. 是否破坏缓存:不涉及(无缓存/游标/去重逻辑)。
  7. 是否改了设计方向:无架构转向,只收紧 logger 内部活性语义;wire 协议与双压缩模式不受影响(logger 是宿主侧基础设施)。
  8. 是否引入新配置:无。行为变更披露核对过:launcher 子进程静音时机由「启动时流对象存在」→「日志文件真实打开且首字节写出」;首次打开失败时 旧=永久静默,新=一条 [warn] + 继续 stderr-only 输出(即 %TEMP%/bili-proxy-<port>.log)。手动启动不变。

小注:PR 描述写 launcher 套件 332/332,我实测 403/403(master 上新测试增多导致计数漂移,无害)。

判定:可合。按 §7.4,本 PR 含 fallback 行为变更(静音时机属产品决策)且触碰 src/server.ts 热文件,不走自动合并,请人工合并:#3858

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

这条说明早于我上一轮审核到达,先同步现状,再报对你两个论断的独立复核结果。

现状更新(详情见我上一条八问回复):分支已从你描述的 48d45e7 基础进一步 rebase 到当前 master(39fb841,含你提到的 #3818/#3819/#3790 及之后的变更),并追加了一个测试确定性修复 commit e5ed953e2——三个 #3857/#2864 测试用 fd 可见性('open' 事件)当等待条件却断言交付侧状态,Linux 实测 1ms 轮询下 22% 概率 fd 已可见但零字节落盘(测试自身 20ms 轮询把窗口盖住了),已改为等待单调交付证据。远程 CI 全部 48 个 check run 绿。当前 head = e5ed953e2,master 此后未再前进,PR 无落后。

对「master 本身在这两个门禁上红」的独立复核(在纯 master 39fb841 的干净 worktree 里复现,不是只信描述):

另外核对过:这两个修复尚未被其他 PR 带进 master(master 今天仍是红的),所以「谁先落地谁带上」目前仍由本 PR 承担。按几行级小修调度纪律,不另开 issue,同意你的处置。

八问(仅针对本轮新增内容——两个 drive-by commit 的正当性与当前分支状态):

  1. 是什么问题:master 自身在 unused-exports 与 docs-sync 两个静态门禁上红,任何新 PR 的 CI 都会连带红,与 logger 修复无关。
  2. 什么原因导致的:[security][web] Diagnostics blob prints private relay hosts verbatim in the config section — the "secrets masked" header only masks values, not hosts | 诊断包 config 段原样输出内网中转地址:标题说"密钥已遮蔽",但只遮值不遮主机 #3811 之后 maskUrlForPublish 只剩模块内调用但 export 未收回;storage 配置语义在 SQLite unified storage degraded to legacy-only (bili v0.1.201, node v20.19.5, win32-x64) #3788/Storage wrap-up: 6 fixes + switch default mode to full (single PR) | 存储收尾六修 + 缺省模式翻 full(一个 PR 统一处理) #3818 演进后,已提交的 CONFIGURATION 文档还留着两行过期表行,生成器输出与提交产物漂移。
  3. 影响范围与严重程度:所有在途 PR 的 CI 门禁连带红(噪音 + 误判风险),不影响运行时行为;流程瑕疵级。
  4. 怎么修复的:最小修复随本 PR 携带——收回多余 export(调用面已全仓审计);重跑文档生成器让提交产物对齐 yaml 单一事实源。两个 commit 均独立标注、无行为变更。
  5. 是否有回归问题:无。export 收回不改任何运行时路径(唯一调用方在模块内);docs 变更纯删 4 行过期重复行,保留行为当前事实超集。纯 master 红→带修复后绿(CI 48/48)已验证。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入新配置:不涉及(docs 只是反映既有配置事实)。

判定维持上一条:可合,人工合并(fallback 行为变更 + src/server.ts 热文件,§7.4 不走自动合并)。#3858

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

windows-latest node-22 分片失败已修复,现全绿。

失败根因(不是测试问题,是实现缺陷):node-22 win32 上 createWriteStream 对某些路径(如目录)会正常发出 open 事件拿到 fd、但后续写入全部失败——这正是 #3823 现场机器的失败形状(MIC 按目标完整性标签拒写)。第一版把「降级 episode 重置」和 onFileLegLive 钩子都挂在 open 事件上,导致每次重试周期重新武装一次性告警(告警风暴)+ 钩子在永不可投递的腿上触发(会把 #2864 的 stderr 静音压到死腿上)。

修复(commit 400afa6):存活判据改为第一个字节真正落盘——open 只记录 fd(孤儿 inode 检测用);新增 noteProvenWrite() 在首次写入成功时结束 episode 并触发钩子;isFileLoggingActive()/onFileLegLive() 要求 proven 状态;error/rotation/close 路径复位。行为变更已在 commit 里披露:launcher 子进程的 stderr 静音从「首个 open 完成后」改为「首字节落盘后」——严格更安全(永不静音死腿),手动启动不受影响。

验证:本地 logger 六套件 17/17 + typecheck 干净;目录 fixture 现在在两个平台上分别走 POSIX(open 失败 EISDIR)/win32(open 成功写失败)两种形状钉住同一契约。CI 现全部通过(test 6 分片 + 静态门禁全绿)。

@ranxianglei
ranxianglei merged commit 930d5b4 into master Oct 20, 2026
48 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.634-master.930d5b4 (rolling master tag).

npm install -g billion-context@0.0.634-master.930d5b4  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…rades immediately (#3857)

Conflict-resolution rebase onto master: master already carries the main
#3857 fix via #3858 (delivered-write liveness + async-failure warn routing),
so this branch now keeps only the residual delta of the parallel
implementation:

- openStream verifies the fd at 'open' (fstat S_IFDIR check): win32 (node 22
  observed in CI) hands out an fd for a DIRECTORY whose writes then always
  fail — reject it at open so the degradation surfaces immediately (one-shot
  warn naming the cause, zero lost lines) instead of on the first lost line.
  Liveness still arms on the first DELIVERED write only (#3858 design kept);
  character devices (/dev/null logFile) stay legal. POSIX unchanged (open
  fails EISDIR before 'open' fires there).
- tests/logger-async-open-failure.test.ts narrowed to the two regression pins
  the merged silent-loss suite does not carry: mid-life leg death stays
  one-shot across the per-line reopen loop; isFileLoggingActive() reports a
  pending stream as not live (feeds the #2864 mute decision).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/other

Projects

None yet

1 participant