Skip to content

[PR] fix: surface async log-file open failures; gate #2864 stderr mute on verified open - #3873

Closed
ranxianglei wants to merge 1 commit into
masterfrom
2026-10-20_logger-async-open-failure
Closed

ranxianglei wants to merge 1 commit into
masterfrom
2026-10-20_logger-async-open-failure

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Closes #3857

What & why

When the OS refuses the log file (EISDIR/EACCES/EPERM — Windows integrity/sandbox/AV locks, permission denials elsewhere), the async open failure reaches only the stream's error event, which destroyed the stream in silence. The one-shot degradation warn (warnReopenFailed) was reachable only from getStream()'s try/catch — sync throws (e.g. mkdirSync failures) alone. Result: every log line dropped with zero warning; only in-memory diagnostic state showed anything.

Worse in launcher-spawned children (BILI_LAUNCH_TOKEN, stderr already redirected to the %TEMP% port log): the #2864 mute decision read isFileLoggingActive() right after configureLogger(), and that check counted a pending createWriteStream as live (writable but not yet open). So stderr got muted before the async open resolved; when the open was then refused, both sinks were dead at once — and the un-mute inside warnReopenFailed could never run because the warn itself never fired.

Fix

  1. Stream error handler routes into warnReopenFailed(err) — one-shot warn + un-mute of the stderr fallback, covering async open/write refusals.
  2. isFileLoggingActive() requires the verified fd (streamFd !== undefined), not just a writable pending stream.
  3. New onFileLegLive(fn) hook fires once per verified open (immediately if already live); server.ts registers the [noise] 观测噪音三件:启动横幅同秒重复 24 次 / reasoning-pair-violated 每请求重报(单会话 276 条) / %TEMP% 端口日志无轮转(5 天 31.7MB) #2864 mute on it instead of probing stream state at configure time. A refused open never fires it → stderr stays live.
  4. reopenWarned reset moved from stream creation to the verified open event: the pending→fail→retry loop otherwise re-arms the one-shot per doomed stream (a warn-per-line storm — the exact anti-pattern logger EPIPE feedback loop: async stream error re-enters uncaughtException handler — 77MB spam, rotation destroys forensic logs #1233 guards against).

Behavior changes (old → new)

  1. [noise] 观测噪音三件:启动横幅同秒重复 24 次 / reasoning-pair-violated 每请求重报(单会话 276 条) / %TEMP% 端口日志无轮转(5 天 31.7MB) #2864 launcher-child stderr mute timing: old = decided synchronously after configureLogger() on pending-stream writability (effectively always muted, even when the open later refused); new = fires from the stream's open event (fd verified). Consequence: a few extra duplicate startup lines may land in the port log between configure and open completion (ms-scale, bounded); on a refused open the mute no longer happens at all — that is the fix.
  2. isFileLoggingActive() semantics: pending-inclusive → verified-open-only. Its sole production caller was server.ts (now uses the hook); tests updated accordingly.
  3. New exported API onFileLegLive() — additive; no behavior change for existing callers.
  4. reopenWarned reset timing: on stream creation → on verified open. Sync-failure path unchanged (the reset never ran on throw before either); async path goes from 0 or N warns to exactly 1 per degradation episode.

Config surface: none (no new env vars / config keys / flags). No kernel/ or pi-subagents/ changes.

Tests

  • New tests/logger-async-open-failure.test.ts (6 tests): deterministic cross-platform async open failure (directory at the log path → EISDIR): muted child gets exactly one warn + stderr fallback alive + lines survive; one-shot holds across the retry loop (a healthy open arms the counter first, then rename+mkdir breaks the path under 5 interleaved writes — 5 doomed streams, 1 warn); hook contract (fires once per successful open / immediately when registered while live / never on refusal); isFileLoggingActive false on the pending stream, true after open. Red before the fix (silent loss reproduced), green after.
  • tests/logger-stderr-mute.test.ts: the "file leg live" assertion no longer races a pending stream — it waits for the verified open via the hook (the old assertion encoded the All logs silently lost when the log file fails to open: no warning on async open failure, stderr fallback permanently si | 日志文件打不开时全部日志静默丢失:异步打开失败不告警、stderr 兜底被永久静音 #3857 defect).
  • Targeted suites green locally: logger-async-open-failure, logger-stderr-mute, logger-epipe, logger-rotate, logger-rotation, logger-session-tag (19/19); typecheck + build clean. Full-suite verdict: CI on this head.

Rebased onto current master (fa9e64f91) — two extra independently-labeled commits for PRE-EXISTING master gate reds that blocked this PR's (and every PR's) test lanes via needs; both verified red on clean master before touching them:

@ranxianglei ranxianglei changed the title fix: surface async log-file open failures; gate #2864 stderr mute on verified open [PR] fix: surface async log-file open failures; gate #2864 stderr mute on verified open Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_logger-async-open-failure (d88dc72)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4642-pr.3873.35346b7

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3873.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 本 PR 即 #3857 的实现(Closes #3857);检索 open+closed 无并行修复同一缺陷的 PR。相邻的 #2864(mute 决策本身)、#1233(stderr EPIPE 风暴)是不同机制,本 PR 只消费其契约不复现其问题。分支内两个顺手小修(docs-sync 再生成、unused-exports 去导出)均为 master 既有红,已独立标注,机制与本 PR 无关、范围正确。

标签: bug / scope/other / p2(日志基础设施缺陷;可用性无损,特定触发——OS 拒绝日志路径——下诊断能力全毁;定级与 #3857 一致)

独立审核结论(基于对 head 2fe290f71 完整源码的逐文件复读,非转述 issue 侧分析)

方向关:修的是根因不是症状。两处生产者都被移除:①异步 open 失败的唯一出口是流 error 事件,原代码该事件只销毁流不告警——现在路由进既有的 warnReopenFailed(一次性告警 + 恢复 stderr 兜底);②存活判定原来把「pending 但未 open」的流当活——现在 mute 决策挂在 onFileLegLive 钩子上,由验证过的 open 事件触发,被拒的 open 永不触发它。没有在任何中间层做字节级掩盖。

回归清点(逐项过 diff 触碰路径的既有行为):

测试有效性:新 6 测断言有牙且双向齐(#3617 门纪律):warn 计数精确等于 1、warn 文案含原因、stderr 兜底行存活、被拒后 isFileLoggingActive()==false;重试循环 5 条死流只出 1 条 warn;钩子契约三向(open 一次一放/注册时已活立即放/被拒不放)。修复前 test 1 会等不到 warn 超时、test 6 首断言直接红——是真回归测试,不是恒真断言。logger-stderr-mute 旧断言「configureLogger 后立即 active」本身就编码了 #3857 缺陷,改为等验证过的 open,改得对。

本地定向验证(PR head 树实跑,非引用 CI):node --import tsx --test 六个 logger 套件(logger-async-open-failure / logger-stderr-mute / logger-epipe / logger-rotate / logger-rotation / logger-session-tag)= 19/19 通过。

CI(head 2fe290f71,与当前 master 同基,无需 rebase):48 项检查 = 32 success / 16 skipped(均为按设计门控的 e2e lane)/ 0 failure;ubuntu node22/24 三分片 + windows node22 两分片 + 全部静态门(docs-sync、unused-exports、lock-sync、config-surface-guard、kernel-guard…)绿。

结构化八问

  1. 是什么问题:日志文件被 OS 拒绝打开(EISDIR/EACCES/EPERM)时,文件腿静默死亡——零告警、后续所有日志行丢失;launcher 派生的子进程更糟,stderr 腿在 open 还没完成时就被判「活」而静音,两条腿同时死。
  2. 什么原因导致的:同一症状、两个独立根因:①降级告警只挂在同步 try/catch 里,而异步 open 失败只以流 error 事件出现,该事件原实现只销毁流不告警;②存活判定把 pending 流当活,[noise] 观测噪音三件:启动横幅同秒重复 24 次 / reasoning-pair-violated 每请求重报(单会话 276 条) / %TEMP% 端口日志无轮转(5 天 31.7MB) #2864 的 mute 决策与异步 open 竞态,可能先于其结果落地。
  3. 影响范围与严重程度:任何 OS 拒绝日志路径的部署(Windows 完整性/AV/沙箱锁、其他平台权限拒绝);代理继续服务(可用性无损),但持久日志与全部告警丢失,launcher 子进程连端口日志也死——诊断能力全毁;p2。
  4. 怎么修复的:告警路由到唯一能看到异步失败的事件;存活判定收紧为「fd 在手且 fstat 验证非目录」;mute 决策从状态探测改为事件驱动(新 onFileLegLive 钩子,被拒永不触发);一次性计数器的重挂点移到「写落盘确认」,使失败重试循环每 episode 恰好一条 warn。
  5. 是否有回归问题:无。高危模式逐项过——同步失败路径不变、轮转语义不变、logger EPIPE feedback loop: async stream error re-enters uncaughtException handler — 77MB spam, rotation destroys forensic logs #1233 stderr 路径未触碰、无 fail-fast 变吞错、无阈值/时序/输出格式漂移;唯一语义变化(mute 时机、isFileLoggingActive 口径、warn 次数)均已在 PR 描述披露且我逐条对照代码核实为真。
  6. 是否破坏缓存:不涉及(宿主侧日志基础设施,不碰缓存/重试/游标/去重键)。
  7. 是否改了设计方向:无转向——既有「降级到 stderr」契约补齐缺失的异步路径,一个决策点由探测改事件驱动;新增导出 API onFileLegLive() 为纯增量。
  8. 是否引入了新的配置:无(无新 env/config/flag,config-surface-guard 门绿)。

判定:可合并。 CI 全绿、本地定向 19/19、方向正确、回归清零、diff 干净(7 文件全部与目的相关,无锁文件/空白噪音)。合并动作留给人工执行(PR 当前 mergeable_state=blocked 为分支保护要求,我无权也不应代合):#3873

@ranxianglei ranxianglei added bug Something isn't working p2 scope/other labels Oct 19, 2026
…rades immediately (#3857)

Conflict-resolution rebase onto master: master already carries the main
#3857 fix via #3858 (delivered-write liveness + async-failure warn routing),
so this branch now keeps only the residual delta of the parallel
implementation:

- openStream verifies the fd at 'open' (fstat S_IFDIR check): win32 (node 22
  observed in CI) hands out an fd for a DIRECTORY whose writes then always
  fail — reject it at open so the degradation surfaces immediately (one-shot
  warn naming the cause, zero lost lines) instead of on the first lost line.
  Liveness still arms on the first DELIVERED write only (#3858 design kept);
  character devices (/dev/null logFile) stay legal. POSIX unchanged (open
  fails EISDIR before 'open' fires there).
- tests/logger-async-open-failure.test.ts narrowed to the two regression pins
  the merged silent-loss suite does not carry: mid-life leg death stays
  one-shot across the per-line reopen loop; isFileLoggingActive() reports a
  pending stream as not live (feeds the #2864 mute decision).
@ranxianglei
ranxianglei force-pushed the 2026-10-20_logger-async-open-failure branch from 2fe290f to d88dc72 Compare October 20, 2026 05:23
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解决:分支 rebase 到当前 master、冲突消解、已推送(新 head d88dc72de),PR 恢复可合并(现仅等 CI)。

查重:与已合并的 #3858 同机制家族(同一 #3857 缺陷)——主修复部分重复(已在 master),本 PR 保留的是它没有的增量(见下),故不关闭。

冲突的真正性质不是普通文本碰撞,而是设计级碰撞:本 PR 在途期间,master 合入了 #3858——同一个 #3857「日志静默丢失」缺陷的另一套并行实现,两者都改了 src/logger.ts 的核心判定:

处置:以已合并的 #3858 设计为基准(它的存活判据更严——不存在"先 mute stderr、随后写失败"的空窗),从本 PR 只保留 master 真正缺失的增量,压成一个干净 commit:

  1. 目录 fd 在 open 时立即拒绝:win32(CI 实测 node 22)对目录路径能开出合法 fd、之后每次写都失败。此前要等第一条日志丢失才暴露、且报错是原始写错误码;现在 open 时 fstat 校验,一次性 warn 直接写明 path is a directory,零丢行。POSIX 行为不变(那边 open 本身就以 EISDIR 失败,走既有错误路径);/dev/null 等字符设备日志仍合法。
  2. 两条 [PR] fix(logger): log file open failure now warns + falls back to stderr instead of total silent loss #3858 测试套件未覆盖的回归钉:① 会话中途日志路径变成目录时,逐行重试循环里恰好一次 warn(而不是每个失败流一次);② open 尚未完成的 pending 流不得被报告为 live(该值喂给 [noise] 观测噪音三件:启动横幅同秒重复 24 次 / reasoning-pair-violated 每请求重报(单会话 276 条) / %TEMP% 端口日志无轮转(5 天 31.7MB) #2864 的 mute 决策)。

原 4 个 commit 里的竞争实现主体、重复测试、文档漂移全部作为重复丢弃(已经由 #3858 在 master 上)。PR 标题/正文仍是原完整修复的描述;当前实际 diff 只有 2 个文件(+120/−4)。

八问:

  1. 是什么问题:上游 PR 冲突通知——本分支与 master 冲突;根子是同一 All logs silently lost when the log file fails to open: no warning on async open failure, stderr fallback permanently si | 日志文件打不开时全部日志静默丢失:异步打开失败不告警、stderr 兜底被永久静音 #3857 缺陷被两套并行实现各修了一遍。
  2. 什么原因导致的:两个 worker 独立实现了同一 logger 降级修复,存活判据不同(落盘写 vs 验证过的 open);master 先收了一套,另一套分支即产生设计级冲突。
  3. 影响范围与严重程度:冲突前后对用户均无影响(主修复早已随 [PR] fix(logger): log file open failure now warns + falls back to stderr instead of total silent loss #3858 上线);增量仅触及"日志路径被配成目录"这一罕见误配置场景,且只在 Windows 上改变表现。
  4. 怎么修复的:rebase 到当前 master,采纳已合并的 [PR] fix(logger): log file open failure now warns + falls back to stderr instead of total silent loss #3858 设计,只并入目录-fd 即时拒绝 + 两条回归钉,单 commit force-push。
  5. 是否有回归问题:逐项审过既有行为——POSIX 异步 open 失败路径(warn 一次 + stderr 回退恢复)不变;落盘写存活判据不变;orphan-inode 检测不变(streamFd 仅在验证通过后设置);一次性 warn 的重置语义不变;/dev/null 字符设备显式保留。有意变更仅 win32 目录场景一处(下条披露)。
  6. 是否破坏缓存:不涉及(未触碰缓存/重试/游标/去重逻辑)。
  7. 是否改了设计方向:没有——反而是收敛:两套并行实现归并为已合并的一套 + 一个小扩展。
  8. 是否引入新配置:无。

行为变更披露(有意):logPath 指向目录时,Windows 上的降级 warn 从"丢第一行后、报原始写错误码"变为 open 时立即触发、消息含 path is a directory、零丢行。原因:win32 目录 fd 是合法 open,等写失败既丢一行又让病因难认。POSIX 完全不变。

验证:本地定向跑三个 logger 套件 8/8 绿(含 #3858 已合并套件全部 4 例)、typecheck + build 绿;全量回归以远程 CI 对新 head 的结果为准。

判定:冲突已解、PR 可合(待 CI 绿 + 人工合并)。若你认为这个增量不值得单独走一轮合并,也可以直接关掉本 PR——主修复无论如何都已在 master;我的建议是合入(小、有测试、给 #3857 收尾闭环)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

目标 issue #3857 已关。列车审查时确认:内容已落地/不再需要,关单处理。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/other

Projects

None yet

1 participant