Skip to content

fix: warn when upstream accepts a request then stalls before first byte (#3434) - #3441

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-17_first-byte-stall-warn
Oct 17, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-17_first-byte-stall-warn

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

#3434: when the upstream accepts a forwarded request and then sends no bytes, bili logged nothing between forward … and the eventual outcome (client disconnected mid-stream after Nms (0 bytes written to client), or the 12-minute idle-watchdog abort). From the log alone you could not tell whether the stall was upstream-side or inside the proxy. This is the exact visibility gap that sent #3396 down a day-long misattribution (the side lane looked guilty while the stall was intermittent upstream/egress silence).

Change

Additive logging only — no behavior change. In relay.forward, arm a one-shot first-byte watchdog around the initial fetch (fetchWithTransportRetry): if response headers have not arrived after 30 s, log ONE warn naming method + masked target + lane:

[sess] upstream awaiting first byte ≥30s: POST https://…/v1/messages (lane=side)
  • The single fetch chokepoint covers every lane (side / main / passthrough), so the DSH side-request case from web_search on DSH never completes: the side request (server-tool utility call) lane forwards upstream but writes 0 bytes back to the client #3396 is included.
  • Deduped once per origin per process (the windowSourceLogged pattern) so a long-prefill local model logs at most one line, not one per slow request. The origin is claimed at fire time (not arm time) so a fast request never suppresses a later genuine stall on the same origin.
  • Cleared on fetch settle (resolve/reject) and on client abort; fires only while genuinely pending. The timer is unref()'d so a pending watchdog cannot hold the event loop open at shutdown.

Deliberately NOT doing

No fast-fail (e.g. returning 504 after N seconds of upstream silence): that is a new default/behavior and would truncate legitimate local-model long prefills (#1706/#1714 retired exactly such guards on purpose). If ever wanted, it is owner-decision territory (possibly scoped to the side/utility lane only).

Config surface

None. The 30 s threshold is a code constant (UPSTREAM_FIRST_BYTE_WARN_MS) with a _setFirstByteWarnMsForTest seam — no new env var and no new config key.

Behavior-change disclosure

None. The request/response flow is untouched; the watchdog only appends a warn line. On rejection the error still flows into the existing outer catch; the try/finally around the awaited fetch adds no semantic change.

Verification

  • npm run typecheck clean; npm run build success; check:unused-exports clean (on the rebased head).
  • New regression suite tests/issue3434-first-byte-watchdog.test.ts — 8 cases: fires-once-on-stall, no-warn-on-settle, per-origin dedupe, cross-origin independence, abort-before-threshold, already-aborted disarm, disabled-threshold disarm, reset seam. All pass locally. Full-suite verdict = remote CI on this head.

Closes #3434

…te (#3434)

Between `forward …` and the outcome (client disconnect "0 bytes written" or
the idle-watchdog abort) bili logged nothing when the upstream accepted the
request and then sent no bytes, so an upstream stall read as proxy-side silence.

Arm a one-shot first-byte watchdog around the initial fetch in relay.forward:
if response headers have not arrived after 30s, log ONE warn naming method +
masked target + lane. Deduped once per origin per process (windowSourceLogged
pattern) so a long-prefill local model logs at most one line, not one per slow
request. Cleared on fetch settle or client abort; fires only while pending.

Additive logging only — deliberately NO fast-fail (would truncate legitimate
long prefills, cf. #1706/#1714) and NO new config surface (code constant +
_setForTest seam).
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-17_first-byte-stall-warn (c3ab538)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4053-pr.3441.2289fad

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3441.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。独立检索 open+closed issues/PRs(first-byte / stall / upstream-silence / awaiting / timeout-visibility 等失败签名与标题关键词),无同症状同机制的既有条目;#3434 是本 PR 的需求来源(可见性缺口),#3396 是该缺口的受害者而非报告——两者均非重复工作项。

PR #3441 审核中(独立审 diff + 既有行为清点 + 测试有效性,不照抄 issue 侧结论)。当前 CI 状态:静态门禁全绿(kernel-guard / config-surface-guard / version-guard / docs-coverage / lane-guard 等),6 个 test shard in_progress。完整八问结论稍后在本线程给出。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

PR #3441 独立审核完成:逐行读了 diff、清点触碰路径的既有行为、验证测试有效性,本地跑了定向门禁(typecheck + 回归套件),未照抄 issue 侧结论。八问如下:

  1. 是什么问题:上游「收下请求后一个字节都不发」时,bili 在 forward … 与最终结果(client 断连 0 bytes written / idle-watchdog abort)之间日志空白,光看日志分不清卡在上游还是代理内部(web_search on DSH never completes: the side request (server-tool utility call) lane forwards upstream but writes 0 bytes back to the client #3396 被这个盲区坑了一整天误判)。
  2. 什么原因导致的:forward 路径在 fetch 拿到响应头之前没有任何计时/告警机制——可观测性缺口(缺一条诊断信号),不是功能缺陷。
  3. 影响范围与严重程度:所有经 bili 转发且上游可能间歇静默的场景(side/main/passthrough 三 lane)。严重度 = 排障体验瑕疵,不影响功能正确性——请求该成功还成功、该失败还失败。
  4. 怎么修复的:在主 fetch 收口点 arm 一个 one-shot 看门狗,30s 没等到响应头打一条 warn(method + masked URL + lane);fire 时才按 origin 认领去重(快请求不压掉后续真 stall);fetch settle(finally clearTimeout)或 client abort({once:true} listener)清掉,timer unref 不拖事件循环。纯增量日志,明确不做 fast-fail(会截断合法长 prefill,Opencode failed to send message with error: upstream stream ended before a completion event; this turn may be incomplete #1706/配置面收敛:移除全部 BILI_*/ACP_* 行为类环境变量,统一迁入 config.json #1714 正是为此撤过同类 guard)。
  5. 是否有回归问题:无。触碰路径(relay.forward 初始 fetch 块)既有行为逐项清点:①重试语义不变(promise 同参创建、同法 await);②reject 仍原样流入外层 catch(recordUpstreamConnection/Alert/CircuitFailure/armFailureShrink/dropRetrievals/throw 全链路未动,finally 只多一个 clearTimeout);③circuit-open 快速失败在看门狗 arm 之前抛出,不受影响;④client abort/disconnect 处理未动;⑤成功路径记账未动;⑥既有日志行逐字节不变,仅新增至多每 origin 一行 warn。高危模式逐项过:无阈值/时序漂移(30s 是纯日志常量)、无 fail-fast 变吞错(未引入任何 fail-fast)、无输出格式变化、无重试策略变化。
  6. 是否破坏缓存:不涉及。不触碰缓存/重试/游标/去重键任何失效策略。
  7. 是否改了设计方向:不涉及。纯增量可观测性,fast-fail 留 owner 决策域未碰。
  8. 是否引入了新的配置:无。UPSTREAM_FIRST_BYTE_WARN_MS 代码常量 + _setFirstByteWarnMsForTest seam,不加 env/config key(config-surface-guard 绿印证)。

独立验证证据(非复述 PR 描述):

  • 收口点全覆盖成立:fetchWithTransportRetry( 在 src/ 下唯一调用点即 relay.ts:659;三 lane 全部汇入——main(handle.ts:1832 amPrepared)、side(handle.ts:1919-1922,sidePassthrough: true)、passthrough(handle.ts:383/419/454/541,prepared=null)。lane 三分派与调用点一一对应。
  • 生命周期无泄漏:arm→fire/settle-clear/abort-clear 三条出口闭合;并发同 origin 多个 timer 由 fire 内 has() 复查保证先到先认领;origin 解析失败兜底用原始 URL 作 key。
  • masking 一致性:warn 行走 maskUrlForLog,与相邻 forward … 等所有日志行同一口径(公共 LLM host 默认可见、私有 host <private-host>、BILI_LOG_MASK_HOSTS=0 可还原)。评估过改用 hostIdForLog(Provider route miss silently drops per-model context override; 502 error then advises setting the already-set compress.modelContextLimit #2317 可区分指纹)——否决:相邻关联行用的就是 maskUrlForLog,换口径反而破坏「按相邻行对号入座」的排障路径。非阻塞观察,不改。
  • 测试有牙:8 条全是真断言(调用次数/级别/消息形状含 masking/disarmed 返回 undefined),覆盖 armFirstByteWatchdog 全部分支(settle 误报、去重失效这类回归都抓得住);确定性(30ms seam 阈值,无环境运气)。唯一缺口:没有「真实 pipeline 里 forward() 确实 arm 了」的集成 case——additive logging 可接受,CI e2e lane 兜底管道健康。
  • 本地门禁(rebased head c3ab538):typecheck 绿;tests/issue3434-first-byte-watchdog.test.ts 8/8 过。
  • CI:静态门禁全绿(static-gates / kernel-guard / config-surface-guard / version-guard / docs-coverage / lane-guard / lock-sync / test-import-graph);test shard 3/6 绿(ubuntu-22 ×3),ubuntu-24×1 + windows-22×2 仍在跑。
  • diff 整洁度:单 commit、2 文件、+187/−1,每一行都与本 PR 目的相关;无 kernel 变更、无版本 bump、无无关改动。

判定:可合并——方向正确、零回归、代码质量达标、测试钉住行为;待剩余 3 个 test shard 转绿即可由人工合入(我无权合并)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 补充结论(补上一轮审核的 Q5 回归判定与合并判定):

全部 6 个测试分片已跑完:5 绿、1 红(test (windows-latest, 22, 2))。我拉取了红分片完整日志(2.5MB)逐段分析,结论:该红分片是与本 PR 无关的既有 CI flake。

红分片性质(不是断言失败):

  • TAP 汇总 # pass 3948 / # fail 0 / # cancelled 5,随后 [ci-tests] all FAILED after 614.6s (exit=1 signal=null)——零断言失败,进程却以 exit 1 收场;
  • 唯一根失败:tests/pi-omp-fork-adopt.test.ts 的 pi lane: an exploding getBranch fails toward capability and still adopts (#2469) 挂起 9.2s,node:test 报 failureType: 'cancelledByParent'、error Promise resolution is still pending but the event loop has already resolved(ERR_TEST_FAILURE);同文件后续 3 条 not ok(1876-1878)duration 为 0,纯级联连带;
  • 该测试起真实 harness 服务器并 await 一个 ctx.getBranch 抛错的 plugin handler——挂点是「本地 HTTP promise 悬空 + 事件循环提前排空」形态,非业务逻辑错误。

排除链(为何不是本 PR 引入):

  1. 同一 commit c3ab538 下,同名测试在 ubuntu-22 shard 3/3(ok 1362)与 ubuntu-24 1/1(ok 4200)均通过——非确定性;
  2. 本 PR diff 只触碰 relay.forward() 初始 fetch 看门狗(unref 的 30s timer + 至多一行 warn),与 plugin-fork adoption 路径零交集;机制上 unref 定时器也不可能产生「event loop already resolved」这类错误(不阻塞排空,且 9.2s 窗口内根本不会触发);
  3. 该签名是 open issue CI flake: stream-keepalive tests (#1647) parent-cancelled on loaded Windows shard — no assertion failures #3356(win22 负载分片 parent-cancelled flake)跟踪家族的第二次独立观测——首次是 stream-keepalive 【#551的相似问题】长时间Prefill超过300秒后断开,原因是bili拦截上游用于防undici超时的保持连接注释,导致触发Node原生undici默认300秒超时断连 #1647 文件,同样 fail 0 + cancelledByParent + 同一错误串;且本文件正属已关闭的 win22 plugin-fork teardown 竞态簇(Windows CI(win22)上 plugin-fork 测试套件挂起:首个子测试卡住后整文件被级联取消 #3179/fix(test): force-drain plugin-fork teardown connections to stop win22 cascade-cancel (#3179) #3180/windows-22 CI lanes: plugin-fork harness teardown race — idle keep-alive sockets starve server close and the test runner tears down pending tests #3181/test: harness teardown — kill idle keep-alive sockets with closeIdleConnections (#3181) #3183)的同族测试。

已做动作:

判定不变:本 PR 自身零回归(上轮八问结论维持)。红分片属环境 flake,可合并——重跑转绿即可直接人工合入;若重跑再红,则是 #3356 家族第三次观测,届时建议优先修 win22 harness 本身,而不是继续让无关 PR 陪跑。

备注:排查过程中我的沙箱本地磁盘写满(ENOSPC),shell/文件工具暂不可用,只能走网络通道;重跑结果确认待环境恢复后补交。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收尾更新(补上条备注里的待办):已触发的重跑转绿——test (windows-latest, 22, 2)(job 117140835036)success,整个 CI run 38946318292 现在 completed / success,全部门禁通过。

最终判定维持:可合并。红分片确认为 #3356 家族的 win22 负载 flake(第二次观测,证据链在 #3356),与本 PR 零关联;本 PR 自身零回归、测试有牙、单点覆盖全 lane。请人工合入。

另:沙箱磁盘问题已自行清理恢复(删掉了自己验证时装的 node_modules 与临时日志,释放约 350MB);整盘写满的主因是共享工作区目录总量过大(fleet 级),非本会话造成,留作运维参考。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.521-master.5fc5fea (rolling master tag).

npm install -g billion-context@0.0.521-master.5fc5fea  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Silent upstream stall is invisible in logs: no 'awaiting first byte' warning between forward POST and client disconnect

1 participant