Repository navigation
feat: in-request model-content deadline (firstContentTimeoutMs/contentGapTimeoutMs, three-level cascade, default off) - #3264
ranxianglei wants to merge 7 commits into
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4004-pr.3264.04eb9f1This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3264.tgz
npm install -g packageInstall instructions are refreshed on each push. |
docs-sync CI lane failed on the PR head: the #3254 seed entries in website/config-reference/{process,providers}.yaml were added, but the derived artifacts (CONFIGURATION.md/.zh-CN.md index tables, website/main.js i18n strings, website/docs/index.html) were hand-edited instead of regenerated, so they drifted from generator output: - network.firstContentTimeoutMs/contentGapTimeoutMs index rows landed after postResponseLingerMs by hand; the generator emits them in seed order (after preflightDeadEndCooldownMs); - the two provider-level index rows were missing entirely; - cr.fcdeadline/cr.gapdeadline i18n entries and the tt.network summary line were absent from website/main.js + index.html. Re-ran tools/gen-config-docs.mjs generate and committed its output; the seed text (already authored in this PR) is the source of truth.
…3264) The per-request cascade read the model from the request body only. Gemini-native requests carry their model in the URL path, not the body, so providers[<key>].models[<m>] deadlines could never match a google-wire request — the model tier silently never applied there, the same silent-failure class this feature rejects for invalid values. Fall back to prepared.google.model (path-derived) when the body has no model field; non-google requests are unaffected (prepared.google is undefined).
Cosmetic: the #3254 edit left five spaces between proxyKeepAliveMaxMs and postResponseLingerMs in the hand-maintained `### network` Type line of CONFIGURATION.md and CONFIGURATION.zh-CN.md.
(ai: vllm-qwen/qwen3.8-87b) PR 审核结论(独立审过 diff、本地跑过测试、复现了 CI 红):机制成立、方向正确;审出 3 处问题已全部在 PR 分支修掉并 push(3 个独立标注 commit,rebase 到分支已并入的 master 之上),等新 head CI 全绿即可合。 结构化八问
审核中发现并已修复的问题(3 个 commit,已 push 到 PR 分支 a6e674b)
测试有效性(④)7 项全有牙且确定性好:fetch 桩换成抛错函数把「不盲重取」钉死(任何内部重取都会炸测试)、legacy pin 钉死关闭时行为、断言最后一个 SSE 块是 代码整洁度(⑤b)/ diff 整洁度(⑤)命名达意(interruptibleBody/armContentDeadline/contentDeadlineSettled)、无死代码、注释全部讲 why(locked-stream cancel 坑、C-rejected 决策、tee dump 分支取舍)且与仓库惯例一致;唯一 cast 复用同文件既有模式,无 as any。修复 commit 后 diff 每一行都与本 PR 相关,三个修复各自独立标注。 状态与判定
Follow-up(线路自适应期限 clamp(3×p95, 120s, 600s),需先补每线路健康首内容耗时采样)按 PR 正文另行立项,本 PR 不阻塞。 |
(ai: vllm-qwen/qwen3.8-27b) CI 全绿确认(head 全矩阵结果
唯一一次瞬红 → 判定 runner 侧 flake,与本 PR 无关 最终判定:可合。 实现与 owner 决策逐条对齐(三级级联、显式 0 不穿透、不做内部盲重取、gap 重新计时、attempt/source 日志位、默认 OFF 零开销且 legacy 逐字节不变),config surface 章节完整,行为变更已披露(旧值=无 bili 侧上限由客户端 ~780s 兜底 → 新值=配置值,仅启用者受影响),全量 CI 绿。合并请人工执行:#3264 |
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决并推送: 结构化八问:
判定:冲突状态应自动转干净;等远程 CI 全绿后由人工合并(按规则我不代合)。 |
In-request summarization had no bili-side wait bound of its own: a stalled relay that kept the TCP connection alive (byte-level idle timer rearmed by pings/comments) could hold a model attempt open until the CLIENT's own content watchdog fired — issue #3254 measured 775s + 773s + 27s for one fold, with bili logs showing only keep-alive notices and zero abort records. Add two opt-in model-content deadlines, each cascading model > provider route > global network.* (explicit 0 wins at its level): - firstContentTimeoutMs: response start -> first model content event (text/reasoning delta, tool call, redacted thinking, standalone signature); - contentGapTimeoutMs: silence between content events after the first arrived, re-armed on every content event (absent -> inherits the resolved initial value). On expiry the loop aborts the stalled upstream stream (interruptible-body wrapper: ReadableStream.cancel() on a locked stream rejects, so the kill goes through the underlying source's controller) and ends the turn with one clean in-band error in the #1453 shape — context intact, resend your last message. No internal blind refetch (owner decision): the client's retry ladder re-arms from the in-band error, which it handles natively (verified against codex-webui's handle_response_stream_error/retry_delay). Deliberately file-only config (no env layer): per-route/per-model values cannot be expressed as one process-wide variable. Strict validation (throw on non-integer/negative) at load time AND per request, unlike the lenient network.* numeric siblings — a typo must not silently disable a safety mechanism. Absent everywhere = byte-identical legacy behavior. Tests: tests/issue3254-first-content-deadline.test.ts (initial/gap fire, healthy-stream no-fire, legacy pin, validator unit, cascade table, load-time validation via ACP_PROVIDERS file path). Docs mirrored en/zh + config-reference yamls. Config-surface guard green.
docs-sync CI lane failed on the PR head: the #3254 seed entries in website/config-reference/{process,providers}.yaml were added, but the derived artifacts (CONFIGURATION.md/.zh-CN.md index tables, website/main.js i18n strings, website/docs/index.html) were hand-edited instead of regenerated, so they drifted from generator output: - network.firstContentTimeoutMs/contentGapTimeoutMs index rows landed after postResponseLingerMs by hand; the generator emits them in seed order (after preflightDeadEndCooldownMs); - the two provider-level index rows were missing entirely; - cr.fcdeadline/cr.gapdeadline i18n entries and the tt.network summary line were absent from website/main.js + index.html. Re-ran tools/gen-config-docs.mjs generate and committed its output; the seed text (already authored in this PR) is the source of truth.
…3264) The per-request cascade read the model from the request body only. Gemini-native requests carry their model in the URL path, not the body, so providers[<key>].models[<m>] deadlines could never match a google-wire request — the model tier silently never applied there, the same silent-failure class this feature rejects for invalid values. Fall back to prepared.google.model (path-derived) when the body has no model field; non-google requests are unaffected (prepared.google is undefined).
Cosmetic: the #3254 edit left five spaces between proxyKeepAliveMaxMs and postResponseLingerMs in the hand-maintained `### network` Type line of CONFIGURATION.md and CONFIGURATION.zh-CN.md.
…llow-up) Owner folded the previously filed follow-up (#3265) into this issue: once a route accumulates >=8 healthy attempts, each configured deadline value becomes a ceiling and the effective deadline tightens to clamp(3 x p95 of the route's recent healthy first-content durations, 120s, 600s) per phase. Sampled only from healthy attempts (completed without truncation/error); samples persist in session metadata (ring of 32); fire log records source=config|adaptive.
…ollowing adaptive) (#3254) Round 3 per owner mandate (out-of-the-box / self-adapting / zero-config): - resolver now falls back to FC_BUILTIN_DEFAULT_MS=720000 when no tier sets a value, so every deployment is bounded with NO config written; explicit 0 at any level still opts out (escape hatch). source gains 'builtin'. - adaptive clamp upper bound now FOLLOWS the resolved ceiling instead of a hardcoded 600s (removes FC_ADAPTIVE_CAP_MS): warm routes tighten toward clamp(3*p95, 120s, resolvedCeiling); slow routes keep their full ceiling. - gap phase inherits the resolved initial CEILING (raw configured/builtin), not the adaptive-tightened value. - corrected the in-band-error design note: an in-band error fails the turn cleanly and lets clients that HAVE a retry ladder back off; it does not assume every client auto-retries (verification was against a fork, not native codex). - docs en/zh + website seeds updated to on-by-default semantics.
Derived docs (CONFIGURATION en/zh, website main.js/index.html) had drifted from the yaml seeds the generator reads: the two follow-up feat commits (route-adaptive refinement, out-of-the-box 720s builtin) updated the yaml wording but the committed derived docs kept stale default-off-era phrasing (e.g. tt.network still said 'firstContentTimeoutMs unset' while the same table row said 'on by default'). Re-ran tools/gen-config-docs.mjs generate; output is byte-stable on a second run.
92881e6 to
bdcbc05
Compare
Closes #3254
背景
#3254:一次折叠三次尝试共 27 分钟(775s + 773s + 27s)。卡在途的 relay 靠 ping/注释不断续命字节级空闲计时器,bili 没有任何「无模型内容」上限——事实断路器是客户端自己的看门狗,而 bili.log 全程只有 keep-alive 提示,零 abort/timeout 记录。
修法(owner 拍板:开箱即用、每场景自适配、零配置)
两个模型内容期限字段,三级级联 model > provider route > global
network.*(显式 0 在所在层生效 = 关闭;gap 缺省继承 initial 的解析上限):firstContentTimeoutMs(阶段一):响应开始 → 首个模型内容事件(文本/推理增量、工具调用、签名;ping/注释不算内容)的最长等待;contentGapTimeoutMs(阶段二):首个内容之后相邻内容事件间的静默上限,每个内容事件重新计时。到期行为:中止卡死的上游流 + 一条干净的带内错误结束回合(#1453 形状:context intact, resend your last message)。不做内部盲重取——带内错误让回合干净失败:有重试梯的客户端退避后重试,没有的则直接停止。触发日志带 phase/waited/limit/attempt/round/source=config|builtin|adaptive(便于与客户端重试梯逐次对齐)。
开箱即用(默认开启,零配置)
默认以
720000ms内置值开启(FC_BUILTIN_DEFAULT_MS,单一具名常量,一行可调)——不写任何配置,每个部署都有界。选 720s 的标准是「别误杀合法长 prefill」:它是 relay 可用 SSE 注释合法续命的量级经验值,且仍短于客户端 ~780s 看门狗(bili 永远先动手并留日志)。任意层显式0= 对该阶段豁免(逃生门)。线路自适应收紧
线路积累 ≥8 次健康尝试后,每个解析出的阶段值转为上限:有效期限收紧为
clamp(3 × p95(该线路近期健康样本), 120s, 该解析上限)(phase 1 用首内容耗时样本,phase 2 用内容间隔样本),永不超过解析上限。上界跟随解析值而非硬编码(避免「冷线路用内置 720s、热到第 9 轮却莫名跳到 600s」的自相矛盾)。采样纪律:只有健康尝试入环(正常完成且无截断/错误;卡死、被期限中止、出错的尝试从不采样);样本存 session metadata(跨重启持久化,环形窗口 32)。冷启动(<8 样本)用配置/内置值原值。常量FC_ADAPTIVE_MIN_SAMPLES=8/FC_ADAPTIVE_WINDOW=32为本 PR 自定(owner 只给了 clamp 区间与 3×p95 公式)。实现要点:
ReadableStream.cancel()对 locked stream 会 reject(实测Invalid state: ReadableStream is locked,kill 静默空转→永久挂起),所以 kill 走 interruptibleBody 包装的 underlying-source controller;四个 adapter 的iterSseEvents都把读失败吞成 EOF,kill 落点 = for-await 干净退出,消费点无需 try/catch。全部显式 0 时零开销。config surface(本节为必填)
新增字段(全部可选;默认开启于内置 720000,非「缺省=关」):
network.firstContentTimeoutMs/.contentGapTimeoutMsproviders[<key>].firstContentTimeoutMs/.contentGapTimeoutMsproviders[<key>].models[<m>]同两键network.*进程全局单值,无法按线路/模型分化;compress.*三层是压缩专用语义;env 只有进程级单值表达不了 per-route(也不属 env-only 六类豁免)→ 刻意仅文件可设、无 env 层。0。行为变更披露(重要)
默认 ON —— 这是本次最大的行为变更,属产品决策(§7.4 default-value change),已由 owner 拍板(「需要做到一个开箱即用、每个场景都能自适配」+ 指定内置 720000)。
0豁免。查重
查重: 无。#3148/#3154(已合入)是死上游 pre-response 传输失败的轮级断路器——机制不同(连接未建成/未出响应头 vs 连接活着但只发 ping 不发内容的流);#2438/#2682 属 preflight 路径。本 PR 机制独立。
测试
tests/issue3254-first-content-deadline.test.ts11 项:initial/gap 两相触发(fetch 计数钉死「不盲重取」)、健康流不误触、legacy pin(ctx 不带字段时零开销路径不变)、parseDeadlineMs 单元、级联真值表(含 builtin/config/adaptive source 标注 + 全局层 0 豁免)、经 ACP_PROVIDERS 文件路径的 load-time 校验、p95/adaptive 钳制单元(含「慢线路 p95=300s 在 720s 上限下保持满额 720s」)、会话环形窗口与坏数据消毒、端到端采样(健康尝试入环、被中止尝试不入环)。本地定向套件绿(issue3254 + issue1455/1453 邻接回归 + config-surface-guard)。d41b7f796;分支已 rebase 到最新 master,含 fix: re-sign the preflight summarization egress over its own body (#3226) #3234/fix: preserve Codex mid-history developer items on Responses proxy #3229 等)。Follow-up
无——原 #3265(线路自适应期限)已按 owner 指示并入本 PR 实现,issue 侧已关闭为重复。