diff --git a/CHANGELOG.md b/CHANGELOG.md index e538ab57a5..bd88b60b6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,8 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag - **系统架构文档体系落地(owner 指令 2026-11-08):`docs/architecture/` 成为实时维护的架构枢纽**:新增 `docs/architecture.md` / `docs/architecture.zh-CN.md` 总枢纽(整体架构图中英双版转正 + 8 模块总表),首个样板模块页 `docs/architecture/storage.{md,zh-CN.md}`(统一存储:职责/原理/工作机制/硬不变量,配模块图中英双版)。文件级地图 `reference/architecture.md` 补齐 125 个缺失条目(新存储层 / marker / fastpath / lanes 等全部新模块)并升级为 CI 强制全量覆盖。新增 CI 车道 `architecture-docs`(`tools/check-architecture-docs.mjs`):地图覆盖全 `src/` 文件、总表↔页面成对、图 svg 优先 + png 备选齐备、SVG 基础良构(未转义 `&` 曾让英文图线上破图)。AGENTS.md「Documentation Layout」新增实时维护规则:改结构 ⇒ 同 PR 更新页 + 图 + 总表。其余 7 个模块页(pipeline / lanes / kernel / identity / fastpaths / webui / update)同一 PR 全部补齐——总表 8 行全 ✅(每页 = 中英双版文档 + 中英双版图)。 ### Fixed +* **普通格式/参数错误的上游 400 不再被误诊为「上下文溢出」(#4452)**:`isMarkerlessOverflowReject` 原判据是 `(400 || 413) && 无任何溢出标记`,于是畸形请求、schema 校验失败、role 顺序不对的 400(各家中转站返回的正文几乎一模一样)全部命中,日志打出「无溢出标记 → 自愈无法启动 → 该路由校准饥饿」的警告——而这类请求根本没有「太大」的证据,纯属制造假警报。现在只有 **413** 才成立:413 本身就是载荷过大这一语义的判据,正文丢了标记才是真正的中转站重写问题;400 交给既有的溢出标记解析器(未命中即普通错误,不告警)。已知溢出的解析与 refold / emergency-shrink / window-learn 全部决策一字未动。 +* **中转站拒绝原因不再被截断在「All targets failed」之后(#4452)**:`← upstream : ` 只保留正文前 600 字节,而扇出型中转站在全部目标失败时返回的是聚合体,真正的死因在最后一个尝试的 `details.error.message` 里——永远落在截断之外,拒绝原因因此无法从日志诊断。现在该行追加一个有界、脱敏的终局原因(`; terminal cause [attempt N/M]: …`):只读 `attempts[last].details.error.message` 一条、经共享 `redactSecretsInText` 洗凭证、折叠全部空白(杜绝换行伪造日志行)、上限 300 字符、正文超过 64 KiB 直接不解析;正文里没有该结构时后缀为空,日志行与原实现逐字节一致。纯诊断,不改重试判定与响应字节。 - **dsh 桌面副本自己更新自己时也跑 boot smoke(#2082 后续)**:#2091 的 boot smoke 门只接在 `refreshDshDesktopCopy` 上(别的 bili 进程替桌面副本刷新的那条路)。代理本身就从桌面副本运行时,`hostManagedInstall` 按 #1575 放行,走的是普通自更新路径——周期检查 / `bili update`(`checkForUpdate`)和 advisory 强制安装 / 面板安装(`forceInstallVersion`)——这两处调 `installViaTarball` 没带 `bootSmoke`,入口能解析但加载即抛错的包只过 `node --check` 就被换上去,正是这道门要挡的状态。现在两条路径在安装目录是 dsh 桌面副本时同样启用 boot smoke,失败回滚到旧副本;其他安装布局行为不变。 - **A compress call that folded nothing no longer rewrites every `` tag on the next request (follow-up to #5608-C/D, #5797).** `applyRanges` armed the deferred one-time ref/ledger alignment (`session.pendingLedgerAlign`) right after committing the kernel's returned state — before the `blocksCreated === 0` returns — so a refused compress (range too small, cannot anchor, already compressed) or a no-op armed it too. The next save then ran `restampRefsToLedger`, which raised `timestampsHighWaterRefSeq` to the highest ref unconditionally: on a session whose refs already sit on their ledger seats (`kept=N restamped=0 rehomed=0`) no number moved, yet every tag minted so far dropped below the mark and lost its `time=` attribute — the whole cached prefix re-billed after a call whose receipt says nothing was folded. Now the marker is armed only once a block was really created; a real fold arms it exactly as before. - **压缩树图:节点标题不再一闪即逝,点击压缩块能真正看到被压缩的消息内容**:此前树节点只有原生 SVG `` 悬停提示——选中节点或 5s 轮询重画时 DOM 一变提示就消失,实测「显示一下马上消失」;右侧 Block details 里块的消息行虽带消息编号(mNNNNN)但内容全是一句「消息已不在会话中」(宿主原生压缩把消息从上下文剪掉后,字节无处可取)。现在:① 悬停提示改为页面内常驻浮层(.g-tip)——跟随指针、显示完整块号/tier/消息范围/完整主题/tokens/消息数,选中节点或轮询重画都不再打断,离开图形区才隐藏;悬停时节点卡片上的主题行同时原地扩宽(~20 字 → 20 字上限的完整截断,离开复原);② 5s 轮询只回传瘦节点(无主题/摘要/tokens),重画前先与详情富块按 blockId 合并——节点标题、浮层元信息、消息数不再在每次 tick 后退化成光秃秃的块号;③ 服务端新增**冻结快照水合**:压缩块落盘时已把原始消息逐条存进 blockContents(`[role]`/`[role • tool]` 头 + 原文,pi 宿主 user 消息还带 mNNNNN 标记行)——水合按**标记锚定**对齐:pi 宿主打进 user 消息正文的 mNNNNN 标记是身份戳,每个标记用两条独立证据之一锚死——**恒等锚**(标记号=该位当前 ref,从未被重编号)或**冻结 span 算术锚**(标记的冻结编号 − 块冻结 span 起点 = 快照位序;span 与标记同批写入,冷加载 restamp(#5196 parked 段换新号)重编了整块 ref 也照样成立——实测 b27 全块 127 条因此恢复,旧规则因标记号已不在 ref 表里整块拒水合);锚点之间的段仅在条数平衡时按位 zip。标记既非恒等也非算术锚(冻结上下文混入外来消息/span 无法证明)、锚点乱序、无锚非叶块(快照 span 的是 effective 超集)一律拒绝,未证明的位置保留诚实的 missing 标记——宁可不显示,不冒错位张冠李戴的风险(预览 + 点击展开取原文,`/message?ref=` 同享该层)。实测大会话(253 块/9771 个直接消息引用):7789 条(80%)恢复出真实预览与原文,其余保持 missing。回归:tests/web-blocks-tree.test.ts 冻结快照水合 7 用例(按位水合 / 条数不符拒 / 标记移位旧锚 / 宿主剪块中消息的锚定恢复 / restamp 整块恢复 / 算术锚优先于巧合 ref 碰撞 / 无锚标记整块拒(含 click-to-open 双向断言)),源码级断言更新为合并后重画(renderCollapSvg(merged)。 diff --git a/reference/architecture.md b/reference/architecture.md index 41bc0ffb6a..0447cd88c0 100644 --- a/reference/architecture.md +++ b/reference/architecture.md @@ -27,6 +27,7 @@ billion-context/ │ │ ├── prepare-google.ts # Gemini request preparation incl. :countTokens twin (#1440) │ │ ├── prepare-openai.ts # OpenAI chat request preparation (#1440) │ │ ├── prepare-responses.ts # Responses request preparation + codex compact/prompt-cache helpers (#1440) +│ │ ├── rejection-diagnostics.ts # Bounded terminal provider cause from direct or nested aggregate rejections │ │ ├── relay.ts # Upstream relay zone: forward() + fake-completion recovery (#1440 P2 cut 4) │ │ ├── request-cost.ts # Per-request local-cost line (outbound measured at the real wire body) + #5408 heartbeat + wall-clock slow-session advisory, facts only (#903/#4528/#5744) │ │ ├── responses-ordering.ts # #564 Responses assistant-run ordering repair + producer-owned turn-separator provenance (#2627) diff --git a/src/server/rejection-diagnostics.ts b/src/server/rejection-diagnostics.ts new file mode 100644 index 0000000000..729238c548 --- /dev/null +++ b/src/server/rejection-diagnostics.ts @@ -0,0 +1,122 @@ +/** + * #4452: bounded terminal-cause diagnostics for a relayed upstream REJECTION. + * + * A relay/gateway that fans a request out across candidate targets may answer a + * total failure with ONE aggregate body carrying every per-target attempt, e.g.: + * + * { "error": { "message": "All targets failed", "type": "upstream_error" }, + * "attempts": [ { "target": "…", "details": { "error": { "message": "…" } } }, … ] } + * + * bili's own log line (`← upstream <status>: <snippet>`) keeps only the first + * 600 bytes of that body, so the operator sees the OUTER narrative — "all + * targets failed" — while the per-target cause sits further in. This module + * reads exactly ONE such nested cause (the last usable attempt's) when a body + * has that shape, reporting its attempt position rather than guessing causality. + * + * Hard bounds, deliberately: + * - ONE message, never a recursive walk: no `JSON.stringify` of the payload, no + * traversal of `cause`/`errors`/`attempts` chains, no echoing of the request + * body. Only one `attempts[].details.error.message`, with the aggregate + * accepted directly or beneath its standard `error` envelope. + * - Credential scrubbing goes through the shared log-mask sanitizer + * (`redactSecretsInText`), the SAME choke point the logger applies, so a + * provider that echoes the rejected key cannot smuggle it into the log. It is + * a high-signal credential scrubber, NOT a general privacy filter: it does not + * mask arbitrary URLs or paths, and the far stronger guarantee here is + * structural — only this one nested string is ever read, never a body dump. + * - Bounded to `MAX_DIAGNOSTIC_CHARS`; ALL control characters (C0+C1, incl. + * NEL U+0085) and whitespace collapsed so a newline (the `\r\n` shape or any + * C1 line-break) can never forge a second log field. + * - Absent, non-string, empty, or structurally-unrelated payloads return + * `undefined`: the caller then logs exactly what it logged before, so a + * rejection that carries no attempt detail is unchanged byte-for-byte. + * + * Recovery policy is NOT touched here — this module only reads a rejection + * body; every retry/refold/arm decision stays in the relay. + */ + +import { redactSecretsInText } from "../log-mask.js"; + +/** Upper bound on the retained cause text, independent of the outer snippet. */ +const MAX_DIAGNOSTIC_CHARS = 300; + +/** Do not even attempt to parse a body larger than this: an aggregate error + * reply is small (a few KB); a multi-megabyte body is a stream (or an HTML + * error page), not a JSON rejection envelope, and parsing it would cost the + * request more than the diagnostic is worth. */ +const MAX_PARSED_BYTES = 64 * 1024; + +interface DiagnosticResult { + /** The last usable attempt's nested cause, sanitized + bounded. */ + message: string; + /** 0-based index of the attempt the cause came from (for the log line). */ + attemptIndex: number; + /** How many attempts the aggregate body carried. */ + attempts: number; +} + +/** Collapse control chars (C0+C1 — incl. NEL U+0085) and all whitespace to + * single spaces, then clamp, so no control byte can survive as a forged field + * break in the log line. */ +function bound(text: string): string { + const collapsed = text.replace(/[\u0000-\u001f\u007f-\u009f]/g, " ").replace(/\s+/g, " ").trim(); + if (collapsed.length <= MAX_DIAGNOSTIC_CHARS) return collapsed; + return `${collapsed.slice(0, MAX_DIAGNOSTIC_CHARS)} …`; +} + +/** Read `value[name]` only when `value` is a non-array object, so a nested + * field is reached without a generic record guard that would prove nothing + * about the field actually consumed here. */ +function child(value: unknown, name: string): unknown { + if (typeof value !== "object" || value === null || Array.isArray(value)) return undefined; + return (value as Record<string, unknown>)[name]; +} + +/** + * Read the single last-reported nested cause out of a relay rejection body. + * + * `payload` is the raw upstream error-body text (the same string the relay + * already buffers to detect an overflow). Returns `undefined` whenever there is + * no usable `attempts[].details.error.message`, leaving the caller's existing + * logging untouched. + * + * Selection rule: the LAST attempt that carries a non-empty + * `details.error.message`, scanning from the end. If the terminal attempt is + * silent, an earlier cause is reported with its actual position and is NOT + * labeled terminal: its existence does not establish why the last stage failed. + */ +export function rejectionDiagnostics(payload: string | null | undefined): DiagnosticResult | undefined { + if (typeof payload !== "string" || payload.length === 0) return undefined; + if (Buffer.byteLength(payload, "utf8") > MAX_PARSED_BYTES) return undefined; + const trimmed = payload.trim(); + // A rejection body must be a JSON object envelope — anything else (an HTML + // error page, a bare text message, an SSE frame) carries no attempts[]. + if (!trimmed.startsWith("{") || !trimmed.endsWith("}")) return undefined; + let parsed: unknown; + try { + parsed = JSON.parse(trimmed); + } catch { + return undefined; + } + const attempts = child(parsed, "attempts") ?? child(child(parsed, "error"), "attempts"); + if (!Array.isArray(attempts) || attempts.length === 0) return undefined; + for (let i = attempts.length - 1; i >= 0; i--) { + const raw = child(child(child(attempts[i], "details"), "error"), "message"); + if (typeof raw !== "string") continue; + const sanitized = bound(redactSecretsInText(raw)); + if (sanitized.length === 0) continue; + return { message: sanitized, attemptIndex: i, attempts: attempts.length }; + } + return undefined; +} + +/** Render the diagnostic as the compact suffix the relay appends to its + * `← upstream <status>` line. Returns "" when there is no usable cause, so the + * caller can concatenate unconditionally and stay byte-identical without one. + * Pure numbers and the sanitized cause only — never the payload. */ +export function rejectionDiagnosticSuffix(payload: string | null | undefined): string { + const diag = rejectionDiagnostics(payload); + if (diag === undefined) return ""; + const position = diag.attempts > 1 ? ` attempt ${diag.attemptIndex + 1}/${diag.attempts}` : ""; + return `; ${diag.attemptIndex === diag.attempts - 1 ? "terminal" : "reported"} cause${position}: ${diag.message}`; +} diff --git a/src/server/relay.ts b/src/server/relay.ts index 7e34790cf8..82612898c2 100644 --- a/src/server/relay.ts +++ b/src/server/relay.ts @@ -73,6 +73,7 @@ import { applyOutputSteering, applyOutputSteeringJson } from "../output-steering import { estimateWireOverhead } from "./budget.js"; import { buildForwardHeaders, connectionNamedHeaders, RESPONSE_ONLY_STRIP_HEADERS, safeSessionId, UPSTREAM_HOP_HEADERS } from "./headers.js"; import { bodyDumpEnabled, logDumpFailure } from "./observability.js"; +import { rejectionDiagnosticSuffix } from "./rejection-diagnostics.js"; import { effectiveAbsorbBlock, effectiveBlockOpsEnabled } from "./prepare-responses.js"; import { awaitDrain, bufferToStream, dumpStreamToFile, pipeThrough, readStreamToBuffer } from "./stream-io.js"; import { buildForwardTarget, measuredBreakdownClause, measuredOutbound, outboundPayloadBreakdown, registerRequestAbort, beginStreamKeepalive, effectiveTokenCount, googlePathKind, imageBillingFor, imageTokenCapFor, inboundResignCredential, outboundContextEstimates, resolveUpstream, stripKernelSummaries, usageGradeInputBaseline, DOOMED_PROBE_ABSOLUTE_BYTES, DOOMED_PROBE_BYTES_PER_TOKEN, type Prepared } from "../server.js"; @@ -2052,14 +2053,18 @@ async function forwardCaptured( } log("warn", `[${s.id}] upstream image-count wall (cap ${info.imageCountCap}, request carried ${info.imageCountSeen ?? "?"}) — learned for pre-send convergence on this route; no token-space arm (#4124)`); } else if (isMarkerlessOverflowReject(upstream.status, errText)) { - // #4452: the gateway rewrote this overflow-shaped reject into a body with no - // parseable marker, so bili's self-heal cannot engage from it — name that here - // (+ the last meter reading) so a body-rewriting gateway's silent starvation is - // diagnosable from the log rather than the opaque line further down. + // #4452: a 413 IS the payload-size verdict, yet this body carries no + // parseable marker, so bili's self-heal cannot engage from it — name that + // here (+ the last meter reading) so a body-rewriting gateway's silent + // starvation is diagnosable from the log rather than the opaque line + // further down. Only 413 reaches this branch: an unmarked 400 is the + // generic malformed/schema-rejected bucket across relays and says nothing + // about size, so it is never diagnosed as overflow here (see + // isMarkerlessOverflowReject). const est = s.stats.lastInputTokens > 0 ? s.stats.lastInputTokens : 0; const win = typeof s.metadata.effectiveContextLimit === "number" && s.metadata.effectiveContextLimit > 0 ? s.metadata.effectiveContextLimit : 0; const meterNote = est > 0 && win > 0 ? `; last settled input ~${Math.round(est)} vs effective window ${win}` : ""; - log("warn", `[${s.id}] ← upstream ${upstream.status} carries no parseable overflow marker (an intermediate gateway rewrote the rejection body?) — bili's overflow self-heal (refold / emergency-shrink / window-learn) CANNOT engage from this rejection${meterNote}; repeated at similar size = calibration-starved route (#4452)`); + log("warn", `[${s.id}] ← upstream ${upstream.status} carries no parseable overflow marker (an intermediate gateway rewrote the payload-size rejection body?) — bili's overflow self-heal (refold / emergency-shrink / window-learn) CANNOT engage from this rejection${meterNote}; repeated at similar size = calibration-starved route (#4452)`); } // #762: learn strict-echo on the MAIN request path too. The loop-only // learner (src/loop/core.ts) never sees client-originated 400s, so a @@ -2096,7 +2101,15 @@ async function forwardCaptured( let snippet = bodyText.slice(0, 600).replace(/\s+/g, " ").trim(); if (bodyText.length > 600) snippet += " …"; if (!snippet) snippet = "(no body)"; - loggerLog("warn", `[${errSid}] ← upstream ${upstream.status}${reqIdText}: ${snippet}`); + // #4452: a relay that fans out to several targets answers a total failure + // with ONE aggregate body whose real cause sits in the LAST attempt's + // `details.error.message` — past the 600-byte snippet above, so the log + // showed only "all targets failed". Append that single terminal cause + // (bounded + credential-scrubbed by the helper; empty when absent, leaving + // this line byte-identical to before). Read-only: no recovery decision + // here depends on it. + const terminalCause = rejectionDiagnosticSuffix(bodyText); + loggerLog("warn", `[${errSid}] ← upstream ${upstream.status}${reqIdText}: ${snippet}${terminalCause}`); // #4204: feed this send's FINAL verdict. A model-scoped KeyRing 429 is // selected only by its structured code plus exact request-model match; // keyring_all_limited and every ambiguous response remain account-wide. diff --git a/src/util.ts b/src/util.ts index 7755425244..4bfd222582 100644 --- a/src/util.ts +++ b/src/util.ts @@ -556,18 +556,30 @@ export function inspectContextOverflow(status: number, bodyText: string): Contex }; } -// #4452: an intermediate gateway that rewrites EVERY rejection into one generic -// body (Cherry Studio et al. strip the token counts / window from each overflow -// 400) makes inspectContextOverflow return isOverflow:false even when the payload -// really overflows — so bili's whole overflow self-heal (#1195 same-request -// refold, armOverflowShrink, window learn) keys off a marker that is never there -// and silently never engages, starving the route of calibration. This predicate -// names exactly that shape — a 4xx overflow-SHAPED reject whose body matches no -// pattern — so the relay can log it loudly instead of the opaque `← upstream 400` -// line. Deliberately EXCLUDES 5xx (those already self-heal via the local estimate, +// #4452: an intermediate gateway that rewrites a payload-size rejection into one +// generic body (Cherry Studio et al. strip the token counts / window) makes +// inspectContextOverflow return isOverflow:false even when the payload really +// was too large — so bili's whole overflow self-heal (#1195 same-request refold, +// armOverflowShrink, window learn) keys off a marker that is never there and +// silently never engages, starving the route of calibration. This predicate +// names exactly that shape — an unmarked size-wall reject whose body matches no +// pattern — so the relay can log it loudly instead of the opaque +// `← upstream 413` line. +// +// ONLY 413 carries that meaning on its own. A 400 is the generic "malformed / +// schema-invalid / role-rejected / auth-shaped" bucket across relays: the SAME +// opaque body (`{"error":{"message":"Internal server error", +// "code":"bad_request"}}`) is emitted for a truncated JSON body, an unsupported +// field and a genuinely oversized payload alike, so treating every unmarked 400 +// as evidence of overflow manufactured a false overflow diagnosis for ordinary +// malformed requests (#4452 follow-up). 413 Request Entity Too Large is the +// status whose SEMANTICS are already the payload-size verdict, which is why it +// can be diagnosed without a body marker. +// +// Deliberately EXCLUDES 5xx (those already self-heal via the local estimate, // #604) and any body that DOES carry a marker (self-heal engages normally then). export function isMarkerlessOverflowReject(status: number, bodyText: string): boolean { - return (status === 400 || status === 413) && !CONTEXT_OVERFLOW_PATTERNS.some((p) => p.test(bodyText)); + return status === 413 && !CONTEXT_OVERFLOW_PATTERNS.some((p) => p.test(bodyText)); } /** Default cap on the output-headroom reservation, as a fraction of the diff --git a/tests/issue4452-markerless-reject.test.ts b/tests/issue4452-markerless-reject.test.ts index 817cbda1bb..d3bd1c9749 100644 --- a/tests/issue4452-markerless-reject.test.ts +++ b/tests/issue4452-markerless-reject.test.ts @@ -2,18 +2,27 @@ import test from "node:test"; import assert from "node:assert/strict"; import { inspectContextOverflow, isMarkerlessOverflowReject } from "../src/util.ts"; -// #4452: an intermediate gateway that rewrites every overflow rejection into one +// #4452: an intermediate gateway that rewrites a payload-size rejection into one // generic body (Cherry Studio strips the token counts / window) defeats the // overflow MARKER gate — inspectContextOverflow returns isOverflow:false, so the // whole self-heal never engages and the route starves of calibration. The relay // must surface that shape explicitly (isMarkerlessOverflowReject) instead of the -// opaque `← upstream 400` line. Gate discipline (#3617): assert BOTH directions — -// the marker-less shape IS flagged, and anything self-heal already handles (a real -// marker, a 5xx, a non-overflow 4xx) is NOT. +// opaque `← upstream 413` line. Gate discipline (#3617): assert BOTH directions — +// the marker-less SIZE verdict (413) IS flagged, and anything self-heal already +// handles (a real marker, a 5xx, a non-size 4xx) is NOT. +// +// #4452 follow-up: the generic 400 is NOT a size verdict. Every ordinary +// malformed / schema-invalid / role-rejected request across relays answers with +// the SAME opaque body, so diagnosing an unmarked 400 as markerless overflow +// manufactured a false overflow warning for requests that carry no evidence of +// being too large at all. // Verbatim from the #4438/#4452 field report — identical for every request id. const GENERIC_BODY = '{"error":{"message":"Internal server error","type":"invalid_request_error","code":"bad_request"}}'; +// A schema/malformed 400 body in the shape a strict backend emits. +const MALFORMED_BODY = '{"error":{"message":"Invalid request: messages[3].content must be a string or an array","type":"invalid_request_error","code":"invalid_value"}}'; + const OPENAI_OVERFLOW_BODY = JSON.stringify({ error: { message: @@ -25,35 +34,53 @@ const OPENAI_OVERFLOW_BODY = JSON.stringify({ const CJK_OVERFLOW_BODY = JSON.stringify({ error: { message: "上下文长度超出限制", type: "invalid_request_error" } }); -test("generic gateway body on 400 is overflow-shaped but carries no parseable marker", () => { +test("generic gateway body on 413 is the markerless SIZE verdict and is flagged", () => { // The first-order failure: the marker gate itself says "not overflow". + assert.equal(inspectContextOverflow(413, GENERIC_BODY).isOverflow, false); + // …yet a 413 whose body dropped the marker is exactly the shape the relay + // must flag as self-heal-starved. + assert.equal(isMarkerlessOverflowReject(413, GENERIC_BODY), true); +}); + +test("an unmarked 400 is NOT diagnosed as overflow (generic malformed/schema rejection)", () => { + // Neither body is overflow evidence: no status semantics and no marker. assert.equal(inspectContextOverflow(400, GENERIC_BODY).isOverflow, false); - // …yet it is exactly the shape the relay must now flag as self-heal-starved. - assert.equal(isMarkerlessOverflowReject(400, GENERIC_BODY), true); + assert.equal(isMarkerlessOverflowReject(400, GENERIC_BODY), false); + // The same must hold for a clearly schema-shaped 400 body. + assert.equal(inspectContextOverflow(400, MALFORMED_BODY).isOverflow, false); + assert.equal(isMarkerlessOverflowReject(400, MALFORMED_BODY), false); }); -test("generic gateway body on 413 is also flagged", () => { - assert.equal(inspectContextOverflow(413, GENERIC_BODY).isOverflow, false); - assert.equal(isMarkerlessOverflowReject(413, GENERIC_BODY), true); +test("an unmarked 413 is not double-counted as a marker overflow", () => { + // 413 without a marker: the self-heal gate is closed, which is exactly why + // the warning exists; it must never claim isOverflow. + assert.equal(inspectContextOverflow(413, MALFORMED_BODY).isOverflow, false); + assert.equal(isMarkerlessOverflowReject(413, MALFORMED_BODY), true); }); test("a body WITH an overflow marker is NOT markerless (self-heal engages normally)", () => { assert.equal(inspectContextOverflow(400, OPENAI_OVERFLOW_BODY).isOverflow, true); assert.equal(isMarkerlessOverflowReject(400, OPENAI_OVERFLOW_BODY), false); + assert.equal(inspectContextOverflow(413, OPENAI_OVERFLOW_BODY).isOverflow, true); + assert.equal(isMarkerlessOverflowReject(413, OPENAI_OVERFLOW_BODY), false); }); test("a CJK-marked overflow body is NOT markerless", () => { assert.equal(inspectContextOverflow(400, CJK_OVERFLOW_BODY).isOverflow, true); assert.equal(isMarkerlessOverflowReject(400, CJK_OVERFLOW_BODY), false); + assert.equal(isMarkerlessOverflowReject(413, CJK_OVERFLOW_BODY), false); }); test("5xx is excluded — those already self-heal via the local estimate (#604)", () => { assert.equal(isMarkerlessOverflowReject(502, GENERIC_BODY), false); assert.equal(isMarkerlessOverflowReject(503, GENERIC_BODY), false); + assert.equal(isMarkerlessOverflowReject(507, GENERIC_BODY), false); }); test("non-overflow 4xx and success codes are excluded (no false alarm on auth/malformed)", () => { assert.equal(isMarkerlessOverflowReject(200, GENERIC_BODY), false); assert.equal(isMarkerlessOverflowReject(401, GENERIC_BODY), false); + assert.equal(isMarkerlessOverflowReject(404, GENERIC_BODY), false); + assert.equal(isMarkerlessOverflowReject(422, GENERIC_BODY), false); assert.equal(isMarkerlessOverflowReject(429, GENERIC_BODY), false); }); diff --git a/tests/issue5152-owner-beat-survives-host-stall.test.ts b/tests/issue5152-owner-beat-survives-host-stall.test.ts index 671977001c..900a53e120 100644 --- a/tests/issue5152-owner-beat-survives-host-stall.test.ts +++ b/tests/issue5152-owner-beat-survives-host-stall.test.ts @@ -53,13 +53,21 @@ async function startReceiver(dir: string): Promise<Receiver> { const portFile = `${log}.port`; const child = spawn(process.execPath, [RECEIVER, log], { stdio: "ignore", detached: true }); child.unref(); + // Poll for a VALID port, not mere file existence: the receiver creates the + // file before its write lands, so existsSync can win that race on a loaded + // runner and read back an empty file (Number("") === 0). + let port = 0; + let portText = ""; const deadline = Date.now() + 15_000; - while (!existsSync(portFile) && Date.now() < deadline) { + while (port === 0 && Date.now() < deadline) { + if (existsSync(portFile)) { + portText = readFileSync(portFile, "utf8").trim(); + const parsed = Number(portText); + if (Number.isInteger(parsed) && parsed > 0) port = parsed; + } Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 20); } - assert.ok(existsSync(portFile), "receiver must publish its port"); - const port = Number(readFileSync(portFile, "utf8").trim()); - assert.ok(Number.isInteger(port) && port > 0, `receiver port must be a number, got ${port}`); + assert.ok(port > 0, `receiver must publish its port (file held "${portText}")`); return { port, beats: () => (existsSync(log) ? readFileSync(log, "utf8").trim().split("\n").filter(Boolean).map(Number) : []), diff --git a/tests/rejection-diagnostics.test.ts b/tests/rejection-diagnostics.test.ts new file mode 100644 index 0000000000..b6802fdb7a --- /dev/null +++ b/tests/rejection-diagnostics.test.ts @@ -0,0 +1,149 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { rejectionDiagnostics, rejectionDiagnosticSuffix } from "../src/server/rejection-diagnostics.ts"; + +// #4452: a fan-out relay answers a total failure with ONE aggregate body whose +// real cause sits nested in the last attempt. bili's log line keeps only the +// first 600 bytes, i.e. "all targets failed" — the cause was undiagnosable. +// These pin the extractor's contract: read ONE terminal nested message, bounded +// and credential-scrubbed, changing nothing about a body that carries none. + +/** Build the aggregate shape a fan-out relay returns on total failure. */ +function aggregate(attempts: unknown[], outer = "All targets failed"): string { + return JSON.stringify({ error: { message: outer, type: "upstream_error" }, attempts }); +} + +function attempt(message: unknown): Record<string, unknown> { + return { target: "t", details: { error: { message } } }; +} + +test("reads the terminal attempt's nested cause", () => { + const body = aggregate([attempt("connection reset by peer"), attempt("upstream returned 529 overloaded")]); + const diag = rejectionDiagnostics(body); + assert.ok(diag); + assert.equal(diag.message, "upstream returned 529 overloaded"); + assert.equal(diag.attemptIndex, 1); + assert.equal(diag.attempts, 2); +}); + +test("a single attempt is indexed 0 and the suffix omits the position", () => { + const diag = rejectionDiagnostics(aggregate([attempt("model not found")])); + assert.ok(diag); + assert.equal(diag.attemptIndex, 0); + assert.equal(diag.attempts, 1); +}); + +test("later attempts win over earlier retry noise", () => { + const body = aggregate([attempt("noise early"), attempt("noise mid"), attempt("densely informative final cause")]); + assert.equal(rejectionDiagnostics(body)?.message, "densely informative final cause"); +}); + +test("a terminal attempt with no usable message falls back to the nearest informative earlier one", () => { + const body = aggregate([attempt("real cause"), { target: "t", details: {} }, attempt(undefined)]); + const diag = rejectionDiagnostics(body); + assert.ok(diag); + assert.equal(diag.message, "real cause"); +}); + +test("an empty nested message is skipped", () => { + const body = aggregate([attempt("")]); + assert.equal(rejectionDiagnostics(body), undefined); + assert.equal(rejectionDiagnosticSuffix(body), ""); +}); + +test("empty / non-string message at every attempt yields no diagnostic", () => { + assert.equal(rejectionDiagnostics(aggregate([attempt(12), attempt(null), attempt("")])), undefined); + assert.equal(rejectionDiagnosticSuffix(aggregate([attempt(12), attempt(null)])), ""); +}); + +test("a payload with no attempts[] is untouched (no diagnostic)", () => { + const bodies = [ + '{"error":{"message":"Internal server error"}}', + '{"error":{"message":"context_length_exceeded: 200000 tokens"}}', + '{"attempts":[]}', + '{"attempts":"not-an-array"}', + "{not json", + "Internal Server Error", + "<html><body>502 Bad Gateway</body></html>", + "", + ]; + for (const body of bodies) { + assert.equal(rejectionDiagnostics(body), undefined, `expected none for: ${body}`); + assert.equal(rejectionDiagnosticSuffix(body), "", `expected empty suffix for: ${body}`); + } +}); + +test("non-object / non-array attempt entries are skipped, not thrown on", () => { + const body = aggregate([null, "string-attempt", 4, [], { details: null }, { details: { error: null } }, attempt("surviving cause")]); + assert.equal(rejectionDiagnostics(body)?.message, "surviving cause"); +}); + +test("null / undefined payload is tolerated and yields no suffix", () => { + assert.equal(rejectionDiagnostics(null), undefined); + assert.equal(rejectionDiagnostics(undefined), undefined); + assert.equal(rejectionDiagnosticSuffix(null), ""); + assert.equal(rejectionDiagnosticSuffix(undefined), ""); +}); + +test("whitespace, newlines and control shapes (incl. C1 NEL) are collapsed into ONE line", () => { + const hostile = "boom\r\n[attacker] ← upstream 200: forged line\u0085forged2\n\tsecond\u009c"; + const suffix = rejectionDiagnosticSuffix(aggregate([attempt(hostile)])); + assert.ok(!suffix.includes("\n"), "no newline may survive into the log line"); + assert.ok(!suffix.includes("\r"), "no carriage return may survive"); + assert.ok(!suffix.includes("\u0085"), "C1 next-line (U+0085) may not survive as a field break"); + assert.ok(!suffix.includes("\u009c"), "C1 file-separator (U+009C) may not survive as a field break"); +}); + +test("credential-shaped text is scrubbed by the shared sanitizer", () => { + const body = aggregate([attempt("rejected: authorization: Bearer abcdefghijklmnopqrstuvwxyz012345")]); + const suffix = rejectionDiagnosticSuffix(body); + assert.ok(!suffix.includes("abcdefghijklmnopqrstuvwxyz012345"), "bearer token must not reach the log"); +}); + +test("an sk- API key echoed by a provider is masked", () => { + const suffix = rejectionDiagnosticSuffix(aggregate([attempt("incorrect key sk-proj-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA supplied")])); + assert.ok(!suffix.includes("sk-proj-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA")); +}); + +test("a long cause is clamped, and the clamp is bounded", () => { + const suffix = rejectionDiagnosticSuffix(aggregate([attempt("x".repeat(4000))])); + // 300 retained chars + the ellipsis marker + the fixed prefix. + assert.ok(suffix.length < 340, `suffix too long: ${suffix.length}`); + assert.ok(suffix.endsWith(" …")); +}); + +test("an oversized body is not parsed at all", () => { + const huge = aggregate([attempt("y".repeat(64 * 1024 + 10))]); + assert.equal(rejectionDiagnostics(huge), undefined); + assert.equal(rejectionDiagnosticSuffix(huge), ""); +}); + +test("the suffix never echoes the outer narrative or the raw payload", () => { + const body = aggregate([attempt("inner cause")], "All targets failed: secret-route-label"); + const suffix = rejectionDiagnosticSuffix(body); + assert.ok(!suffix.includes("All targets failed")); + assert.ok(!suffix.includes("secret-route-label")); + assert.ok(!suffix.includes("{"), "no JSON structure may leak into the log"); +}); + +test("reads the terminal provider cause from CCR's nested error envelope", () => { + const body = JSON.stringify({ + error: { + message: "All target providers failed.", + attempts: [attempt("earlier transient error"), attempt("messages.32: tool_use has no following tool_result")], + }, + }); + assert.equal(rejectionDiagnostics(body)?.message, "messages.32: tool_use has no following tool_result"); + assert.equal(rejectionDiagnostics(body)?.attemptIndex, 1); +}); + +test("terminal control bytes cannot reach the diagnostic line", () => { + const body = aggregate([attempt("refused\u001b[2J\u0000\u007f by provider")]); + assert.equal(rejectionDiagnostics(body)?.message, "refused [2J by provider"); +}); + +test("the parsing budget counts UTF-8 bytes rather than UTF-16 characters", () => { + const body = aggregate([attempt("错".repeat(24 * 1024))]); + assert.ok(body.length < 64 * 1024); + assert.equal(rejectionDiagnostics(body), undefined); +});