Skip to content
Closed
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag
- **系统架构文档体系落地(owner 指令 2026-11-08):`docs/architecture/` 成为实时维护的架构枢纽**:新增 `docs/architecture.md` / `docs/architecture.zh-CN.md` 总枢纽(整体架构图中英双版转正 + 8 模块总表),首个样板模块页 `docs/architecture/storage.{md,zh-CN.md}`(统一存储:职责/原理/工作机制/硬不变量,配模块图中英双版)。文件级地图 `reference/architecture.md` 补齐 125 个缺失条目(新存储层 / marker / fastpath / lanes 等全部新模块)并升级为 CI 强制全量覆盖。新增 CI 车道 `architecture-docs`(`tools/check-architecture-docs.mjs`):地图覆盖全 `src/` 文件、总表↔页面成对、图 svg 优先 + png 备选齐备、SVG 基础良构(未转义 `&` 曾让英文图线上破图)。AGENTS.md「Documentation Layout」新增实时维护规则:改结构 ⇒ 同 PR 更新页 + 图 + 总表。其余 7 个模块页(pipeline / lanes / kernel / identity / fastpaths / webui / update)同一 PR 全部补齐——总表 8 行全 ✅(每页 = 中英双版文档 + 中英双版图)。

### Fixed
* **普通格式/参数错误的上游 400 不再被误诊为「上下文溢出」(#4452)**:`isMarkerlessOverflowReject` 原判据是 `(400 || 413) && 无任何溢出标记`,于是畸形请求、schema 校验失败、role 顺序不对的 400(各家中转站返回的正文几乎一模一样)全部命中,日志打出「无溢出标记 → 自愈无法启动 → 该路由校准饥饿」的警告——而这类请求根本没有「太大」的证据,纯属制造假警报。现在只有 **413** 才成立:413 本身就是载荷过大这一语义的判据,正文丢了标记才是真正的中转站重写问题;400 交给既有的溢出标记解析器(未命中即普通错误,不告警)。已知溢出的解析与 refold / emergency-shrink / window-learn 全部决策一字未动。
* **中转站拒绝原因不再被截断在「All targets failed」之后(#4452)**:`← upstream <status>: <snippet>` 只保留正文前 600 字节,而扇出型中转站在全部目标失败时返回的是聚合体,真正的死因在最后一个尝试的 `details.error.message` 里——永远落在截断之外,拒绝原因因此无法从日志诊断。现在该行追加一个有界、脱敏的终局原因(`; terminal cause [attempt N/M]: …`):只读 `attempts[last].details.error.message` 一条、经共享 `redactSecretsInText` 洗凭证、折叠全部空白(杜绝换行伪造日志行)、上限 300 字符、正文超过 64 KiB 直接不解析;正文里没有该结构时后缀为空,日志行与原实现逐字节一致。纯诊断,不改重试判定与响应字节。
- **dsh 桌面副本自己更新自己时也跑 boot smoke(#2082 后续)**:#2091 的 boot smoke 门只接在 `refreshDshDesktopCopy` 上(别的 bili 进程替桌面副本刷新的那条路)。代理本身就从桌面副本运行时,`hostManagedInstall` 按 #1575 放行,走的是普通自更新路径——周期检查 / `bili update`(`checkForUpdate`)和 advisory 强制安装 / 面板安装(`forceInstallVersion`)——这两处调 `installViaTarball` 没带 `bootSmoke`,入口能解析但加载即抛错的包只过 `node --check` 就被换上去,正是这道门要挡的状态。现在两条路径在安装目录是 dsh 桌面副本时同样启用 boot smoke,失败回滚到旧副本;其他安装布局行为不变。
- **A compress call that folded nothing no longer rewrites every `<acp>` tag on the next request (follow-up to #5608-C/D, #5797).** `applyRanges` armed the deferred one-time ref/ledger alignment (`session.pendingLedgerAlign`) right after committing the kernel's returned state — before the `blocksCreated === 0` returns — so a refused compress (range too small, cannot anchor, already compressed) or a no-op armed it too. The next save then ran `restampRefsToLedger`, which raised `timestampsHighWaterRefSeq` to the highest ref unconditionally: on a session whose refs already sit on their ledger seats (`kept=N restamped=0 rehomed=0`) no number moved, yet every tag minted so far dropped below the mark and lost its `time=` attribute — the whole cached prefix re-billed after a call whose receipt says nothing was folded. Now the marker is armed only once a block was really created; a real fold arms it exactly as before.
- **压缩树图:节点标题不再一闪即逝,点击压缩块能真正看到被压缩的消息内容**:此前树节点只有原生 SVG `<title>` 悬停提示——选中节点或 5s 轮询重画时 DOM 一变提示就消失,实测「显示一下马上消失」;右侧 Block details 里块的消息行虽带消息编号(mNNNNN)但内容全是一句「消息已不在会话中」(宿主原生压缩把消息从上下文剪掉后,字节无处可取)。现在:① 悬停提示改为页面内常驻浮层(.g-tip)——跟随指针、显示完整块号/tier/消息范围/完整主题/tokens/消息数,选中节点或轮询重画都不再打断,离开图形区才隐藏;悬停时节点卡片上的主题行同时原地扩宽(~20 字 → 20 字上限的完整截断,离开复原);② 5s 轮询只回传瘦节点(无主题/摘要/tokens),重画前先与详情富块按 blockId 合并——节点标题、浮层元信息、消息数不再在每次 tick 后退化成光秃秃的块号;③ 服务端新增**冻结快照水合**:压缩块落盘时已把原始消息逐条存进 blockContents(`[role]`/`[role • tool]` 头 + 原文,pi 宿主 user 消息还带 mNNNNN 标记行)——水合按**标记锚定**对齐:pi 宿主打进 user 消息正文的 mNNNNN 标记是身份戳,每个标记用两条独立证据之一锚死——**恒等锚**(标记号=该位当前 ref,从未被重编号)或**冻结 span 算术锚**(标记的冻结编号 − 块冻结 span 起点 = 快照位序;span 与标记同批写入,冷加载 restamp(#5196 parked 段换新号)重编了整块 ref 也照样成立——实测 b27 全块 127 条因此恢复,旧规则因标记号已不在 ref 表里整块拒水合);锚点之间的段仅在条数平衡时按位 zip。标记既非恒等也非算术锚(冻结上下文混入外来消息/span 无法证明)、锚点乱序、无锚非叶块(快照 span 的是 effective 超集)一律拒绝,未证明的位置保留诚实的 missing 标记——宁可不显示,不冒错位张冠李戴的风险(预览 + 点击展开取原文,`/message?ref=` 同享该层)。实测大会话(253 块/9771 个直接消息引用):7789 条(80%)恢复出真实预览与原文,其余保持 missing。回归:tests/web-blocks-tree.test.ts 冻结快照水合 7 用例(按位水合 / 条数不符拒 / 标记移位旧锚 / 宿主剪块中消息的锚定恢复 / restamp 整块恢复 / 算术锚优先于巧合 ref 碰撞 / 无锚标记整块拒(含 click-to-open 双向断言)),源码级断言更新为合并后重画(renderCollapSvg(merged)。
Expand Down
1 change: 1 addition & 0 deletions reference/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ billion-context/
│ │ ├── prepare-google.ts # Gemini request preparation incl. :countTokens twin (#1440)
│ │ ├── prepare-openai.ts # OpenAI chat request preparation (#1440)
│ │ ├── prepare-responses.ts # Responses request preparation + codex compact/prompt-cache helpers (#1440)
│ │ ├── rejection-diagnostics.ts # Bounded terminal provider cause from direct or nested aggregate rejections
│ │ ├── relay.ts # Upstream relay zone: forward() + fake-completion recovery (#1440 P2 cut 4)
│ │ ├── request-cost.ts # Per-request local-cost line (outbound measured at the real wire body) + #5408 heartbeat + wall-clock slow-session advisory, facts only (#903/#4528/#5744)
│ │ ├── responses-ordering.ts # #564 Responses assistant-run ordering repair + producer-owned turn-separator provenance (#2627)
Expand Down
122 changes: 122 additions & 0 deletions src/server/rejection-diagnostics.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
/**
* #4452: bounded terminal-cause diagnostics for a relayed upstream REJECTION.
*
* A relay/gateway that fans a request out across candidate targets may answer a
* total failure with ONE aggregate body carrying every per-target attempt, e.g.:
*
* { "error": { "message": "All targets failed", "type": "upstream_error" },
* "attempts": [ { "target": "…", "details": { "error": { "message": "…" } } }, … ] }
*
* bili's own log line (`← upstream <status>: <snippet>`) keeps only the first
* 600 bytes of that body, so the operator sees the OUTER narrative — "all
* targets failed" — while the per-target cause sits further in. This module
* reads exactly ONE such nested cause (the last usable attempt's) when a body
* has that shape, reporting its attempt position rather than guessing causality.
*
* Hard bounds, deliberately:
* - ONE message, never a recursive walk: no `JSON.stringify` of the payload, no
* traversal of `cause`/`errors`/`attempts` chains, no echoing of the request
* body. Only one `attempts[].details.error.message`, with the aggregate
* accepted directly or beneath its standard `error` envelope.
* - Credential scrubbing goes through the shared log-mask sanitizer
* (`redactSecretsInText`), the SAME choke point the logger applies, so a
* provider that echoes the rejected key cannot smuggle it into the log. It is
* a high-signal credential scrubber, NOT a general privacy filter: it does not
* mask arbitrary URLs or paths, and the far stronger guarantee here is
* structural — only this one nested string is ever read, never a body dump.
* - Bounded to `MAX_DIAGNOSTIC_CHARS`; ALL control characters (C0+C1, incl.
* NEL U+0085) and whitespace collapsed so a newline (the `\r\n` shape or any
* C1 line-break) can never forge a second log field.
* - Absent, non-string, empty, or structurally-unrelated payloads return
* `undefined`: the caller then logs exactly what it logged before, so a
* rejection that carries no attempt detail is unchanged byte-for-byte.
*
* Recovery policy is NOT touched here — this module only reads a rejection
* body; every retry/refold/arm decision stays in the relay.
*/

import { redactSecretsInText } from "../log-mask.js";

/** Upper bound on the retained cause text, independent of the outer snippet. */
const MAX_DIAGNOSTIC_CHARS = 300;

/** Do not even attempt to parse a body larger than this: an aggregate error
* reply is small (a few KB); a multi-megabyte body is a stream (or an HTML
* error page), not a JSON rejection envelope, and parsing it would cost the
* request more than the diagnostic is worth. */
const MAX_PARSED_BYTES = 64 * 1024;

interface DiagnosticResult {
/** The last usable attempt's nested cause, sanitized + bounded. */
message: string;
/** 0-based index of the attempt the cause came from (for the log line). */
attemptIndex: number;
/** How many attempts the aggregate body carried. */
attempts: number;
}

/** Collapse control chars (C0+C1 — incl. NEL U+0085) and all whitespace to
* single spaces, then clamp, so no control byte can survive as a forged field
* break in the log line. */
function bound(text: string): string {
const collapsed = text.replace(/[\u0000-\u001f\u007f-\u009f]/g, " ").replace(/\s+/g, " ").trim();
if (collapsed.length <= MAX_DIAGNOSTIC_CHARS) return collapsed;
return `${collapsed.slice(0, MAX_DIAGNOSTIC_CHARS)} …`;
}

/** Read `value[name]` only when `value` is a non-array object, so a nested
* field is reached without a generic record guard that would prove nothing
* about the field actually consumed here. */
function child(value: unknown, name: string): unknown {
if (typeof value !== "object" || value === null || Array.isArray(value)) return undefined;
return (value as Record<string, unknown>)[name];
}

/**
* Read the single last-reported nested cause out of a relay rejection body.
*
* `payload` is the raw upstream error-body text (the same string the relay
* already buffers to detect an overflow). Returns `undefined` whenever there is
* no usable `attempts[].details.error.message`, leaving the caller's existing
* logging untouched.
*
* Selection rule: the LAST attempt that carries a non-empty
* `details.error.message`, scanning from the end. If the terminal attempt is
* silent, an earlier cause is reported with its actual position and is NOT
* labeled terminal: its existence does not establish why the last stage failed.
*/
export function rejectionDiagnostics(payload: string | null | undefined): DiagnosticResult | undefined {
if (typeof payload !== "string" || payload.length === 0) return undefined;
if (Buffer.byteLength(payload, "utf8") > MAX_PARSED_BYTES) return undefined;
const trimmed = payload.trim();
// A rejection body must be a JSON object envelope — anything else (an HTML
// error page, a bare text message, an SSE frame) carries no attempts[].
if (!trimmed.startsWith("{") || !trimmed.endsWith("}")) return undefined;
let parsed: unknown;
try {
parsed = JSON.parse(trimmed);
} catch {
return undefined;
}
const attempts = child(parsed, "attempts") ?? child(child(parsed, "error"), "attempts");
if (!Array.isArray(attempts) || attempts.length === 0) return undefined;
for (let i = attempts.length - 1; i >= 0; i--) {
const raw = child(child(child(attempts[i], "details"), "error"), "message");
if (typeof raw !== "string") continue;
const sanitized = bound(redactSecretsInText(raw));
if (sanitized.length === 0) continue;
return { message: sanitized, attemptIndex: i, attempts: attempts.length };
}
return undefined;
}

/** Render the diagnostic as the compact suffix the relay appends to its
* `← upstream <status>` line. Returns "" when there is no usable cause, so the
* caller can concatenate unconditionally and stay byte-identical without one.
* Pure numbers and the sanitized cause only — never the payload. */
export function rejectionDiagnosticSuffix(payload: string | null | undefined): string {
const diag = rejectionDiagnostics(payload);
if (diag === undefined) return "";
const position = diag.attempts > 1 ? ` attempt ${diag.attemptIndex + 1}/${diag.attempts}` : "";
return `; ${diag.attemptIndex === diag.attempts - 1 ? "terminal" : "reported"} cause${position}: ${diag.message}`;
}
25 changes: 19 additions & 6 deletions src/server/relay.ts
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ import { applyOutputSteering, applyOutputSteeringJson } from "../output-steering
import { estimateWireOverhead } from "./budget.js";
import { buildForwardHeaders, connectionNamedHeaders, RESPONSE_ONLY_STRIP_HEADERS, safeSessionId, UPSTREAM_HOP_HEADERS } from "./headers.js";
import { bodyDumpEnabled, logDumpFailure } from "./observability.js";
import { rejectionDiagnosticSuffix } from "./rejection-diagnostics.js";
import { effectiveAbsorbBlock, effectiveBlockOpsEnabled } from "./prepare-responses.js";
import { awaitDrain, bufferToStream, dumpStreamToFile, pipeThrough, readStreamToBuffer } from "./stream-io.js";
import { buildForwardTarget, measuredBreakdownClause, measuredOutbound, outboundPayloadBreakdown, registerRequestAbort, beginStreamKeepalive, effectiveTokenCount, googlePathKind, imageBillingFor, imageTokenCapFor, inboundResignCredential, outboundContextEstimates, resolveUpstream, stripKernelSummaries, usageGradeInputBaseline, DOOMED_PROBE_ABSOLUTE_BYTES, DOOMED_PROBE_BYTES_PER_TOKEN, type Prepared } from "../server.js";
Expand Down Expand Up @@ -2052,14 +2053,18 @@ async function forwardCaptured(
}
log("warn", `[${s.id}] upstream image-count wall (cap ${info.imageCountCap}, request carried ${info.imageCountSeen ?? "?"}) — learned for pre-send convergence on this route; no token-space arm (#4124)`);
} else if (isMarkerlessOverflowReject(upstream.status, errText)) {
// #4452: the gateway rewrote this overflow-shaped reject into a body with no
// parseable marker, so bili's self-heal cannot engage from it — name that here
// (+ the last meter reading) so a body-rewriting gateway's silent starvation is
// diagnosable from the log rather than the opaque line further down.
// #4452: a 413 IS the payload-size verdict, yet this body carries no
// parseable marker, so bili's self-heal cannot engage from it — name that
// here (+ the last meter reading) so a body-rewriting gateway's silent
// starvation is diagnosable from the log rather than the opaque line
// further down. Only 413 reaches this branch: an unmarked 400 is the
// generic malformed/schema-rejected bucket across relays and says nothing
// about size, so it is never diagnosed as overflow here (see
// isMarkerlessOverflowReject).
const est = s.stats.lastInputTokens > 0 ? s.stats.lastInputTokens : 0;
const win = typeof s.metadata.effectiveContextLimit === "number" && s.metadata.effectiveContextLimit > 0 ? s.metadata.effectiveContextLimit : 0;
const meterNote = est > 0 && win > 0 ? `; last settled input ~${Math.round(est)} vs effective window ${win}` : "";
log("warn", `[${s.id}] ← upstream ${upstream.status} carries no parseable overflow marker (an intermediate gateway rewrote the rejection body?) — bili's overflow self-heal (refold / emergency-shrink / window-learn) CANNOT engage from this rejection${meterNote}; repeated at similar size = calibration-starved route (#4452)`);
log("warn", `[${s.id}] ← upstream ${upstream.status} carries no parseable overflow marker (an intermediate gateway rewrote the payload-size rejection body?) — bili's overflow self-heal (refold / emergency-shrink / window-learn) CANNOT engage from this rejection${meterNote}; repeated at similar size = calibration-starved route (#4452)`);
}
// #762: learn strict-echo on the MAIN request path too. The loop-only
// learner (src/loop/core.ts) never sees client-originated 400s, so a
Expand Down Expand Up @@ -2096,7 +2101,15 @@ async function forwardCaptured(
let snippet = bodyText.slice(0, 600).replace(/\s+/g, " ").trim();
if (bodyText.length > 600) snippet += " …";
if (!snippet) snippet = "(no body)";
loggerLog("warn", `[${errSid}] ← upstream ${upstream.status}${reqIdText}: ${snippet}`);
// #4452: a relay that fans out to several targets answers a total failure
// with ONE aggregate body whose real cause sits in the LAST attempt's
// `details.error.message` — past the 600-byte snippet above, so the log
// showed only "all targets failed". Append that single terminal cause
// (bounded + credential-scrubbed by the helper; empty when absent, leaving
// this line byte-identical to before). Read-only: no recovery decision
// here depends on it.
const terminalCause = rejectionDiagnosticSuffix(bodyText);
loggerLog("warn", `[${errSid}] ← upstream ${upstream.status}${reqIdText}: ${snippet}${terminalCause}`);
// #4204: feed this send's FINAL verdict. A model-scoped KeyRing 429 is
// selected only by its structured code plus exact request-model match;
// keyring_all_limited and every ambiguous response remain account-wide.
Expand Down
Loading
Loading