From 32b8bb7152a472265b6180085ebce31ee3063032 Mon Sep 17 00:00:00 2001 From: mz Date: Sun, 1 Nov 2026 16:53:25 +0300 Subject: [PATCH 1/9] fix(diagnostics): report large histories without attributing stalls --- CHANGELOG.md | 1 + reference/architecture.md | 1 + src/server.ts | 34 ------------------------ src/server/handle.ts | 3 ++- src/server/request-cost.ts | 34 ++++++++++++++++++++++++ tests/issue4528-event-loop-yield.test.ts | 13 ++++----- 6 files changed, 43 insertions(+), 43 deletions(-) create mode 100644 src/server/request-cost.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b28c8d421..eea0d587af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ### Fixed +* **「慢会话」告警把「消息条数多」说成了「事件循环已被卡死」(#4528 遗留文案)**:这条告警的触发条件只有一条——本轮入站消息数超过 1000,然后正文却断言「每回合的 O(messages) 同步扫描正在饿死事件循环」,并建议用户「关闭/重启会话或开启压缩」。实测里大历史请求常常几十毫秒就完成本地准备(热缓存、请求体小),此时告警内容与同一行 `local=` 毫秒数直接自相矛盾,还会把用户引向「重启会话」这种既不必要又有损的操作。现在告警改名为 `large-history`,只陈述已实测的事实(消息条数、入站字节、本地耗时毫秒),明说「本行只报条数,不把它归因为慢请求,实际耗时请看客户端」;触发条件、warn 级别、`local=`/`inbound=`/`outbound=`/`view=` 字段与 #5408 的 `noteRequest` 观测点全部逐字保留。历史长停顿的真实成因仍未证实,因此没有新增任何延迟阈值、重试、配置项或对会话历史的改写。该单元从 `src/server.ts` 抽出到 `src/server/request-cost.ts`(连同它的阈值常量与 `formatBytes`),`src/server/handle.ts` 的 12 处调用改从新模块导入,`src/server.ts` 不再转出。 * **#5412: 异步存储后端的「压缩冻结会话」现在也能自愈。** #5408 事故(一个会话连续 17 次摘要漂移后被保护机制冻住、8 小时无人知晓)的主修复 #5409 只教了同步存储路径(SQLite 直写)自救:漂移挂起期间保存会自动升级为全量重导入并清掉索引里的残留行。异步路径(opt-in 网络后端,如 mysql)那份孪生代码没搬——同样被冻住的会话在那上面永远救不回来,每次保存都重写不完整视图、下轮采样继续漂移。本 PR 在 `src/storage/shadow.ts` 的异步入队路径补上同一行对齐(`completeLog = hasRawTail || driftedSessions.has(id)`),行为与同步路径逐字一致;回归测试同时钉住正门(latch 挂起的折叠保存清掉外来行)与负门(未挂起保存不清扫)。 - **中断频繁的会话压缩提醒几乎不触发,积到几十万 token 才压一次(#5415,0.1.210 引入)**:#5293 的修复把每次用户中断后的增长参照点直接抬到当前读数,连「中断前已实测的真实增长」也一起清零;高频中断的客户端(实测一份日志 4.5 小时 85 次中断)永远攒不到增长台阶,30 万可压内容 idle 数小时,最后一次性压 33 万。现在吸收时把被清零的实测增长记入「欠账」(`nudge.absorbedDebt`)并累计吸收次数(`nudge.absorbCount`),两者都只在**第 2 次起**的中断后生效:单次中断字节级保持 #5293 的安静(不打扰用户刚改的主意——欠账再多也不当场弹提醒);第二次中断才证明会话是「高频中断型」而非「改主意」,欠账从此与新增量一起计入增长台阶,恢复 209 时代的节奏。另加「大积压排水」兑底:欠账>0 且可压量≥2×阈值且当前无未消化提醒时不再等增长台阶(与 #194 同哲学);无欠账/单次中断的会话决策字节不变。压缩成功时欠账与计数随其它基线一起清零;idle 日志同步显示 `growth X + absorbed Y across N aborts < floor Z`(单次中断显示 `held (single abort, #5293)`)便于排查。tests:kernel/tests/issue5415-absorb-debt-nudge.test.ts(欠账付台阶/单次中断钉死不弹/排水兑底/忽略提醒不重发/无欠账字节不变/压缩后清账+清计数);tests/issue5415-absorb-debt.test.ts(吸收记账算术、欠账+计数、累次累计、参照点已越过 anchor 不记账也计数、陈旧 flag 丢弃)。 - **长会话每回合把 Node 主线程卡死十几秒、网页面板冻结(#5337)**:每回合的压缩筛查把历史切成若干可折叠单元,逐个跑「能否折叠」判定——而每次判定都从零重扫全量消息数组(id 查找无索引,五处线性 `.find()`/`includes`),还重复重算全套整史统计(受保护/过期工具调用集合、工具配对表、末尾锚点、完整性预扫)。总工作量≈单元数×全史长度=历史规模平方级,且同步跑在主线程:2.3万条核心消息实测单回合约17秒(事件循环被占死,请求排队;#5304 同签名报告)。修复=把这批 O(n) 推导提升为每轮筛查**一次**构建(`buildProjectionShared`)、共享给同一轮所有单元;id→序号一次性建表替五处线性查找;「已丢弃」成员判断从 includes-in-some 改为 Set。判定逻辑本身一字未动(#3236 单一判定纪律),只是输入到达判定的方式变了。同机同语料实测:2万条消息完整推荐链 9028ms→约572ms(≈16×),回落到线性量级。单次调用路径(apply/report)保持原有内联计算,输出逐字节不变。kernel v0.0.175。tests/issue5337-shared-screening.test.ts(共享/非共享投影逐字段等价+降级边界一致+n=20000 耗时上界钉);既有 issue3406 曲线钉与 #3236 一致性回归不变。 diff --git a/reference/architecture.md b/reference/architecture.md index baa641963f..237fcf4b92 100644 --- a/reference/architecture.md +++ b/reference/architecture.md @@ -28,6 +28,7 @@ billion-context/ │ │ ├── prepare-openai.ts # OpenAI chat request preparation (#1440) │ │ ├── prepare-responses.ts # Responses request preparation + codex compact/prompt-cache helpers (#1440) │ │ ├── relay.ts # Upstream relay zone: forward() + fake-completion recovery (#1440 P2 cut 4) +│ │ ├── request-cost.ts # Per-request local-cost line + #5408 observation heartbeat + count-triggered large-history advisory (#903/#4528) │ │ ├── side-request.ts # Tool-surface check for auxiliary side requests │ │ └── stream-io.ts # Small response-body reader for non-2xx inspection │ ├── knobs.ts # Single knob resolver: env > config file > default (#2030) diff --git a/src/server.ts b/src/server.ts index 479fd32575..0367b528c3 100644 --- a/src/server.ts +++ b/src/server.ts @@ -96,7 +96,6 @@ import { conflictScanEnabled, isDesignBenign, scanClientPlugins, sniffScanClient import { clearConflictEvents, recordConflict, summarizeConflicts } from "./conflict-watch.js"; import { getStore, setGlobalSessionCommitListener } from "./persist.js"; import { log as loggerLog, configureLogger, getLogPath, closeLogger, isStreamWriteError, isBenignSocketRaceError, enterSessionContext, muteStderrLeg, onFileLegLive, shouldMuteStderrLegAfterFileLive, logTimestamp } from "./logger.js"; -import { noteRequest } from "./attach-heartbeat.js"; import { queryLogLines } from "./web/logs-query.js"; import { configFile, defaultLogFile, sessionsDir, stateDir } from "./paths.js"; import { atomicWriteInstanceFile, clearProxyInstanceFile, classifySameLanePeer, entryShape, findPresentUnverifiedSameLaneTwin, findSameLanePredecessor, findVerifiedSameLaneTwin, isPidAlive, listInstances, markPortReleased, probePid, readProxyInstanceFile, registerInstanceAndWarn, resolveOwnFingerprint, sameInstanceShape, SPAWN_FINGERPRINT_ENV, unregisterInstance, warnOnNewPeers, writeSpawnPortLog, type InstanceShape, type PidProbe, type ProxyInstanceFile, type RegistryEntry, type SameLanePeer } from "./instance.js"; @@ -153,7 +152,6 @@ import { BILI_HOP_HEADER, anthropicBetaContextWindow, capRegistryWindowByStandar import { connectionNamedHeaders, NO_IDENTITY_MESSAGE, stripRemoteCompactionBetaFeature, UPSTREAM_HOP_HEADERS } from "./server/headers.js"; import { installWebSocketBridge } from "./ws-bridge.js"; import { codexResponsesCodec, responsesCodec } from "./responses-ws.js"; -import { currentFetchTransport } from "./fetch-transport.js"; import { demoteGate, hasLeakedBiliToolsOnly, isSideRequest, outputBudgetField, resolveSideLane, restoreOutputBudget, SIDE_REQUEST_MAX_TOKENS, sideRequestGuard, stripLeakedBiliTools } from "./server/side-request.js"; import { DSH_COMPACTION_SHAPE_MSGS, dshCompactionRefusal, isDshCompactionCall } from "./server/dsh-compaction-guard.js"; import { clampOutgoingOutput, countSystemAndToolsTokens, emergencyNudge, estimateInputTokens, estimateWireOverhead, projectThinkingMass, readOutputBudget } from "./server/budget.js"; @@ -4544,38 +4542,6 @@ export function headerValue(req: http.IncomingMessage, name: string): string | u return undefined; } -function formatBytes(n: number): string { - if (n < 1024) return `${n}B`; - if (n < 1024 * 1024) return `${(n / 1024).toFixed(1)}KiB`; - if (n < 1024 * 1024 * 1024) return `${(n / (1024 * 1024)).toFixed(1)}MiB`; - return `${(n / (1024 * 1024 * 1024)).toFixed(1)}GiB`; -} - -// #903: per-request local-cost line, logged at every point where bili finishes -// its own processing and hands the request off (upstream forward, forged local -// response, or fail-fast error) — see handle(). outbound is the exact body value -// handed to forward() (string OR Buffer — Prepared.body is string|Buffer); wire -// bytes are counted with Buffer.byteLength since undici sends UTF-8 bytes even -// for string bodies (.length would count UTF-16 chars and undercount any -// non-ASCII injection). Omitted on paths that fail fast without forwarding. -// #4528: inbound histories past this size pay an O(messages) synchronous scan -// pass per turn on the main thread, large enough to starve the event loop and -// freeze every endpoint on the same listener — warn so a pathological session -// shows up in the log instead of being found through a frozen UI. -const SLOW_SESSION_MSG_WARN = 1000; -export function logRequestCost(log: (level: string, msg: string) => void, sessionId: string, msgs: number | null, inboundBytes: number, t0: number, outbound?: string | Buffer): void { - const ms = Math.max(0, Math.round(performance.now() - t0)); - const outboundField = outbound !== undefined ? `, outbound=${formatBytes(Buffer.byteLength(outbound))}` : ""; - const view = currentFetchTransport() ? ", view=ws-expanded" : ""; - log("info", `[${sessionId}] request: ${msgs ?? "?"} msgs, inbound=${formatBytes(inboundBytes)}${outboundField}, local=${ms}ms${view}`); - // #5408: this line is THE observation point — a hot-at-attach session that - // never produces one is silent compression loss (the incident signature). - noteRequest(sessionId); - if (msgs !== null && msgs > SLOW_SESSION_MSG_WARN) { - log("warn", `[${sessionId}] slow-session: ${msgs} msgs / inbound=${formatBytes(inboundBytes)} took local=${ms}ms — per-turn O(messages) synchronous scans are starving the event loop (#4528); consider closing/restarting the session or enabling compression`); - } -} - /** Thrown by readBody when the request body exceeds the configured receive-body * budget (limitsReceiveBytes). handle() catches this and attempts a 413 response; * readBody also destroys the request socket so a client that keeps streaming a diff --git a/src/server/handle.ts b/src/server/handle.ts index 9c212fb2ff..710f65ac00 100644 --- a/src/server/handle.ts +++ b/src/server/handle.ts @@ -71,7 +71,8 @@ import { handleIncrementalTurn, handleIncrementalProbe, incrementalTurnBodyShape import { handlePluginRender, PLUGIN_RENDER_PATH } from "../plugin-render.js"; import { ccrPluginWireOk, storeEffectiveCcr, storeEffectiveImageRescale, type CcrSettings } from "../store.js"; import { DEFAULT_DECIDE_MAX_TOKENS } from "../nudge-decide.js"; -import { BodyTooLargeError, buildForwardTarget, forward, forwardUpstreamUrl, googleModelFromPath, googlePathKind, headroomEffectiveLogged, headroomFallbackLogged, headerValue, imageBillingFor, imageReserveFor, imageTokenCapFor, isCountTokensRequest, isPreflightFailFast, logRequestCost, NON_CONVERSATION_RELAY_WARN_CAP, nonConversationRelayWarned, prepareCountTokens, preflightCompressIfNeeded, readBody, resolveKnownOutputCeiling, resolveUpstream, resignSettingsFor, selfAddressedLocalPath, scrubAnthropicPck, scrubCompatDrop, vertexAnthropicModelFromPath, warnRouteMissIfNew, type Prepared } from "../server.js"; +import { BodyTooLargeError, buildForwardTarget, forward, forwardUpstreamUrl, googleModelFromPath, googlePathKind, headroomEffectiveLogged, headroomFallbackLogged, headerValue, imageBillingFor, imageReserveFor, imageTokenCapFor, isCountTokensRequest, isPreflightFailFast, NON_CONVERSATION_RELAY_WARN_CAP, nonConversationRelayWarned, prepareCountTokens, preflightCompressIfNeeded, readBody, resolveKnownOutputCeiling, resolveUpstream, resignSettingsFor, selfAddressedLocalPath, scrubAnthropicPck, scrubCompatDrop, vertexAnthropicModelFromPath, warnRouteMissIfNew, type Prepared } from "../server.js"; +import { logRequestCost } from "./request-cost.js"; import { normalizeVertexModelId } from "../vertex.js"; // #5243: warn-once per model when the live codex model table is unavailable and diff --git a/src/server/request-cost.ts b/src/server/request-cost.ts new file mode 100644 index 0000000000..5d19945850 --- /dev/null +++ b/src/server/request-cost.ts @@ -0,0 +1,34 @@ +import { noteRequest } from "../attach-heartbeat.js"; +import { currentFetchTransport } from "../fetch-transport.js"; + +// #903: per-request local-cost line, logged at every point where bili finishes +// its own processing and hands the request off (upstream forward, forged local +// response, or fail-fast error) — see handle(). outbound is the exact body value +// handed to forward() (string OR Buffer — Prepared.body is string|Buffer); wire +// bytes are counted with Buffer.byteLength since undici sends UTF-8 bytes even +// for string bodies (.length would count UTF-16 chars and undercount any +// non-ASCII injection). Omitted on paths that fail fast without forwarding. +function formatBytes(n: number): string { + if (n < 1024) return `${n}B`; + if (n < 1024 * 1024) return `${(n / 1024).toFixed(1)}KiB`; + if (n < 1024 * 1024 * 1024) return `${(n / (1024 * 1024)).toFixed(1)}MiB`; + return `${(n / (1024 * 1024 * 1024)).toFixed(1)}GiB`; +} + +// #4528: the existing count threshold is an advisory policy, not a measured +// event-loop stall. Report the count and measured local preparation time; +// neither the count nor this diagnostic attributes historical slow requests. +const LARGE_HISTORY_MSG_WARN = 1000; + +export function logRequestCost(log: (level: string, msg: string) => void, sessionId: string, msgs: number | null, inboundBytes: number, t0: number, outbound?: string | Buffer): void { + const ms = Math.max(0, Math.round(performance.now() - t0)); + const outboundField = outbound !== undefined ? `, outbound=${formatBytes(Buffer.byteLength(outbound))}` : ""; + const view = currentFetchTransport() ? ", view=ws-expanded" : ""; + log("info", `[${sessionId}] request: ${msgs ?? "?"} msgs, inbound=${formatBytes(inboundBytes)}${outboundField}, local=${ms}ms${view}`); + // #5408: this line is THE observation point — a hot-at-attach session that + // never produces one is silent compression loss (the incident signature). + noteRequest(sessionId); + if (msgs !== null && msgs > LARGE_HISTORY_MSG_WARN) { + log("warn", `[${sessionId}] large-history: ${msgs} msgs / inbound=${formatBytes(inboundBytes)}, local=${ms}ms — count-triggered advisory (#4528), not evidence of event-loop starvation; use measured request timings to investigate slow requests`); + } +} diff --git a/tests/issue4528-event-loop-yield.test.ts b/tests/issue4528-event-loop-yield.test.ts index f9d361fd51..ec6c414585 100644 --- a/tests/issue4528-event-loop-yield.test.ts +++ b/tests/issue4528-event-loop-yield.test.ts @@ -2,7 +2,7 @@ import { describe, it } from "node:test"; import assert from "node:assert/strict"; import { performance } from "node:perf_hooks"; import { yieldToEventLoop } from "../src/yield-loop.js"; -import { logRequestCost } from "../src/server.js"; +import { logRequestCost } from "../src/server/request-cost.js"; // #4528: a large resent history makes bili run many O(messages) synchronous // passes back-to-back on Node's single main thread, starving every other @@ -43,20 +43,17 @@ describe("#4528 yieldToEventLoop hands control to the poll phase", () => { }); }); -describe("#4528 logRequestCost slow-session guard", () => { +describe("#4528 logRequestCost large-history guard", () => { function capture(): { lines: Array<[string, string]>; log: (level: string, msg: string) => void } { const lines: Array<[string, string]> = []; return { lines, log: (level, msg) => lines.push([level, msg]) }; } - it("warns when the inbound history has grown past the message threshold", () => { + it("warns once when the inbound history has grown past the message threshold", () => { const { lines, log } = capture(); logRequestCost(log, "sess-big", 1500, 11_000_000, performance.now() - 8000); - const warns = lines.filter(([l]) => l === "warn"); - assert.equal(warns.length, 1); - assert.match(warns[0][1], /slow-session: 1500 msgs/); - assert.match(warns[0][1], /inbound=/); - assert.match(warns[0][1], /local=\d+ms/); + assert.equal(lines.filter(([l]) => l === "warn").length, 1); + assert.ok(lines.some(([l, m]) => l === "info" && m.includes("sess-big") && m.includes("1500 msgs"))); }); it("does not warn for a healthy-sized session", () => { From e95c88c9ae96a74cc56b4f262e5d2c8bc65a4699 Mon Sep 17 00:00:00 2001 From: ci-check Date: Mon, 2 Nov 2026 00:35:11 +0800 Subject: [PATCH 2/9] test: pin large-history warning wording against stall-attribution regression (#5481) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The positive test only asserted warn-count and info-line content, so the factual rewording — the entire point of the PR — had no regression pin: reintroducing 'starving the event loop' attribution would pass CI. Restore content assertions on the warn line (label, inbound=/local= fields) and pin the explicit non-attribution clause. --- tests/issue4528-event-loop-yield.test.ts | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tests/issue4528-event-loop-yield.test.ts b/tests/issue4528-event-loop-yield.test.ts index ec6c414585..dce5dd46c1 100644 --- a/tests/issue4528-event-loop-yield.test.ts +++ b/tests/issue4528-event-loop-yield.test.ts @@ -52,7 +52,13 @@ describe("#4528 logRequestCost large-history guard", () => { it("warns once when the inbound history has grown past the message threshold", () => { const { lines, log } = capture(); logRequestCost(log, "sess-big", 1500, 11_000_000, performance.now() - 8000); - assert.equal(lines.filter(([l]) => l === "warn").length, 1); + const warns = lines.filter(([l]) => l === "warn"); + assert.equal(warns.length, 1); + assert.match(warns[0][1], /large-history: 1500 msgs/); + assert.match(warns[0][1], /inbound=/); + assert.match(warns[0][1], /local=\d+ms/); + // #5481: the wording IS the fix — factual count report, no stall attribution. + assert.match(warns[0][1], /not evidence of event-loop starvation/); assert.ok(lines.some(([l, m]) => l === "info" && m.includes("sess-big") && m.includes("1500 msgs"))); }); From 6ed0a2ef4df4dd03cc905907ccfb4eaf65c3b9cf Mon Sep 17 00:00:00 2001 From: ci-check Date: Mon, 2 Nov 2026 00:35:11 +0800 Subject: [PATCH 3/9] =?UTF-8?q?docs:=20correct=20changelog=20wording=20?= =?UTF-8?q?=E2=80=94=20the=20import=20moved,=2010=20call=20sites=20untouch?= =?UTF-8?q?ed=20(#5481)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Entry said 12 call sites were changed to import from the new module; git grep shows exactly 10 logRequestCost call sites in handle.ts (unchanged), and what moved was the single import statement. --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eea0d587af..62cede7c6a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ### Fixed -* **「慢会话」告警把「消息条数多」说成了「事件循环已被卡死」(#4528 遗留文案)**:这条告警的触发条件只有一条——本轮入站消息数超过 1000,然后正文却断言「每回合的 O(messages) 同步扫描正在饿死事件循环」,并建议用户「关闭/重启会话或开启压缩」。实测里大历史请求常常几十毫秒就完成本地准备(热缓存、请求体小),此时告警内容与同一行 `local=` 毫秒数直接自相矛盾,还会把用户引向「重启会话」这种既不必要又有损的操作。现在告警改名为 `large-history`,只陈述已实测的事实(消息条数、入站字节、本地耗时毫秒),明说「本行只报条数,不把它归因为慢请求,实际耗时请看客户端」;触发条件、warn 级别、`local=`/`inbound=`/`outbound=`/`view=` 字段与 #5408 的 `noteRequest` 观测点全部逐字保留。历史长停顿的真实成因仍未证实,因此没有新增任何延迟阈值、重试、配置项或对会话历史的改写。该单元从 `src/server.ts` 抽出到 `src/server/request-cost.ts`(连同它的阈值常量与 `formatBytes`),`src/server/handle.ts` 的 12 处调用改从新模块导入,`src/server.ts` 不再转出。 +* **「慢会话」告警把「消息条数多」说成了「事件循环已被卡死」(#4528 遗留文案)**:这条告警的触发条件只有一条——本轮入站消息数超过 1000,然后正文却断言「每回合的 O(messages) 同步扫描正在饿死事件循环」,并建议用户「关闭/重启会话或开启压缩」。实测里大历史请求常常几十毫秒就完成本地准备(热缓存、请求体小),此时告警内容与同一行 `local=` 毫秒数直接自相矛盾,还会把用户引向「重启会话」这种既不必要又有损的操作。现在告警改名为 `large-history`,只陈述已实测的事实(消息条数、入站字节、本地耗时毫秒),明说「本行只报条数,不把它归因为慢请求,实际耗时请看客户端」;触发条件、warn 级别、`local=`/`inbound=`/`outbound=`/`view=` 字段与 #5408 的 `noteRequest` 观测点全部逐字保留。历史长停顿的真实成因仍未证实,因此没有新增任何延迟阈值、重试、配置项或对会话历史的改写。该单元从 `src/server.ts` 抽出到 `src/server/request-cost.ts`(连同它的阈值常量与 `formatBytes`),`src/server/handle.ts` 的 `logRequestCost` 导入改指新模块(10 处调用本身未动),`src/server.ts` 不再转出。 * **#5412: 异步存储后端的「压缩冻结会话」现在也能自愈。** #5408 事故(一个会话连续 17 次摘要漂移后被保护机制冻住、8 小时无人知晓)的主修复 #5409 只教了同步存储路径(SQLite 直写)自救:漂移挂起期间保存会自动升级为全量重导入并清掉索引里的残留行。异步路径(opt-in 网络后端,如 mysql)那份孪生代码没搬——同样被冻住的会话在那上面永远救不回来,每次保存都重写不完整视图、下轮采样继续漂移。本 PR 在 `src/storage/shadow.ts` 的异步入队路径补上同一行对齐(`completeLog = hasRawTail || driftedSessions.has(id)`),行为与同步路径逐字一致;回归测试同时钉住正门(latch 挂起的折叠保存清掉外来行)与负门(未挂起保存不清扫)。 - **中断频繁的会话压缩提醒几乎不触发,积到几十万 token 才压一次(#5415,0.1.210 引入)**:#5293 的修复把每次用户中断后的增长参照点直接抬到当前读数,连「中断前已实测的真实增长」也一起清零;高频中断的客户端(实测一份日志 4.5 小时 85 次中断)永远攒不到增长台阶,30 万可压内容 idle 数小时,最后一次性压 33 万。现在吸收时把被清零的实测增长记入「欠账」(`nudge.absorbedDebt`)并累计吸收次数(`nudge.absorbCount`),两者都只在**第 2 次起**的中断后生效:单次中断字节级保持 #5293 的安静(不打扰用户刚改的主意——欠账再多也不当场弹提醒);第二次中断才证明会话是「高频中断型」而非「改主意」,欠账从此与新增量一起计入增长台阶,恢复 209 时代的节奏。另加「大积压排水」兑底:欠账>0 且可压量≥2×阈值且当前无未消化提醒时不再等增长台阶(与 #194 同哲学);无欠账/单次中断的会话决策字节不变。压缩成功时欠账与计数随其它基线一起清零;idle 日志同步显示 `growth X + absorbed Y across N aborts < floor Z`(单次中断显示 `held (single abort, #5293)`)便于排查。tests:kernel/tests/issue5415-absorb-debt-nudge.test.ts(欠账付台阶/单次中断钉死不弹/排水兑底/忽略提醒不重发/无欠账字节不变/压缩后清账+清计数);tests/issue5415-absorb-debt.test.ts(吸收记账算术、欠账+计数、累次累计、参照点已越过 anchor 不记账也计数、陈旧 flag 丢弃)。 - **长会话每回合把 Node 主线程卡死十几秒、网页面板冻结(#5337)**:每回合的压缩筛查把历史切成若干可折叠单元,逐个跑「能否折叠」判定——而每次判定都从零重扫全量消息数组(id 查找无索引,五处线性 `.find()`/`includes`),还重复重算全套整史统计(受保护/过期工具调用集合、工具配对表、末尾锚点、完整性预扫)。总工作量≈单元数×全史长度=历史规模平方级,且同步跑在主线程:2.3万条核心消息实测单回合约17秒(事件循环被占死,请求排队;#5304 同签名报告)。修复=把这批 O(n) 推导提升为每轮筛查**一次**构建(`buildProjectionShared`)、共享给同一轮所有单元;id→序号一次性建表替五处线性查找;「已丢弃」成员判断从 includes-in-some 改为 Set。判定逻辑本身一字未动(#3236 单一判定纪律),只是输入到达判定的方式变了。同机同语料实测:2万条消息完整推荐链 9028ms→约572ms(≈16×),回落到线性量级。单次调用路径(apply/report)保持原有内联计算,输出逐字节不变。kernel v0.0.175。tests/issue5337-shared-screening.test.ts(共享/非共享投影逐字段等价+降级边界一致+n=20000 耗时上界钉);既有 issue3406 曲线钉与 #3236 一致性回归不变。 From 62ff052b6f7a2b8a25b87cae2e3e70627748e4bb Mon Sep 17 00:00:00 2001 From: mz Date: Sun, 1 Nov 2026 21:16:09 +0300 Subject: [PATCH 4/9] fix(diagnostics): measure the final forwarded request body --- CHANGELOG.md | 2 +- src/server/handle.ts | 7 +++---- src/server/relay.ts | 9 +++++++-- src/server/request-cost.ts | 6 +++--- tests/issue903-request-cost-log.test.ts | 25 ++++++++++--------------- 5 files changed, 24 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 62cede7c6a..294de45152 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ### Fixed -* **「慢会话」告警把「消息条数多」说成了「事件循环已被卡死」(#4528 遗留文案)**:这条告警的触发条件只有一条——本轮入站消息数超过 1000,然后正文却断言「每回合的 O(messages) 同步扫描正在饿死事件循环」,并建议用户「关闭/重启会话或开启压缩」。实测里大历史请求常常几十毫秒就完成本地准备(热缓存、请求体小),此时告警内容与同一行 `local=` 毫秒数直接自相矛盾,还会把用户引向「重启会话」这种既不必要又有损的操作。现在告警改名为 `large-history`,只陈述已实测的事实(消息条数、入站字节、本地耗时毫秒),明说「本行只报条数,不把它归因为慢请求,实际耗时请看客户端」;触发条件、warn 级别、`local=`/`inbound=`/`outbound=`/`view=` 字段与 #5408 的 `noteRequest` 观测点全部逐字保留。历史长停顿的真实成因仍未证实,因此没有新增任何延迟阈值、重试、配置项或对会话历史的改写。该单元从 `src/server.ts` 抽出到 `src/server/request-cost.ts`(连同它的阈值常量与 `formatBytes`),`src/server/handle.ts` 的 `logRequestCost` 导入改指新模块(10 处调用本身未动),`src/server.ts` 不再转出。 +* **「慢会话」告警把「消息条数多」说成了「事件循环已被卡死」(#4528 遗留文案)**:这条告警的触发条件只有一条——本轮入站消息数超过 1000,然后正文却断言「每回合的 O(messages) 同步扫描正在饿死事件循环」,并建议用户「关闭/重启会话或开启压缩」。实测里大历史请求常常几十毫秒就完成本地准备(热缓存、请求体小),此时告警内容与同一行 `local=` 毫秒数直接自相矛盾,还会把用户引向「重启会话」这种既不必要又有损的操作。现在告警改名为 `large-history`,只陈述已实测的事实(消息条数、入站字节、本地耗时毫秒),明说「本行只报条数,不把它归因为慢请求,实际耗时请看客户端」;触发条件、warn 级别、`local=`/`inbound=`/`outbound=`/`view=` 字段与 #5408 的 `noteRequest` 观测点全部逐字保留。历史长停顿的真实成因仍未证实,因此没有新增任何延迟阈值、重试、配置项或对会话历史的改写。该单元从 `src/server.ts` 抽出到 `src/server/request-cost.ts`(连同它的阈值常量与 `formatBytes`),`src/server/handle.ts` 的 `logRequestCost` 导入改指新模块,`src/server.ts` 不再转出。正常上游路径现在在兼容重写与 V1 封装后的最终 fetch body 上计量 `outbound`,不再拿转发前字符串代替实际上游字节;本地 forge、side 与 fail-fast 的既有记账边界不变。 * **#5412: 异步存储后端的「压缩冻结会话」现在也能自愈。** #5408 事故(一个会话连续 17 次摘要漂移后被保护机制冻住、8 小时无人知晓)的主修复 #5409 只教了同步存储路径(SQLite 直写)自救:漂移挂起期间保存会自动升级为全量重导入并清掉索引里的残留行。异步路径(opt-in 网络后端,如 mysql)那份孪生代码没搬——同样被冻住的会话在那上面永远救不回来,每次保存都重写不完整视图、下轮采样继续漂移。本 PR 在 `src/storage/shadow.ts` 的异步入队路径补上同一行对齐(`completeLog = hasRawTail || driftedSessions.has(id)`),行为与同步路径逐字一致;回归测试同时钉住正门(latch 挂起的折叠保存清掉外来行)与负门(未挂起保存不清扫)。 - **中断频繁的会话压缩提醒几乎不触发,积到几十万 token 才压一次(#5415,0.1.210 引入)**:#5293 的修复把每次用户中断后的增长参照点直接抬到当前读数,连「中断前已实测的真实增长」也一起清零;高频中断的客户端(实测一份日志 4.5 小时 85 次中断)永远攒不到增长台阶,30 万可压内容 idle 数小时,最后一次性压 33 万。现在吸收时把被清零的实测增长记入「欠账」(`nudge.absorbedDebt`)并累计吸收次数(`nudge.absorbCount`),两者都只在**第 2 次起**的中断后生效:单次中断字节级保持 #5293 的安静(不打扰用户刚改的主意——欠账再多也不当场弹提醒);第二次中断才证明会话是「高频中断型」而非「改主意」,欠账从此与新增量一起计入增长台阶,恢复 209 时代的节奏。另加「大积压排水」兑底:欠账>0 且可压量≥2×阈值且当前无未消化提醒时不再等增长台阶(与 #194 同哲学);无欠账/单次中断的会话决策字节不变。压缩成功时欠账与计数随其它基线一起清零;idle 日志同步显示 `growth X + absorbed Y across N aborts < floor Z`(单次中断显示 `held (single abort, #5293)`)便于排查。tests:kernel/tests/issue5415-absorb-debt-nudge.test.ts(欠账付台阶/单次中断钉死不弹/排水兑底/忽略提醒不重发/无欠账字节不变/压缩后清账+清计数);tests/issue5415-absorb-debt.test.ts(吸收记账算术、欠账+计数、累次累计、参照点已越过 anchor 不记账也计数、陈旧 flag 丢弃)。 - **长会话每回合把 Node 主线程卡死十几秒、网页面板冻结(#5337)**:每回合的压缩筛查把历史切成若干可折叠单元,逐个跑「能否折叠」判定——而每次判定都从零重扫全量消息数组(id 查找无索引,五处线性 `.find()`/`includes`),还重复重算全套整史统计(受保护/过期工具调用集合、工具配对表、末尾锚点、完整性预扫)。总工作量≈单元数×全史长度=历史规模平方级,且同步跑在主线程:2.3万条核心消息实测单回合约17秒(事件循环被占死,请求排队;#5304 同签名报告)。修复=把这批 O(n) 推导提升为每轮筛查**一次**构建(`buildProjectionShared`)、共享给同一轮所有单元;id→序号一次性建表替五处线性查找;「已丢弃」成员判断从 includes-in-some 改为 Set。判定逻辑本身一字未动(#3236 单一判定纪律),只是输入到达判定的方式变了。同机同语料实测:2万条消息完整推荐链 9028ms→约572ms(≈16×),回落到线性量级。单次调用路径(apply/report)保持原有内联计算,输出逐字节不变。kernel v0.0.175。tests/issue5337-shared-screening.test.ts(共享/非共享投影逐字段等价+降级边界一致+n=20000 耗时上界钉);既有 issue3406 曲线钉与 #3236 一致性回归不变。 diff --git a/src/server/handle.ts b/src/server/handle.ts index 710f65ac00..4b782cafb7 100644 --- a/src/server/handle.ts +++ b/src/server/handle.ts @@ -2605,7 +2605,6 @@ async function handleRequest( if (mode === "intercept" && gatePre) prepared = await runPrepare(); if (prepared) memProbe(opts.debug, log, session.id, "prepared", typeof prepared.body === "string" ? Buffer.byteLength(prepared.body) : prepared.body.byteLength); if (prepared?.codexForge) { - logRequestCost(log, session.id, inboundMsgs, inboundBytes, reqT0, prepared.body); return { body: prepared.body, prepared }; } // runPrepare may have mutated parsed before failing to forge. @@ -2617,7 +2616,6 @@ async function handleRequest( const forwardBody = normalized ? Buffer.from(JSON.stringify({ ...original, input: items })) : bodyBuffer; const why = mode !== "intercept" ? "BILI_CODEX_COMPACT=pass" : !gatePre ? "gate preconditions not met" : "transform/forge failed"; log("info", `[${session.id}] codex compaction_trigger request not intercepted (${why}) — forwarding ${normalized ? `with bili summaries normalized (replaced=${replaced}, dropped=${dropped})` : "verbatim"} (no preflight, no rebuild, no window clamp)`); - logRequestCost(log, session.id, inboundMsgs, inboundBytes, reqT0, forwardBody); return { body: forwardBody, prepared: null }; } // #3450: ONE wall per request — on the refold path the @@ -2805,7 +2803,6 @@ async function handleRequest( // forward — anything that queued during the turn is served // here rather than behind the response stream setup. await yieldToEventLoop(); - logRequestCost(log, session.id, inboundMsgs, inboundBytes, reqT0, prepared!.body); return { body: prepared!.body, prepared: prepared! }; }; // #1884/#2090 (un-armed signed traffic): a request that already @@ -2891,7 +2888,9 @@ async function handleRequest( return next; } : undefined; - await forward(req, res, opts, pendingForward.body, pendingForward.prepared, core, reqConfig, log, route, instanceId, affinity, overflowRefold, v1Meta); + await forward(req, res, opts, pendingForward.body, pendingForward.prepared, core, reqConfig, log, route, instanceId, affinity, overflowRefold, v1Meta, (outbound) => { + logRequestCost(log, session.id, inboundMsgs, inboundBytes, reqT0, outbound); + }); // Remember for ALL modes (not just plugin): wire clients (dsh, // hermes, unplug'd pi) read the same panel via /__bili/plugin/status // and need the nudge/breakdown sections too; locked so a racing diff --git a/src/server/relay.ts b/src/server/relay.ts index 1b7642d5f3..126c853add 100644 --- a/src/server/relay.ts +++ b/src/server/relay.ts @@ -267,12 +267,14 @@ async function forwardCaptured( affinity?: string, overflowRefold?: (realWindow: number | undefined) => Promise<{ body: string | Buffer; prepared: Prepared | null } | null>, v1Meta?: Record, + onWireBody?: (body?: Buffer | string) => void, ): Promise { const forwardStartedAt = Date.now(); const reply = prepared && !prepared.sidePassthrough ? createHandoffReply(prepared.session, () => !res.destroyed) : undefined; // E2: a codex native-compaction request intercepted in prepare() carries a // forged success response — serve it without contacting upstream. if (prepared?.codexForge) { + onWireBody?.(); log("info", `[${prepared.session.id}] codex compact served locally (${prepared.codexForge.kind}); upstream not contacted`); if (!res.headersSent) res.writeHead(200, { "content-type": prepared.codexForge.contentType }); res.end(prepared.codexForge.body); @@ -659,11 +661,13 @@ async function forwardCaptured( } } } + const requestBody = req.method === "GET" || req.method === "HEAD" ? undefined : v1Wrap(wireBody); const init: Omit & { dispatcher?: object } = { method: req.method ?? "GET", headers, - body: req.method === "GET" || req.method === "HEAD" ? undefined : v1Wrap(wireBody), + body: requestBody, }; + onWireBody?.(requestBody); if (dispatcher) init.dispatcher = dispatcher; // #1843 L1: capture this round's image facts at the SEND chokepoint so the // turn's usage settle pairs with the bytes actually sent — streamed turns @@ -725,7 +729,8 @@ async function forwardCaptured( // previous request's body. Loop re-fetches re-note their own rebuilt bodies // (loop/core.ts fetchUpstream); side requests never settle usage and must // not clobber the slot. - if (prepared?.session && !prepared.sidePassthrough && req.method !== "GET" && req.method !== "HEAD") { + const mainSend = prepared !== null && prepared.session !== undefined && !prepared.sidePassthrough && req.method !== "GET" && req.method !== "HEAD"; + if (mainSend && prepared?.session) { const sentBody = typeof wireBody === "string" ? wireBody : wireBody.toString("utf8"); // #2078: ONE parse of the outbound body for the whole seam block — the // estimate's projection, its wire-overhead term, and the image reserve diff --git a/src/server/request-cost.ts b/src/server/request-cost.ts index 5d19945850..2e62ca1050 100644 --- a/src/server/request-cost.ts +++ b/src/server/request-cost.ts @@ -3,12 +3,12 @@ import { currentFetchTransport } from "../fetch-transport.js"; // #903: per-request local-cost line, logged at every point where bili finishes // its own processing and hands the request off (upstream forward, forged local -// response, or fail-fast error) — see handle(). outbound is the exact body value -// handed to forward() (string OR Buffer — Prepared.body is string|Buffer); wire +// response, or fail-fast error) — see handle(). outbound is the final body value +// handed to fetch (string OR Buffer) after relay and V1 transformations; wire // bytes are counted with Buffer.byteLength since undici sends UTF-8 bytes even // for string bodies (.length would count UTF-16 chars and undercount any // non-ASCII injection). Omitted on paths that fail fast without forwarding. -function formatBytes(n: number): string { +export function formatBytes(n: number): string { if (n < 1024) return `${n}B`; if (n < 1024 * 1024) return `${(n / 1024).toFixed(1)}KiB`; if (n < 1024 * 1024 * 1024) return `${(n / (1024 * 1024)).toFixed(1)}MiB`; diff --git a/tests/issue903-request-cost-log.test.ts b/tests/issue903-request-cost-log.test.ts index 5791b42060..fb63129da9 100644 --- a/tests/issue903-request-cost-log.test.ts +++ b/tests/issue903-request-cost-log.test.ts @@ -11,6 +11,7 @@ import type { ProxyOptions } from "../src/config.ts"; import { SessionStore, _setStoreForTest } from "../src/persist.ts"; import { _setForTest as setRegistryForTest } from "../src/registry.ts"; import { setLogCapture } from "../src/logger.ts"; +import { formatBytes } from "../src/server/request-cost.ts"; // #903: long sessions keep resending the full harness history to bili even // when the folded context stays small — the per-request cost line makes that @@ -36,21 +37,13 @@ function inboundBytesOf(line: string): number { return Math.round(v * mult); } -function outboundBytesOf(line: string): number { - const m = line.match(/outbound=(\d+(?:\.\d+)?)(B|KiB|MiB|GiB)/); - assert.ok(m, `no outbound field in: ${line}`); - const v = Number(m![1]); - const mult = m![2] === "B" ? 1 : m![2] === "KiB" ? 1024 : m![2] === "MiB" ? 1024 ** 2 : 1024 ** 3; - return Math.round(v * mult); -} - test("e2e: per-request cost line reports msg count, inbound wire size and local duration", async () => { - let upstreamSeen = 0; + const upstreamBodies: Buffer[] = []; const upstream = http.createServer((req, res) => { const chunks: Buffer[] = []; req.on("data", (c: Buffer) => chunks.push(c)); req.on("end", () => { - upstreamSeen = Buffer.concat(chunks).length; + upstreamBodies.push(Buffer.concat(chunks)); res.writeHead(200, { "content-type": "application/json" }); res.end(okJson()); }); @@ -79,7 +72,7 @@ test("e2e: per-request cost line reports msg count, inbound wire size and local chainContentDetection: false, autoUpdate: false, mitm: { enabled: false, domains: [] }, - compat: { roles: {} }, + compat: { roles: { assistant: "rewritten-assistant-role-with-a-deliberately-long-name" } }, streamErrorShape: "protocol", passthroughSource: null, autoRestartOnUpdate: false, @@ -114,13 +107,15 @@ test("e2e: per-request cost line reports msg count, inbound wire size and local const r1 = await fetch(url, { method: "POST", headers, body: first.raw }); assert.equal(r1.status, 200); await r1.text(); - const up1 = upstreamSeen; + assert.equal(upstreamBodies.length, 1); + const up1 = upstreamBodies[0]!.length; const second = mkBody(3); const r2 = await fetch(url, { method: "POST", headers, body: second.raw }); assert.equal(r2.status, 200); await r2.text(); - const up2 = upstreamSeen; + assert.equal(upstreamBodies.length, 2); + const up2 = upstreamBodies[1]!.length; // Blocked side request (#554 gate): tiny output budget + raw-history // estimate reaching the effective window -> 413 locally, upstream never @@ -138,12 +133,12 @@ test("e2e: per-request cost line reports msg count, inbound wire size and local const l1 = costLines[0]!; assert.match(l1, /^\[cost-sess\] request: 5 msgs, inbound=\d+(?:\.\d+)?(B|KiB|MiB|GiB), outbound=\d+(?:\.\d+)?(B|KiB|MiB|GiB), local=\d+ms$/); assert.ok(Math.abs(inboundBytesOf(l1) - Buffer.byteLength(first.raw)) <= 100, `inbound ${l1} vs sent ${Buffer.byteLength(first.raw)}`); - assert.ok(Math.abs(outboundBytesOf(l1) - up1) <= 100, `outbound ${l1} vs upstream-received ${up1}`); + assert.ok(l1.includes(`outbound=${formatBytes(up1)}`), `outbound ${l1} vs upstream-received ${up1}`); const l2 = costLines[1]!; assert.match(l2, /^\[cost-sess\] request: 7 msgs, inbound=\d+(?:\.\d+)?(B|KiB|MiB|GiB), outbound=\d+(?:\.\d+)?(B|KiB|MiB|GiB), local=\d+ms$/); assert.ok(Math.abs(inboundBytesOf(l2) - Buffer.byteLength(second.raw)) <= 100, `inbound ${l2} vs sent ${Buffer.byteLength(second.raw)}`); - assert.ok(Math.abs(outboundBytesOf(l2) - up2) <= 100, `outbound ${l2} vs upstream-received ${up2}`); + assert.ok(l2.includes(`outbound=${formatBytes(up2)}`), `outbound ${l2} vs upstream-received ${up2}`); assert.ok(inboundBytesOf(l2) > inboundBytesOf(l1), "growing history must show up as growing inbound"); const l3 = costLines[2]!; From 411e5e181bee324d22fe7c270e6a6c9160841875 Mon Sep 17 00:00:00 2001 From: ci-check Date: Mon, 2 Nov 2026 19:55:40 +0800 Subject: [PATCH 5/9] test: pin large-history threshold boundary and unknown-count case (#5481) Review nits on PR #5481: the msgs>1000 predicate is claimed preserved verbatim, so pin both sides of the boundary (1000 -> no warn, 1001 -> warn) plus the unknown-count passthrough lane (no warn, '? msgs'), which the existing 1500/500 pair left untested. --- tests/issue4528-event-loop-yield.test.ts | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/issue4528-event-loop-yield.test.ts b/tests/issue4528-event-loop-yield.test.ts index dce5dd46c1..d020437c2c 100644 --- a/tests/issue4528-event-loop-yield.test.ts +++ b/tests/issue4528-event-loop-yield.test.ts @@ -68,4 +68,26 @@ describe("#4528 logRequestCost large-history guard", () => { assert.equal(lines.filter(([l]) => l === "warn").length, 0); assert.ok(lines.some(([l, m]) => l === "info" && m.includes("sess-small") && m.includes("500 msgs"))); }); + + // #5481 review: pin both sides of the preserved msgs>1000 predicate (a > / >= drift would pass silently). + it("does not warn exactly at the threshold (1000 msgs)", () => { + const { lines, log } = capture(); + logRequestCost(log, "sess-edge", 1000, 1_000_000, performance.now() - 5); + assert.equal(lines.filter(([l]) => l === "warn").length, 0); + }); + + it("warns just past the threshold (1001 msgs)", () => { + const { lines, log } = capture(); + logRequestCost(log, "sess-edge", 1001, 1_000_000, performance.now() - 5); + const warns = lines.filter(([l]) => l === "warn"); + assert.equal(warns.length, 1); + assert.match(warns[0][1], /large-history: 1001 msgs/); + }); + + it("never warns when the message count is unknown (passthrough lane)", () => { + const { lines, log } = capture(); + logRequestCost(log, "sess-unknown", null, 1_000_000, performance.now() - 5); + assert.equal(lines.filter(([l]) => l === "warn").length, 0); + assert.ok(lines.some(([l, m]) => l === "info" && m.includes("? msgs"))); + }); }); From fe4ffdb0be6fc581cd2087d8e226f3b4b998d60a Mon Sep 17 00:00:00 2001 From: ci-check Date: Mon, 2 Nov 2026 19:55:40 +0800 Subject: [PATCH 6/9] docs: correct CHANGELOG local-forge accounting wording (#5481) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The entry said all local-forge/side/fail-fast accounting boundaries are unchanged, but the codex local-forged path now logs its cost line without an outbound= field (nothing is handed to fetch there) — disclosed in the PR thread but not in the changelog sentence. auto-mode forge, side and fail-fast lines are byte-identical. --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 294de45152..1467f11d51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ### Fixed -* **「慢会话」告警把「消息条数多」说成了「事件循环已被卡死」(#4528 遗留文案)**:这条告警的触发条件只有一条——本轮入站消息数超过 1000,然后正文却断言「每回合的 O(messages) 同步扫描正在饿死事件循环」,并建议用户「关闭/重启会话或开启压缩」。实测里大历史请求常常几十毫秒就完成本地准备(热缓存、请求体小),此时告警内容与同一行 `local=` 毫秒数直接自相矛盾,还会把用户引向「重启会话」这种既不必要又有损的操作。现在告警改名为 `large-history`,只陈述已实测的事实(消息条数、入站字节、本地耗时毫秒),明说「本行只报条数,不把它归因为慢请求,实际耗时请看客户端」;触发条件、warn 级别、`local=`/`inbound=`/`outbound=`/`view=` 字段与 #5408 的 `noteRequest` 观测点全部逐字保留。历史长停顿的真实成因仍未证实,因此没有新增任何延迟阈值、重试、配置项或对会话历史的改写。该单元从 `src/server.ts` 抽出到 `src/server/request-cost.ts`(连同它的阈值常量与 `formatBytes`),`src/server/handle.ts` 的 `logRequestCost` 导入改指新模块,`src/server.ts` 不再转出。正常上游路径现在在兼容重写与 V1 封装后的最终 fetch body 上计量 `outbound`,不再拿转发前字符串代替实际上游字节;本地 forge、side 与 fail-fast 的既有记账边界不变。 +* **「慢会话」告警把「消息条数多」说成了「事件循环已被卡死」(#4528 遗留文案)**:这条告警的触发条件只有一条——本轮入站消息数超过 1000,然后正文却断言「每回合的 O(messages) 同步扫描正在饿死事件循环」,并建议用户「关闭/重启会话或开启压缩」。实测里大历史请求常常几十毫秒就完成本地准备(热缓存、请求体小),此时告警内容与同一行 `local=` 毫秒数直接自相矛盾,还会把用户引向「重启会话」这种既不必要又有损的操作。现在告警改名为 `large-history`,只陈述已实测的事实(消息条数、入站字节、本地耗时毫秒),明说「本行只报条数,不把它归因为慢请求,实际耗时请看客户端」;触发条件、warn 级别、`local=`/`inbound=`/`outbound=`/`view=` 字段与 #5408 的 `noteRequest` 观测点全部逐字保留。历史长停顿的真实成因仍未证实,因此没有新增任何延迟阈值、重试、配置项或对会话历史的改写。该单元从 `src/server.ts` 抽出到 `src/server/request-cost.ts`(连同它的阈值常量与 `formatBytes`),`src/server/handle.ts` 的 `logRequestCost` 导入改指新模块,`src/server.ts` 不再转出。正常上游路径现在在兼容重写与 V1 封装后的最终 fetch body 上计量 `outbound`,不再拿转发前字符串代替实际上游字节;codex 本地 forge 路径仍记一行但不再携带 `outbound` 字段(该路径没有交给 fetch 的 body),auto-mode forge、side 与 fail-fast 的既有记账逐字不变。 * **#5412: 异步存储后端的「压缩冻结会话」现在也能自愈。** #5408 事故(一个会话连续 17 次摘要漂移后被保护机制冻住、8 小时无人知晓)的主修复 #5409 只教了同步存储路径(SQLite 直写)自救:漂移挂起期间保存会自动升级为全量重导入并清掉索引里的残留行。异步路径(opt-in 网络后端,如 mysql)那份孪生代码没搬——同样被冻住的会话在那上面永远救不回来,每次保存都重写不完整视图、下轮采样继续漂移。本 PR 在 `src/storage/shadow.ts` 的异步入队路径补上同一行对齐(`completeLog = hasRawTail || driftedSessions.has(id)`),行为与同步路径逐字一致;回归测试同时钉住正门(latch 挂起的折叠保存清掉外来行)与负门(未挂起保存不清扫)。 - **中断频繁的会话压缩提醒几乎不触发,积到几十万 token 才压一次(#5415,0.1.210 引入)**:#5293 的修复把每次用户中断后的增长参照点直接抬到当前读数,连「中断前已实测的真实增长」也一起清零;高频中断的客户端(实测一份日志 4.5 小时 85 次中断)永远攒不到增长台阶,30 万可压内容 idle 数小时,最后一次性压 33 万。现在吸收时把被清零的实测增长记入「欠账」(`nudge.absorbedDebt`)并累计吸收次数(`nudge.absorbCount`),两者都只在**第 2 次起**的中断后生效:单次中断字节级保持 #5293 的安静(不打扰用户刚改的主意——欠账再多也不当场弹提醒);第二次中断才证明会话是「高频中断型」而非「改主意」,欠账从此与新增量一起计入增长台阶,恢复 209 时代的节奏。另加「大积压排水」兑底:欠账>0 且可压量≥2×阈值且当前无未消化提醒时不再等增长台阶(与 #194 同哲学);无欠账/单次中断的会话决策字节不变。压缩成功时欠账与计数随其它基线一起清零;idle 日志同步显示 `growth X + absorbed Y across N aborts < floor Z`(单次中断显示 `held (single abort, #5293)`)便于排查。tests:kernel/tests/issue5415-absorb-debt-nudge.test.ts(欠账付台阶/单次中断钉死不弹/排水兑底/忽略提醒不重发/无欠账字节不变/压缩后清账+清计数);tests/issue5415-absorb-debt.test.ts(吸收记账算术、欠账+计数、累次累计、参照点已越过 anchor 不记账也计数、陈旧 flag 丢弃)。 - **长会话每回合把 Node 主线程卡死十几秒、网页面板冻结(#5337)**:每回合的压缩筛查把历史切成若干可折叠单元,逐个跑「能否折叠」判定——而每次判定都从零重扫全量消息数组(id 查找无索引,五处线性 `.find()`/`includes`),还重复重算全套整史统计(受保护/过期工具调用集合、工具配对表、末尾锚点、完整性预扫)。总工作量≈单元数×全史长度=历史规模平方级,且同步跑在主线程:2.3万条核心消息实测单回合约17秒(事件循环被占死,请求排队;#5304 同签名报告)。修复=把这批 O(n) 推导提升为每轮筛查**一次**构建(`buildProjectionShared`)、共享给同一轮所有单元;id→序号一次性建表替五处线性查找;「已丢弃」成员判断从 includes-in-some 改为 Set。判定逻辑本身一字未动(#3236 单一判定纪律),只是输入到达判定的方式变了。同机同语料实测:2万条消息完整推荐链 9028ms→约572ms(≈16×),回落到线性量级。单次调用路径(apply/report)保持原有内联计算,输出逐字节不变。kernel v0.0.175。tests/issue5337-shared-screening.test.ts(共享/非共享投影逐字段等价+降级边界一致+n=20000 耗时上界钉);既有 issue3406 曲线钉与 #3236 一致性回归不变。 From 64b1bac3377c785f42af73419505eaeb79d79368 Mon Sep 17 00:00:00 2001 From: ci-check Date: Fri, 6 Nov 2026 19:29:56 +0800 Subject: [PATCH 7/9] docs: re-apply local-forge accounting wording lost in the master-merge (#5481) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The merge of origin/master into the branch (4954a1226) resolved the CHANGELOG conflict with the pre-review wording of the #5481 entry, silently dropping the correction from fe4ffdb0b: the codex local-forge path logs its cost line WITHOUT an outbound field now (nothing is handed to fetch there), so '本地 forge ... 既有记账边界不变' was inaccurate for that path. Re-applies the corrected sentence verbatim. --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f601fecab..dddbb05a8e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag - **The full-resend watermark fastpath (#5403/#5418) is now ON by default (GA)** after its §7.4 human review: plugin-lane chat-completions turns whose prefix is byte-stable skip the full history re-parse/re-pin on hit (the #5403 OOM producer), any doubt still falls back to the untouched full-parse path per turn. Opt out per install with `rollout.features.pluginFullResendWatermark: false` (or a percent) or `BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK=0`. ### Fixed -* **「慢会话」告警把「消息条数多」说成了「事件循环已被卡死」(#4528 遗留文案)**:这条告警的触发条件只有一条——本轮入站消息数超过 1000,然后正文却断言「每回合的 O(messages) 同步扫描正在饿死事件循环」,并建议用户「关闭/重启会话或开启压缩」。实测里大历史请求常常几十毫秒就完成本地准备(热缓存、请求体小),此时告警内容与同一行 `local=` 毫秒数直接自相矛盾,还会把用户引向「重启会话」这种既不必要又有损的操作。现在告警改名为 `large-history`,只陈述已实测的事实(消息条数、入站字节、本地耗时毫秒),明说「本行只报条数,不把它归因为慢请求,实际耗时请看客户端」;触发条件、warn 级别、`local=`/`inbound=`/`outbound=`/`view=` 字段与 #5408 的 `noteRequest` 观测点全部逐字保留。历史长停顿的真实成因仍未证实,因此没有新增任何延迟阈值、重试、配置项或对会话历史的改写。该单元从 `src/server.ts` 抽出到 `src/server/request-cost.ts`(连同它的阈值常量与 `formatBytes`),`src/server/handle.ts` 的 `logRequestCost` 导入改指新模块,`src/server.ts` 不再转出。正常上游路径现在在兼容重写与 V1 封装后的最终 fetch body 上计量 `outbound`,不再拿转发前字符串代替实际上游字节;本地 forge、side 与 fail-fast 的既有记账边界不变。 +* **「慢会话」告警把「消息条数多」说成了「事件循环已被卡死」(#4528 遗留文案)**:这条告警的触发条件只有一条——本轮入站消息数超过 1000,然后正文却断言「每回合的 O(messages) 同步扫描正在饿死事件循环」,并建议用户「关闭/重启会话或开启压缩」。实测里大历史请求常常几十毫秒就完成本地准备(热缓存、请求体小),此时告警内容与同一行 `local=` 毫秒数直接自相矛盾,还会把用户引向「重启会话」这种既不必要又有损的操作。现在告警改名为 `large-history`,只陈述已实测的事实(消息条数、入站字节、本地耗时毫秒),明说「本行只报条数,不把它归因为慢请求,实际耗时请看客户端」;触发条件、warn 级别、`local=`/`inbound=`/`outbound=`/`view=` 字段与 #5408 的 `noteRequest` 观测点全部逐字保留。历史长停顿的真实成因仍未证实,因此没有新增任何延迟阈值、重试、配置项或对会话历史的改写。该单元从 `src/server.ts` 抽出到 `src/server/request-cost.ts`(连同它的阈值常量与 `formatBytes`),`src/server/handle.ts` 的 `logRequestCost` 导入改指新模块,`src/server.ts` 不再转出。正常上游路径现在在兼容重写与 V1 封装后的最终 fetch body 上计量 `outbound`,不再拿转发前字符串代替实际上游字节;codex 本地 forge 路径仍记一行但不再携带 `outbound` 字段(该路径没有交给 fetch 的 body),auto-mode forge、side 与 fail-fast 的既有记账逐字不变。 - **KeyRing model-local 429s no longer throttle other models on the same credential.** Bili gives an independent origin+credential+model hold only to HTTP 429 responses whose structured KeyRing or Bili marker has an exact request-model match; `keyring_all_limited`, direct OpenCode monthly quotas, malformed bodies, and mismatches remain origin+credential holds. Large passthrough bodies use a top-level model scan, while Gemini/Vertex/v1 and side-lane paths carry their actual wire model. Account/model probe slots are request-owned and canceled sends release only their token; concurrent provisional 429 observations settle independently. Continuation retries reserve bounded classification capacity per origin+credential before egress; saturation skips only the optional refetch, without a global gate or discarding account-wide evidence. Continuation reads retain the upstream idle/abort policy. Side-lane SSE retains its stream mode so render-tag stripping still runs. Regressions: `tests/issue4204-throttle-governor.test.ts`, `tests/issue4743-degenerate-refetch-429.test.ts`, `tests/proxy-noinject-tag-strip.test.ts`, and `tests/upstream-error-log.test.ts`. - **External-summary mode no longer reports protected or already-folded ranges as failed compression.** An unchanged kernel preview now returns the normal idempotent no-op or kernel-error receipt without a paid summarizer call; validated internal probe summaries prevent an irrelevant submitted-hint length from blocking that receipt. Probe-created blocks with no recoverable content still refuse rather than folding an unreviewed hint. Regression: `tests/external-summary-runtime.test.ts` (24/24). From bef87a7c13db91ffe975a8bc46e167f0215a246d Mon Sep 17 00:00:00 2001 From: ci-check Date: Mon, 9 Nov 2026 18:43:05 +0800 Subject: [PATCH 8/9] =?UTF-8?q?test:=20rerun=20ci=20=E2=80=94=20#5631=20re?= =?UTF-8?q?start-survival=20dsh-lane=20startup=20flake=20(no=20code=20chan?= =?UTF-8?q?ge)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI job 129092915572 (test ubuntu-24 shard 1): 'proxy exited before becoming healthy (dsh-main-4, code=1)' in tests/issue5631-restart-survival.test.ts — the 4th proxy spawn crashed at boot; pi lane and all other 3553 tests passed. Not reproducible on this head: 3 consecutive full local runs green (d2ad8bf484). PR diff is diagnostics-only (request-cost log move), untouched by the failing path. Empty commit to re-trigger CI. From 5f09d75e8a8b5f368f62ffd37ce022b1362f8516 Mon Sep 17 00:00:00 2001 From: awork Date: Tue, 10 Nov 2026 02:49:46 +0800 Subject: [PATCH 9/9] docs: point stale slow-session constant comment at its new home (#5481) --- src/web/sessions-data.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/web/sessions-data.ts b/src/web/sessions-data.ts index 1cf04f9117..aec78bc19a 100644 --- a/src/web/sessions-data.ts +++ b/src/web/sessions-data.ts @@ -38,7 +38,7 @@ import * as path from "node:path"; // the model, not the host's own history — so per-step cost climbs with it. On // plugin lanes (~2 host messages per request, dsh-measured) that becomes // visible around 10k messages ≈ 5k requests. The server log warns much earlier -// (SLOW_SESSION_MSG_WARN = 1000 messages, src/server.ts); this display gate is +// (SLOW_SESSION_MS_WARN = 1000ms local-processing budget, src/server/request-cost.ts); this display gate is // deliberately higher — the badge should mean "worth starting a new session", // not "notable". const SLOW_SESSION_REQUESTS_WARN = 5000;