Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed
| `compress.emergencyThresholdPercent` | number \| % | "95%" | — | Emergency truncation of oversized tool outputs when history passes this share of the window (must be >= maxContextLimit). |
| `compress.outputHeadroomMaxPct` | number \| % | 0.25 | — | Cap on the share of the window reserved for output via max_tokens. |
| `compress.nudgeGrowthTokens` | number | 50000 (kernel flat cadence) | — | Growth gate: nudges fire only when a foldable range exceeds baseline growth by this many tokens (flat by design, independent of window size). |
| `compress.streamSummary` | boolean | false (unset) | — | Force preflight summarization to run as a streaming (SSE) call from the first attempt. Needed when the upstream sits behind a gateway that times out long non-streaming completions (e.g. Cloudflare HTTP 524): the error-driven self-learn only sees 400 "stream required" rejections and never arms on gateway timeouts. |
| `compress.preserveRecentMessages` | number | kernel ≈5 | — | The most recent messages stay soft-protected from folds. |
| `compress.preserveRecentTokens` | number | kernel ≈5000 | — | The most recent tokens stay soft-protected from folds. |
| `compress.minCompressRangeChars` | number (deprecated alias: minCompressRange) | kernel ≈5000 | — | Smallest foldable range, in characters; shorter ranges never fold. |
Expand Down
1 change: 1 addition & 0 deletions CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@
| `compress.emergencyThresholdPercent` | number \| % | "95%" | — | 历史超过窗口该占比时对超大工具输出做紧急截断(必须 >= maxContextLimit)。 |
| `compress.outputHeadroomMaxPct` | number \| % | 0.25 | — | max_tokens 输出预留占窗口的最大比例。 |
| `compress.nudgeGrowthTokens` | number | 50000 (kernel flat cadence) | — | 增长门槛:可折叠片段超出基线增长达到该 token 数才发提醒(按设计恒定,与窗口大小无关)。 |
| `compress.streamSummary` | boolean | false (unset) | — | 强制 preflight 摘要从首次尝试起就走流式(SSE)请求。适用于上游位于会掐断长非流式补全的网关之后(如 Cloudflare HTTP 524):错误驱动的自学习只认 400 "stream required",网关超时永远无法触发。 |
| `compress.preserveRecentMessages` | number | kernel ≈5 | — | 最近的消息软保护、免于折叠。 |
| `compress.preserveRecentTokens` | number | kernel ≈5000 | — | 最近的 token 软保护、免于折叠。 |
| `compress.minCompressRangeChars` | number (deprecated alias: minCompressRange) | kernel ≈5000 | — | 可折叠片段的最小字符数;更短的永不折叠。 |
Expand Down
3 changes: 2 additions & 1 deletion src/compress-settings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -90,9 +90,10 @@ export function mergeCompress(
imageCompression: imageCompressionLevels.length > 0 ? Object.assign({}, ...imageCompressionLevels) : undefined,
rules: pick("rules"),

stripImages: pick("stripImages"),
stripImages: pick("stripImages"),
stripImagesKeepRecent: pick("stripImagesKeepRecent"),
visibilityMarkers: pick("visibilityMarkers"),
streamSummary: pick("streamSummary"),
// `reasoning` is a third nested-object field merged sub-field-wise
// exactly like `absorb`/`prompts`: a model-level `threshold` must not
// discard a provider-level `drop: false`.
Expand Down
20 changes: 19 additions & 1 deletion src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,20 @@ export type CompressSettings = {
* rebuilt history. `false` suppresses them entirely, for deployments where
* models imitate or narrate around the markers (#862). Default `true`. */
visibilityMarkers?: boolean;
/** Force preflight summarization calls to use streaming (SSE) instead of
* the default non-stream call (#2133). The error-driven self-learn flag
* (`session.metadata.preflightStreamSummary`) only fires on a 400 whose
* body says "stream … true" — a gateway that cuts long non-streaming
* completions with a timeout status (Cloudflare's HTTP 524 is the
* canonical case) never triggers it, so every summary round-trip fails
* and preflight spins without folding until an operator hand-edits the
* session file. Set this where such a gateway sits between bili and the
* origin: every preflight summary call then streams from the first
* attempt. Deepest level wins (global → provider → model), so it can be
* scoped to just the affected route; unset = legacy behavior (non-stream
* first, learn on 400). No-op on the Google wire (its summary calls are
* always streamed via :streamGenerateContent). */
streamSummary?: boolean;
/** Override the kernel's compression prompt text (compressPhilosophy /
* howToCompressRules / tier2DistillRules / tier3CondenseRules). All four
* fields are LOAD-BEARING: the kernel rules were tuned in production and
Expand Down Expand Up @@ -1475,7 +1489,7 @@ const COMPRESS_SETTING_FIELDS = new Set([
"visibilityMarkers", "rules", "injectTool", "injectNudge",
"acknowledgePromptsRisk", "absorb", "ccr", "search", "imageCompression",
"prompts", "promptPack", "reasoningGuard", "outputSteering", "priceProfile",
"reconcile",
"reconcile", "streamSummary",
]);

// Deduped per unique key set per process (same pattern as
Expand Down Expand Up @@ -1822,6 +1836,10 @@ export function parseCompressSettings(v: unknown): (CompressSettings & { injectT
if (typeof obj.visibilityMarkers !== "boolean") ok = false;
else out.visibilityMarkers = obj.visibilityMarkers;
}
if ("streamSummary" in obj) {
if (typeof obj.streamSummary !== "boolean") ok = false;
else out.streamSummary = obj.streamSummary;
}
if ("rules" in obj) {
if (typeof obj.rules !== "boolean") ok = false;
else out.rules = obj.rules;
Expand Down
7 changes: 6 additions & 1 deletion src/preflight.ts
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,8 @@ export interface PreflightDeps {
unknownBaseline?: boolean;
/** #1933 F1: origin of the upstream this request routes to. When it matches the route that learned the session's estimate-calibration factor k̂, every per-round text estimate is scaled by k̂ so gate, per-round exit and final fit judge one payload on the same scale as the trigger that started this invocation; absent or mismatched → raw estimates (legacy behavior). */
upstreamOrigin?: string;
/** #2133: compress.streamSummary resolved true for this request (three-level cascade). The self-learn flag only sees 400 "stream required" rejections, so gateways that time out long non-streaming completions (Cloudflare 524) can never self-heal — this forces SSE from the first attempt instead. */
forceStreamSummary?: boolean;
}

export type PreflightFailureKind = "upstream" | "exhausted" | "aborted";
Expand Down Expand Up @@ -684,7 +686,10 @@ async function summarizeRange(deps: PreflightDeps, content: string, startRef: st
// reject the max_output_tokens parameter. Each capability is learned at
// most once (guarded below), so the compatibility retries are bounded:
// at most one extra attempt per capability, in either rejection order.
let stream = deps.session.metadata.preflightStreamSummary === true;
// #2133: compress.streamSummary forces SSE from the first attempt — the
// learn path above only sees 400 "stream required" rejections, which a
// gateway timeout (524) never produces.
let stream = deps.session.metadata.preflightStreamSummary === true || deps.forceStreamSummary === true;
let includeMaxOutputTokens = !(deps.protocol === "responses" && hasLearnedNoMaxOutputTokens(deps));
for (;;) {
try {
Expand Down
6 changes: 6 additions & 0 deletions src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5871,6 +5871,11 @@ async function preflightCompressIfNeeded(
stopHold = beginPreflightHold(res, prepared, log);
}, preflightHoldGraceMs());
holdTimer.unref();
// #2133: compress.streamSummary (same three-level cascade as this request's
// own reqConfig resolution) forces SSE summarization — the error-driven
// learn path can't see gateway timeouts (524/503), so operators behind such
// gateways need a deterministic escape hatch.
const forceStreamSummary = resolveCompress(opts.routes, route?.rewrittenUrl, model, opts.compress).streamSummary === true;
let result: PreflightResult;
try {
result = await preflightCompress(
Expand All @@ -5892,6 +5897,7 @@ async function preflightCompressIfNeeded(
wireOverhead: overheadEstimate,
unknownBaseline,
upstreamOrigin: currentOrigin,
forceStreamSummary,
},
prepared.originalMessages,
);
Expand Down
Loading
Loading