Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag

### Fixes

- **Preflight fold budget raised 8 → 16: a payload far over the window can now be brought under it (#868, PR #869)**: one range folds per round and both per-invocation budgets were 8, so the loop gave up before finishing even when every fold succeeded. A live Gemini session (1.39M-token payload against a 1,048,576 window) folded 8 ranges, saved ~329k tokens, landed ~8k short of the window, and then fail-fast 502'd with `the compress budget was exhausted after 8 rounds` — a message that also read misleadingly, because it quoted the ORIGINAL context rather than the folded size. `MAX_PREFLIGHT_ROUNDS` and `MAX_SUMMARY_CALLS_PER_PREFLIGHT` are now 16 (`src/preflight.ts`, both exported so budget tests derive from them); the fail-fast 502 now reports the post-fold size ("~N tokens (down from ~M before preflight)") and the remaining compressible-range count (`result.rangesRemaining`) instead of only the original token count; the fold-depth coverage bound (best case ~9.6x-window overshoot, lower in practice) is documented beside the constants. `tests/preflight-round-budget.test.ts` pins the behaviour on the openai wire: an ~83k-token fixture against a 30k window is refused at 8 rounds and forwarded at 16, using more than 8 summary calls; a second case scales the same history past the 16-round bound and pins the honest 502 (post-fold size, remaining-range count, exactly 16 summary calls); the #574 budget-cap case in `tests/preflight-multi-range.test.ts` no longer re-pins the literal 8.
- **Plugin passthrough re-issues a turn whose only visible output was a stripped tag echo (#870, PR #871)**: the plugin-mode direct pipe strips model-emitted render tags, and when that echo was a turn's ENTIRE visible content the emptied chunk was suppressed, so the host received a turn with no text and no tool call and aborted it. Releasing the echo instead is not an option (the #463 contract keeps a pure-echo turn swallowed, and #674/#673 forbid leaking the echo as content), so the pipe now re-issues: on a clean terminal (`stop` / `end_turn` / `stop_sequence`) with zero visible text, zero tool calls and no error finish reason, `src/plugin.ts` refetches the same request with a continuation nudge (`src/degenerate-retry.ts`: `DEGENERATE_RETRY_NUDGE`, `injectContinuationNudge`, `makeContinuationRefetch`) and relays the retry inside the original framing — Anthropic keeps a single `message_start`/`message_stop` with the retry's block indices offset, and the OpenAI terminal is the `choices[].finish_reason` chunk rather than `[DONE]`, so the first attempt's terminal never reaches the client twice. One retry per turn; if the retry is also empty the original turn passes through with the existing `[degenerate-turn]` warning, and an unissuable retry presents the original terminal with valid JSON. `appendTrailingUserText` moved to `src/wire-body.ts` so the fake-completion hint and the retry share one body builder. Tests: `tests/plugin-empty-turn-retry.test.ts` (8, driving the pipe with a fake response and a fake refetch).
The same gate now covers the unclosed case: when a render tag is dropped and its interior
survives as the turn's only visible text, the host receives orphan markup with no tool call and
stalls, so that interior is counted as markup rather than as visible output and the turn is re-issued. When the retry degenerates too, the client now receives an in-band stream error rather
than a silent empty turn: an empty turn is indistinguishable from a model that produced nothing,
and the host reports neither, so the session reads as idle while it is dead.

- **Degenerate-turn auto-retry reachable on OpenAI/Anthropic wires (#821, PR #822)**: the #732 invisible single-retry for a completed turn containing ONLY a thinking block (zero visible text, zero tool calls) was unreachable on the OpenAI/Anthropic wires: its gate required `!forwardedAny`, but on those wires reasoning deltas are forwarded verbatim to the client, so at `done` time both `forwardedAny` and `sawThinking` are true — mutually exclusive for exactly the shape this retry exists for. Only the Responses wire (round>1 framing suppression) could ever fire it. Wild-caught via WorkBuddy + deepseek-v4.1-flash through an OpenAI-chat gateway: two round-1 thinking-only turns (~6–8 min of thinking, zero visible output) logged the #673 `[degenerate-turn]` warning but passed through as empty turns, and the host aborted with an "empty stream" error. The loop now tracks model-VISIBLE output separately (`forwardedVisible`, set only on text forwards; reasoning/meta keep setting `forwardedAny`), and the retry gate uses it — a forwarded thinking prefix is invisible to host turn semantics and its chunks carry no `finish_reason` (adapters strip it), so appending the retry's content to the same stream is safe. Unchanged: one-shot bound, mandatory `sawThinking` (a genuinely empty no-reasoning turn still passes through untouched), and the #413 truncation retry's strict `!forwardedAny`. Regression tests pin the openai-wire round-1 shape (retry fires, nudge present, thinking prefix precedes retried content in the same stream) and the no-reasoning control (no retry).
- **Cross-process startup-window race: concurrent launches no longer double-spawn proxies (#707)**: `ensureProxyRunning` only attached when it found a *healthy* instance at probe time — between the launcher's spawn and the child's first successful listen there was zero on-disk evidence that a bring-up was in flight, so two launches inside that window (~1s+) each spawned their own child: two live proxies over one sessions dir (double writers), last-writer-wins instance file, and the extra proxy living until ITS OWN spawner exited. The starter now claims `<state>/proxy-starting` with O_EXCL before spawning (carrying `{token, pid, host, port, startedAt}`); a concurrent caller that sees an active marker (owner pid alive AND younger than SPAWN_WAIT_MS + 30s) waits for the starter's instance with the same probe as attach (pid alive + `/__bili/health` + instanceId match), bounded by SPAWN_WAIT_MS, and attaches on success; marker disappearance (starter gave up / crashed) breaks the wait early so the caller spawns immediately. The starter clears the marker on every terminal path via a token-checked clear; a hard crash leaves a stale marker that the dead-owner/TTL checks render inert. The O_EXCL claim itself is the cross-process arbiter: the pre-spawn read is only a fast path, so a loser of the claim (read→claim race, or a stale/garbage marker file left by a crash mid-write) re-checks and either waits for the live holder or removes-and-reclaims the slot — it never spawns blindly, and an unreadable marker file self-heals on the next launch instead of permanently degrading coordination. A final probe after the wait deadline catches a starter that succeeded in its last moments — its success path clears the marker, which otherwise looks identical to a failure bail. Incompatible-config fall-through is preserved (a waited-for instance that isn't compatible with our launch options still spawns, matching today's incompatible-healthy-instance behavior); explicit `bili start` bypasses the marker entirely (it's a user action, not auto-bootstrap). In-process dedup (`singleFlight`, #706) is untouched — this is the cross-process half.
- **DeepSeek thinking mode 400 fixed: turns are atomic, reasoning-drop self-disables on strict-echo upstreams (#684)**: two paths could ship a rebuilt request whose assistant `tool_calls` turn lost its `reasoning_content` — (1) the proxy's recent-zone carve kept the last tool-call of a multi-call turn visible while its reasoning run folded into the block (kernel boundary logic only pulled the FIRST companion message after reasoning), and (2) #651's reasoning-drop stripped reasoning from a closed compress turn even on upstreams that document thinking-mode `reasoning_content` as mandatory round-trip. DeepSeek rejects both with 400 (`The \`reasoning_content\` in the thinking mode must be passed back to the API`) and the turn dies. Fixes: the kernel now computes atomic turn groups (reasoning run + its full assistant burst + every result paired by `toolCallId` — all fold together or none do; a range that would split a turn is withdrawn with a warning, or rejected with an explanatory error when nothing else remains), the forward boundary pull covers the whole burst, and the reasoning-drop never fires when the session is known strict-echo — statically for `deepseek` upstreams, or learned from the first 400 whose body mentions `reasoning_content` (persisted on the session, self-healing like #552). Three wire-level exit sentinels (`[reasoning-pair-violated]` on chat/completions and Responses, `[thinking-pair-violated]` on Anthropic) log loudly if any future path reintroduces a split turn.
Expand Down
28 changes: 25 additions & 3 deletions src/preflight.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import { peekRegistryOutputLimit } from "./registry.js";
// summarization calls sized to fit the smaller window, before the payload is
// forwarded.

const MAX_PREFLIGHT_ROUNDS = 8;
export const MAX_PREFLIGHT_ROUNDS = 16;
const CHUNK_FRACTION = 0.6;
const MIN_CHUNK_TOKENS = 2000;
const MIN_SUMMARY_CHARS = 50;
Expand All @@ -41,7 +41,20 @@ const MIN_SUMMARY_CHARS = 50;
const MAX_SUMMARY_OUTPUT_TOKENS = 32768;
// #574: bound on upstream summarization calls per invocation — the multi-range
// walk can otherwise spend a call per viable range in a block-dense history.
const MAX_SUMMARY_CALLS_PER_PREFLIGHT = 8;
export const MAX_SUMMARY_CALLS_PER_PREFLIGHT = 16;

// #869 review: coverage bound of the two depth budgets above. One round folds
// ONE range and each fold removes at most CHUNK_FRACTION x window tokens (the
// per-call chunk budget), so MAX_PREFLIGHT_ROUNDS rounds cover an overshoot of
// at most MAX_PREFLIGHT_ROUNDS x CHUNK_FRACTION x window ~= 9.6x the window —
// a payload of up to ~10.6x the window in the best case. Real coverage is
// lower: a range smaller than the chunk budget saves less, and the #726
// halving worklist can spend several calls on one range without completing a
// fold. Beyond the bound the loop still exits cleanly — the fail-fast reports
// the post-fold size and the remaining compressible-range count, so an
// operator sees exactly how far the budget ran out. 16 is tuned to the
// incident class behind #868 (a 1.39x-window payload); it is a fixed depth,
// not scaled to the overshoot — scaling it is a separate design question.

export type PreflightProtocol = "anthropic" | "openai" | "responses";

Expand Down Expand Up @@ -101,6 +114,12 @@ export interface PreflightResult {
* stale (e.g. a double-counted usage report, #300). The caller uses it
* to decide whether forwarding as-is is actually safe. */
payloadEstimate: number;
/** Compressible ranges still visible in the kernel's final view after the
* walk stopped — how much foldable headroom a deeper budget would find
* (0 when nothing foldable remains). Surfaced in the fail-fast message
* so an operator can see why the payload is still over the window
* (#869 review). */
rangesRemaining: number;
/** Whether the final payload fits the window, judged with the same
* measure the loop used: the optimistic token estimate for
* measured-baseline sessions (#300 — a stale HIGH baseline must not
Expand Down Expand Up @@ -696,7 +715,7 @@ function noEmergencyTruncate(config: Config): Config {
export async function preflightCompress(deps: PreflightDeps, messages: CoreMessage[]): Promise<PreflightResult> {
const limit = deps.config.modelContextLimit;
let target = Math.min(limit, deps.compressionTarget ?? limit);
const result: PreflightResult = { compressedRanges: 0, savedTokens: 0, payloadEstimate: estimateCoreMessages(messages) + (deps.imageFloor ?? 0) + (deps.wireOverhead ?? 0), fitsWindow: true };
const result: PreflightResult = { compressedRanges: 0, savedTokens: 0, payloadEstimate: estimateCoreMessages(messages) + (deps.imageFloor ?? 0) + (deps.wireOverhead ?? 0), rangesRemaining: 0, fitsWindow: true };
if (limit <= 0) return result;
const budget = Math.max(MIN_CHUNK_TOKENS, Math.floor(limit * CHUNK_FRACTION));
// applyCompression rejects ranges below config.compress.minCompressRange
Expand Down Expand Up @@ -732,6 +751,7 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa
let summaryCalls = 0;
let budgetHit = false;
let rangesTried = 0;
let rangesRemaining = 0;
for (let round = 0; round < MAX_PREFLIGHT_ROUNDS; round++) {
if (deps.signal?.aborted) {
failure = ABORTED_FAILURE;
Expand Down Expand Up @@ -769,6 +789,7 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa
if (startTokens < 0) startTokens = currentTokens;
if (currentTokens < target) break;
const ranges = viableRanges(turn.nudge?.compressibleRanges ?? []);
rangesRemaining = ranges.length;
if (ranges.length === 0) {
// #330: nothing foldable outside the soft-protected recent zone.
// Relax the soft zone (oldest-first within it) and retry — the hard
Expand Down Expand Up @@ -943,6 +964,7 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa
deps.session.stats.lastInputTokensSource = "estimate";
}
}
result.rangesRemaining = rangesRemaining;
result.savedTokens = Math.max(0, startTokens - currentTokens);
if (currentTokens >= limit) result.failure = failure;
result.fitsWindow = baselineKnown ? result.payloadEstimate < limit : finalUpper < limit;
Expand Down
14 changes: 11 additions & 3 deletions src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2876,12 +2876,20 @@ async function preflightCompressIfNeeded(
// session.stats.lastInputTokens, which can be stale — e.g. a
// double-counted usage report (#300) — and must not turn a fitting
// payload into a fail-fast false positive.
const failFast = (status: number, detail: string, retryable: boolean): PreflightFailFast => {
// #869 review: quote the POST-FOLD size once folding happened — reporting
// only the original tokenCount reads as "nothing happened" even when 16
// folds removed hundreds of thousands of tokens. foldedTokens/rangesLeft
// stay undefined when no fold ran, so the original phrasing holds there.
const failFast = (status: number, detail: string, retryable: boolean, foldedTokens?: number, rangesLeft?: number): PreflightFailFast => {
const imageNote = imageTokens >= limit
? ` Images alone account for ~${imageTokens} tokens (≥ window ${limit}); compression cannot remove them — shrink or remove the images, or raise the window.`
: "";
const sizeClause = foldedTokens !== undefined && foldedTokens < tokenCount
? `context ~${foldedTokens} tokens (down from ~${tokenCount} before preflight) exceeds the model window ${limit}`
: `context ~${tokenCount} tokens exceeds the model window ${limit}`;
const rangesClause = rangesLeft !== undefined ? `, with ${rangesLeft} compressible range(s) still visible` : "";
const message =
`context ~${tokenCount} tokens exceeds the model window ${limit} (model=${model}) ` +
`${sizeClause} (model=${model})${rangesClause} ` +
`and preflight compression could not bring it under: ${detail}.` +
imageNote +
` The over-window payload was NOT forwarded.`;
Expand Down Expand Up @@ -3007,7 +3015,7 @@ async function preflightCompressIfNeeded(
}
const status = f?.kind === "upstream" && f.status === 429 ? 503 : 502;
const retryable = f?.retryable === true || (f?.kind === "upstream" && f.status !== undefined && (f.status === 429 || f.status >= 500));
const ff = failFast(status, f?.detail ?? "the payload still exceeds the window after preflight compression", retryable);
const ff = failFast(status, f?.detail ?? "the payload still exceeds the window after preflight compression", retryable, result.compressedRanges > 0 ? session.stats.lastInputTokens : undefined, result.rangesRemaining);
const contentDeadEnd = f?.kind === "exhausted" || (f?.kind === "upstream" && f.status !== undefined && f.status >= 400 && f.status < 500 && !retryable);
if (contentDeadEnd && result.compressedRanges === 0) {
const cooldownMs = preflightDeadEndCooldownMs();
Expand Down
4 changes: 3 additions & 1 deletion tests/preflight-multi-range.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { startServer, type ProxyOptions } from "../src/server.ts";
import { SessionStore, _setStoreForTest } from "../src/persist.ts";
import { _setForTest as setRegistryForTest } from "../src/registry.ts";
import { listSessions } from "../src/session.ts";
import { MAX_SUMMARY_CALLS_PER_PREFLIGHT } from "../src/preflight.ts";

// #574/#569: preflight used to try exactly ONE range per round (the oldest)
// and declare GLOBAL exhaustion when that single range yielded no applied
Expand Down Expand Up @@ -198,9 +199,10 @@ test("#574 budget cap: many unusable ranges → exactly MAX_SUMMARY_CALLS_PER_PR
assert.equal(json.error?.code, "preflight_compress_failed");
assert.equal(json.error?.retryable, false);
assert.match(json.error?.message ?? "", /summarization budget/i, `the budget variant is reported (got: ${json.error?.message})`);
assert.match(json.error?.message ?? "", /compressible range\(s\) still visible/, `reports how many compressible ranges remain (got: ${json.error?.message})`);

const summaryCalls = calls.filter((c) => !c.stream);
assert.equal(summaryCalls.length, 8, `exactly MAX_SUMMARY_CALLS_PER_PREFLIGHT summary calls (got ${summaryCalls.length})`);
assert.equal(summaryCalls.length, MAX_SUMMARY_CALLS_PER_PREFLIGHT, `the call cap bounds the walk (got ${summaryCalls.length})`);
assert.equal(calls.filter((c) => c.stream).length, 0, "the over-window payload was NOT forwarded");

const s = listSessions()[0];
Expand Down
Loading
Loading