Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,7 @@ Environment variables take precedence over the config file. They are useful for
| `ACP_COMPRESS_NUDGE` | Set to `0` to disable nudge injection (same as `"compress.injectNudge": false`). |
| `ACP_MODEL_CONTEXT_LIMIT` | Override the context limit globally (absolute token count). |
| `BILI_IMAGE_TOKEN_CAP` | Cap the per-image token estimate used by the preflight size gate and output clamp (#488/#496). By default an inline `data:` image counts as `base64 length / 4` tokens with **no cap** — correct for byte-billing relays, but a large over-estimate for pixel-tile upstreams (official Anthropic/OpenAI), which bill each image at roughly 1.1K–1.6K tokens regardless of byte size. Set this to your upstream's per-image tile cost so the gate reflects real billing; unset = no cap (current default). |
| `BILI_PREFLIGHT_HOLD_MS` | Grace period (ms) before a long preflight compression starts holding the client with keep-alive bytes (default `30000`; see #568 / README "Preflight hold"). |
| `BILI_CONFIG_FILE` | Override the config file path (point at any JSON file). |
| `ACP_PORT` / `PORT` | Override the listen port. |
| `ACP_HOST` | Override the listen host. |
Expand Down
1 change: 1 addition & 0 deletions CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -357,6 +357,7 @@
| `ACP_COMPRESS_NUDGE` | 设为 `0` 禁用 nudge 注入(等同 `"compress.injectNudge": false`)。 |
| `ACP_MODEL_CONTEXT_LIMIT` | 全局覆盖上下文上限(绝对 token 数)。 |
| `BILI_IMAGE_TOKEN_CAP` | 预检尺寸门与输出钳制用的单图 token 估算上限(#488/#496)。默认内联 `data:` 图片按 `base64 长度 / 4` 计 token、**无上限** —— 对字节计费 relay 正确,但对像素 tile 计费的官方上游(Anthropic/OpenAI)会严重高估(后者无论字节多少,每图约计 1.1K–1.6K token)。设为你上游的单图 tile 成本,可让尺寸门反映真实计费;不设置 = 无上限(当前默认)。 |
| `BILI_PREFLIGHT_HOLD_MS` | 预压缩超过该宽限期(毫秒)后,代理提前提交响应并用保活字节挂住客户端(默认 `30000`;见 #568 / README「预压缩挂起」)。 |
| `BILI_CONFIG_FILE` | 覆盖配置文件路径(指向任意 JSON 文件)。 |
| `ACP_PORT` / `PORT` | 覆盖监听端口。 |
| `ACP_HOST` | 覆盖监听主机。 |
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ turn is a semantic mismatch the model tolerates (it is clearly marked
summaries (their tool call isn't in the agent's history and the agent's view
skips `acp_summary`) — but that needs the id match above, which doesn't occur.


## Which do I need?

Pick by your client:
Expand Down
1 change: 1 addition & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ bili --host 0.0.0.0 # 所有网卡(或直接用局域网 IP)
端点仍仅限本机访问。
- 启动时的 `[security]` 警告会提醒上述事项。


### 调试

三种方式打开详细日志(优先级:参数 > 环境变量 > 配置):
Expand Down
183 changes: 144 additions & 39 deletions src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ import { getSession, listSessions, type Session, initSessions, markDirty, flushA
import { COMPRESS_TOOL, ACP_TOOLS_ANTHROPIC, ACP_TOOLS_OPENAI, ACP_TOOLS_RESPONSES, ACP_READONLY_TOOLS_RESPONSES, COMPRESS_TOOL_NAME, buildCompressSystemPrompt, buildCompressHybridSystemPrompt, withStagedCompressGuidance } from "./compress-tool.js";
import { rewriteJsonResponse, type RewriteCtx } from "./stream.js";
import { applyRanges } from "./stream.js";
import { preflightCompress, estimateCoreMessages, estimateRawBodyTokens, estimateCoreMessagesUpper } from "./preflight.js";
import { preflightCompress, estimateCoreMessages, estimateRawBodyTokens, estimateCoreMessagesUpper, type PreflightResult } from "./preflight.js";
import { imageTokensInRawBody, imageTokensInParsedBody } from "./image-tokens.js";
import { renderUI, handleConfigGet, handleConfigPut } from "./web/index.js";
import { reapOrphanBlocks } from "./orphan-gc.js";
Expand All @@ -63,7 +63,7 @@ import { stripAcpPanelMessages, stripAcpPanelResponsesInput } from "./acp-panel.
import { rewriteOpenaiJsonResponse } from "./stream-openai.js";
import { rewriteResponsesJsonResponse } from "./stream-responses.js";
import { observeResponsesTerminalState } from "./stream-terminal.js";
import { emitStreamError } from "./stream-error.js";
import { emitPreflightError, emitStreamError } from "./stream-error.js";
import { affinityToken, clientConversationHeader, codexTurnIdentity, preferPromptCacheKeyIdentity, type ConversationIdentity } from "./session-id.js";
import { prefixAffinity, type AnonymousAffinity } from "./prefix-affinity.js";
import { flushPrefixAffinity, hydratePrefixAffinity, scheduleAffinityPersist } from "./affinity-persist.js";
Expand Down Expand Up @@ -1581,22 +1581,43 @@ async function handle(
// #301: the payload still overflows the window and
// preflight could not fix it — answer with a
// structured error instead of forwarding.
if (outcome.respond && !res.headersSent && !res.destroyed) {
// Retry-After on the 503 (rate-limited) path: gives
// well-behaved clients a backoff signal instead of
// hammering the rate-limited upstream (#301).
res.writeHead(outcome.status, {
"content-type": "application/json",
...(outcome.status === 503 ? { "retry-after": "30" } : {}),
});
res.end(JSON.stringify({
error: {
type: "server_error",
code: "preflight_compress_failed",
message: outcome.message,
retryable: outcome.retryable,
},
}));
if (outcome.respond && !res.destroyed) {
if (res.headersSent) {
// #568: the hold already committed 200 early — the status
// can no longer change, so deliver the same error in-band
// (protocol error event for SSE, identical JSON body for
// non-stream) instead of a status code we lost.
if (prepared!.stream) {
emitPreflightError(res, prepared!.protocol, { message: outcome.message, retryable: outcome.retryable }, (m) => log("warn", m));
} else {
try {
res.end(JSON.stringify({
error: {
type: "server_error",
code: "preflight_compress_failed",
message: outcome.message,
retryable: outcome.retryable,
},
}));
} catch { /* client gone */ }
}
} else {
// Retry-After on the 503 (rate-limited) path: gives
// well-behaved clients a backoff signal instead of
// hammering the rate-limited upstream (#301).
res.writeHead(outcome.status, {
"content-type": "application/json",
...(outcome.status === 503 ? { "retry-after": "30" } : {}),
});
res.end(JSON.stringify({
error: {
type: "server_error",
code: "preflight_compress_failed",
message: outcome.message,
retryable: outcome.retryable,
},
}));
}
}
return;
}
Expand Down Expand Up @@ -2753,6 +2774,65 @@ function isPreflightFailFast(outcome: Prepared | PreflightFailFast): outcome is
return "failFast" in outcome;
}

// #568: preflight compression sends ZERO bytes to the client while it runs, so
// undici's default headersTimeout (300s) kills any multi-round compression that
// crosses it — the proxy then aborts on the detected disconnect and the client
// retries into the same wall (5-minute death loop). Once the work outlives this
// grace period the response is committed early (200 + protocol framing) and the
// client is held with periodic keep-alive bytes until the real response exists.
// 30s protects every client whose header deadline exceeds 30s (undici's 300s
// default included); shorter preflights keep full status-code fidelity.
const PREFLIGHT_HOLD_GRACE_DEFAULT_MS = 30_000;
const PREFLIGHT_KEEPALIVE_MS = 15_000;

function preflightHoldGraceMs(): number {
const raw = process.env.BILI_PREFLIGHT_HOLD_MS;
if (!raw) return PREFLIGHT_HOLD_GRACE_DEFAULT_MS;
const v = Number(raw);
return Number.isFinite(v) && v >= 0 ? Math.floor(v) : PREFLIGHT_HOLD_GRACE_DEFAULT_MS;
}

/** #568: commit the response early so a long preflight cannot lose the client
* to its header timeout. Streaming clients get an SSE stream with keep-alive
* comment lines (`: bili-preflight` — a spec-mandated no-op for every SSE
* consumer, same pattern as OpenAI's SSE pings); non-streaming clients get
* chunked JSON padded with whitespace (valid JSON padding). Each byte resets
* undici's bodyTimeout (inactivity-based), holding the client for the whole
* compression. Returns a stop() ending the keep-alive, or undefined when
* nothing could be committed (headers already sent / socket gone — the
* existing res "close" abort then handles cancellation). */
function beginPreflightHold(res: http.ServerResponse, prepared: Prepared, log: (level: string, msg: string) => void): (() => void) | undefined {
if (res.headersSent || res.destroyed || res.writableEnded) return undefined;
const sid = prepared.session.id;
const keepAlive = prepared.stream ? ": bili-preflight\n\n" : " ";
try {
if (prepared.stream) {
res.writeHead(200, {
"content-type": "text/event-stream",
"cache-control": "no-cache",
"x-accel-buffering": "no",
"x-bili-preflight": "compressing",
});
} else {
res.writeHead(200, { "content-type": "application/json", "x-bili-preflight": "compressing" });
}
} catch {
return undefined;
}
log("info", `[${sid}] preflight still running after ${preflightHoldGraceMs()}ms grace — committed early ${prepared.stream ? "SSE" : "JSON"} headers + keep-alive to hold the client (#568)`);
try {
res.write(keepAlive);
} catch { /* client gone */ }
const iv = setInterval(() => {
try {
res.write(keepAlive);
} catch {
clearInterval(iv);
}
}, PREFLIGHT_KEEPALIVE_MS);
return () => clearInterval(iv);
}

async function preflightCompressIfNeeded(
prepared: Prepared,
runPrepare: () => Prepared,
Expand Down Expand Up @@ -2868,25 +2948,36 @@ async function preflightCompressIfNeeded(
if (!res.writableEnded) clientAbort.abort();
});
const started = Date.now();
const result = await preflightCompress(
{
core,
session,
config,
prompts: prepared.prompts ?? defaultPrompts,
protocol: prepared.protocol,
url: upstreamUrl,
headers,
model,
proxyUrl,
signal: clientAbort.signal,
log,
imageFloor: imageTokens,
wireOverhead: overheadEstimate,
unknownBaseline,
},
prepared.originalMessages,
);
let stopHold: (() => void) | undefined;
const holdTimer = setTimeout(() => {
stopHold = beginPreflightHold(res, prepared, log);
}, preflightHoldGraceMs());
holdTimer.unref();
let result: PreflightResult;
try {
result = await preflightCompress(
{
core,
session,
config,
prompts: prepared.prompts ?? defaultPrompts,
protocol: prepared.protocol,
url: upstreamUrl,
headers,
model,
proxyUrl,
signal: clientAbort.signal,
log,
imageFloor: imageTokens,
wireOverhead: overheadEstimate,
unknownBaseline,
},
prepared.originalMessages,
);
} finally {
clearTimeout(holdTimer);
stopHold?.();
}
// #330: decide forward/fail on the payload actually forwarded, not
// result.payloadEstimate — the preflight's relaxed-zone processTurn trims
// that estimate more than the normal-config prepare does, which can turn a
Expand Down Expand Up @@ -2939,7 +3030,7 @@ async function forward(
// forged success response — serve it without contacting upstream.
if (prepared?.codexForge) {
log("info", `[${prepared.session.id}] codex compact served locally (${prepared.codexForge.kind}); upstream not contacted`);
res.writeHead(200, { "content-type": prepared.codexForge.contentType });
if (!res.headersSent) res.writeHead(200, { "content-type": prepared.codexForge.contentType });
res.end(prepared.codexForge.body);
return;
}
Expand Down Expand Up @@ -3302,6 +3393,18 @@ async function forward(
if (bodyText.length > 600) snippet += " …";
if (!snippet) snippet = "(no body)";
loggerLog("warn", `[${errSid}] ← upstream ${upstream.status}${reqIdText}: ${snippet}`);
if (res.headersSent) {
// #568: the preflight hold already committed 200 early — the status can no
// longer change, so deliver the upstream failure in-band (protocol error
// event for streams; verbatim error body under 200 otherwise).
if (prepared?.stream) {
emitStreamError(res, prepared.protocol, `upstream HTTP ${upstream.status}: ${snippet}`, (m) => loggerLog("info", m));
} else {
try { res.end(errBody ?? undefined); } catch { /* client gone */ }
}
clearUpstreamTimer();
return;
}
const errHeaders: Record<string, string> = { ...respHeaders };
// Drop the upstream framing headers unconditionally: when errBody is
// present a fixed-length write replaces them, and when errBody is
Expand All @@ -3316,7 +3419,9 @@ async function forward(
return;
}
// 2xx path: now safe to commit the status + headers, then stream the body.
res.writeHead(upstream.status, respHeaders);
// When the #568 hold already committed an early 200, the upstream's own
// headers (x-request-id etc.) are dropped — informational only.
if (!res.headersSent) res.writeHead(upstream.status, respHeaders);
if (!upstream.body) {
res.end();
clearUpstreamTimer();
Expand Down
32 changes: 32 additions & 0 deletions src/stream-error.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,35 @@ export function emitStreamError(res: http.ServerResponse, protocol: Protocol, me
}
}
}

/**
* #568: deliver a preflight fail-fast error IN-BAND, after the proxy already
* committed `200` early to hold the client through a long compression (see
* beginPreflightHold in server.ts). The status line can no longer change, so
* the failure rides on a protocol-native error event:
* - openai: top-level `error` in a data frame, then `[DONE]`.
* - anthropic: `event: error` (the SDK's standard API-error channel).
* - responses: `event: error` (Responses SSE spec).
* Never throws.
*/
export function emitPreflightError(res: http.ServerResponse, protocol: Protocol, error: { message: string; retryable: boolean }, log?: (msg: string) => void): void {
const err = { type: "server_error", code: "preflight_compress_failed", message: error.message, retryable: error.retryable };
log?.(`[acp-proxy: preflight failed after early response commit — delivering in-band: ${error.message}]`);
try {
if (protocol === "openai") {
safeWrite(res, `data: ${JSON.stringify({ error: err })}\n\ndata: [DONE]\n\n`);
} else if (protocol === "responses") {
safeWrite(res, `event: error\ndata: ${JSON.stringify({ type: "error", code: err.code, message: err.message })}\n\n`);
} else {
safeWrite(res, `event: error\ndata: ${JSON.stringify({ type: "error", error: { type: "server_error", code: err.code, message: err.message } })}\n\n`);
}
} catch {
/* best-effort */
} finally {
try {
res.end();
} catch {
/* already closed */
}
}
}
Loading
Loading