Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,10 +225,10 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed
| Key | Type | Default | Env | Description |
|-----|------|---------|-----|-------------|
| `diagnostics` | object | {} (all defaults) | — | Local troubleshooting knobs (dumps, render/injection toggles, count-tokens passthrough). |
| `diagnostics.dumpBody` | boolean | false | ACP_DUMP_BODY | Dump full request/response bodies to the log for troubleshooting. |
| `diagnostics.dumpReq` | boolean | true | ACP_DUMP_REQ | Log outgoing upstream request metadata per call. |
| `diagnostics.dumpBody` | boolean | false | ACP_DUMP_BODY | Dump full request/response bodies to the log for troubleshooting. Dumps are PLAINTEXT and may contain credentials — scrub before sharing (#4482). |
| `diagnostics.dumpReq` | boolean | true | ACP_DUMP_REQ | Log outgoing upstream request metadata per call; with dumpBody on, also gates the plaintext body dumps (scrub before sharing, #4482). |
| `diagnostics.rawDumpDir` | string | <state dir>/raw | ACP_RAW_DUMP_DIR | Directory for raw wire dumps. |
| `diagnostics.dump4xx` | boolean | false | BILI_DUMP_4XX | Persist upstream 4xx responses to disk for post-mortem inspection. |
| `diagnostics.dump4xx` | boolean | false | BILI_DUMP_4XX | Persist upstream 4xx responses to disk for post-mortem inspection. Bodies are PLAINTEXT and may contain credentials — scrub before sharing (#4482). |
| `diagnostics.dump4xxMaxBytes` | number | 2097152 (floor 1024) | BILI_DUMP_4XX_MAX_BYTES | Size cap for a single 4xx dump file. |
| `diagnostics.maxTotalBytes` | number | unset (off) | BILI_DUMP_MAX_TOTAL_BYTES | Total-size cap for debug dump dirs (dumps/, raw/, SSE dump dir); oldest files deleted first when exceeded (floor 1 MiB). Dumps have no rotation by default and can grow tens of GB/day under active capture (#2412). |
| `diagnostics.maxAgeDays` | number | unset (off) | BILI_DUMP_MAX_AGE_DAYS | Age cap in days for debug dump dirs; files older than this are deleted (floor 1 h). Off by default. |
Expand Down
6 changes: 3 additions & 3 deletions CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,10 +225,10 @@
| Key | Type | Default | Env | Description |
|-----|------|---------|-----|-------------|
| `diagnostics` | object | {} (all defaults) | — | 本地排障旋钮(转储、渲染/注入开关、count-tokens 透传)。 |
| `diagnostics.dumpBody` | boolean | false | ACP_DUMP_BODY | 把完整请求/响应体转储到日志用于排障。 |
| `diagnostics.dumpReq` | boolean | true | ACP_DUMP_REQ | 逐次调用记录发往上游的请求元数据。 |
| `diagnostics.dumpBody` | boolean | false | ACP_DUMP_BODY | 把完整请求/响应体转储到日志用于排障。转储内容为明文,可能包含凭据——分享前必须清洗(#4482)。 |
| `diagnostics.dumpReq` | boolean | true | ACP_DUMP_REQ | 逐次调用记录发往上游的请求元数据;开启 dumpBody 时同时是明文请求体转储的闸门(分享前必须清洗,#4482)。 |
| `diagnostics.rawDumpDir` | string | <state dir>/raw | ACP_RAW_DUMP_DIR | 原始线路转储目录。 |
| `diagnostics.dump4xx` | boolean | false | BILI_DUMP_4XX | 把上游 4xx 响应落盘以便事后检查。 |
| `diagnostics.dump4xx` | boolean | false | BILI_DUMP_4XX | 把上游 4xx 响应落盘以便事后检查。转储内容为明文,可能包含凭据——分享前必须清洗(#4482)。 |
| `diagnostics.dump4xxMaxBytes` | number | 2097152 (floor 1024) | BILI_DUMP_4XX_MAX_BYTES | 单个 4xx 转储文件的大小上限。 |
| `diagnostics.maxTotalBytes` | number | unset (off) | BILI_DUMP_MAX_TOTAL_BYTES | 调试转储目录(dumps/、raw/、SSE dump 目录)的总大小上限;超限时按最旧优先删除(下限 1 MiB)。转储默认无任何轮转,持续捕获下每天可增长数十 GB(#2412)。 |
| `diagnostics.maxAgeDays` | number | unset (off) | BILI_DUMP_MAX_AGE_DAYS | 调试转储目录的保留天数上限;超过该天数的文件被删除(下限 1 小时)。默认关闭。 |
Expand Down
5 changes: 5 additions & 0 deletions scripts/live-files.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ export const LIVE_FILES = [
// #4434 (split-brain surface): spawns bili CLI children against a throwaway
// origin; own probe server is listen(0) so only the zone band matters
"tests/issue-split-brain-surface.test.ts",
// #4449 (af8e5dc80): both landed without registration and tripped the
// run-ci-tests guard on every PR branch; neither binds a port (shell
// child coordinated via ENTRY_REPORT_FILE; version-sync is execFileSync)
"tests/codex-mcp-shell.test.ts",
"tests/codex-plugin-version-sync.test.ts",
];

/** Band width per live file: the zone ladder drifts +1 per EADDRINUSE retry;
Expand Down
8 changes: 8 additions & 0 deletions src/acp-cache-diff.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
// its producer instead of surfacing as an unattributed TTL bucket (#1254).
import fs from "node:fs";
import path from "node:path";
import { DUMP_PLAINTEXT_BANNER } from "./dump-banner.js";
import { defaultLogFile } from "./paths.js";
import { safeSessionId } from "./server/headers.js";

Expand Down Expand Up @@ -130,6 +131,13 @@ function collectDumps(root: string): DumpMeta[] {
function parseDumpFile(file: string): DumpContent | null {
let buf: Buffer;
try { buf = fs.readFileSync(file); } catch { return null; }
// #4482: new-format dumps open with the plaintext-warning banner line; drop
// it explicitly (not by relying on the banner lacking ": ") so header
// parsing and body bytes stay identical to pre-banner files.
const nl = buf.indexOf(0x0a);
if (nl > 0 && buf.subarray(0, nl).toString("utf8").replace(/\r$/, "") === DUMP_PLAINTEXT_BANNER) {
buf = buf.subarray(nl + 1);
}
// #3331: the request-line + headers block can be terminated by \r\n\r\n as
// well as \n\n (dump files written on Windows / by CRLF-emitting clients);
// a bare \n\n probe misses the header block entirely on such files.
Expand Down
16 changes: 16 additions & 0 deletions src/dump-banner.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
// #4482: body dumps exist for byte-exact forensics (prefix diffs, rejection
// post-mortems), so their content is NEVER redacted — the full plaintext
// conversation body lands on disk and may carry real credentials (tool
// arguments, host-injected content). Every plaintext-body dump file therefore
// starts with this standing warning so a dump that gets shared (issue
// attachments, pastes) announces what it carries before the reader scrolls
// past it. Keep the text free of ": " — dump parsers treat head lines as
// headers and the first line is skipped as the request line.
export const DUMP_PLAINTEXT_BANNER = "PLAINTEXT DUMP — full conversation body, may contain credentials; scrub before sharing";

/** Drop one leading banner line if `text` starts with it (LF or CRLF). */
export function stripDumpBanner(text: string): string {
const nl = text.indexOf("\n");
if (nl > 0 && text.slice(0, nl).replace(/\r$/, "") === DUMP_PLAINTEXT_BANNER) return text.slice(nl + 1);
return text;
}
7 changes: 5 additions & 2 deletions src/error-dump.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import fs from "node:fs";
import path from "node:path";
import { DUMP_PLAINTEXT_BANNER } from "./dump-banner.js";
import { dumpsDir } from "./paths.js";
import { log as loggerLog } from "./logger.js";
import { dump4xxEnabled as knobDump4xxEnabled, dump4xxMaxBytes as knobDump4xxMaxBytes } from "./knobs.js";
Expand All @@ -10,6 +11,8 @@ import { gcDumpDirIfConfigured } from "./state-gc.js";
// standing body dump (ACP_DUMP_BODY=1) must be armed BEFORE the incident; this
// one fires on the failure itself. Still off by default — conversation bodies
// leak to disk (#276) — enable with BILI_DUMP_4XX=1 or diagnostics.dump4xx.
// #4482: like every plaintext-body dump, each file starts with the standing
// credentials warning banner (DUMP_PLAINTEXT_BANNER).

let failCount = 0;
let lastFailLog = 0;
Expand Down Expand Up @@ -48,7 +51,7 @@ export function dumpRejectedBody(status: number, sessionId: string, body: string
fs.mkdirSync(dir, { recursive: true });
const sid = sessionId.replace(/[^a-zA-Z0-9_-]/g, "_");
const out = path.join(dir, `err-${Date.now()}-${sid}-${status}.json`);
fs.writeFileSync(out, `${text}${marker}`);
fs.writeFileSync(out, `${DUMP_PLAINTEXT_BANNER}\n${text}${marker}`);
loggerLog("info", `[dump] upstream ${status} rejected body written to ${out}`);
gcDumpDirIfConfigured(dir);
return out;
Expand Down Expand Up @@ -80,7 +83,7 @@ export function dumpSummaryRejection(status: number, sessionId: string, request:
fs.mkdirSync(dir, { recursive: true });
const sid = sessionId.replace(/[^a-zA-Z0-9_-]/g, "_");
const out = path.join(dir, `summary-err-${Date.now()}-${sid}-${status}.json`);
fs.writeFileSync(out, JSON.stringify({ status, request: side(request), response: side(response) }, null, 2));
fs.writeFileSync(out, `${DUMP_PLAINTEXT_BANNER}\n${JSON.stringify({ status, request: side(request), response: side(response) }, null, 2)}`);
loggerLog("info", `[dump] upstream ${status} rejected summary exchange written to ${out}`);
gcDumpDirIfConfigured(dir);
return out;
Expand Down
3 changes: 2 additions & 1 deletion src/loop/core.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ import { formatUpstreamError, proxyDispatcher } from "../upstream-proxy.js";
import { warnCacheCollapse } from "../cache-warn.js";
import { dumpRejectedBody } from "../error-dump.js";
import { shedEgressIfOverBudget } from "../egress-shed.js";
import { DUMP_PLAINTEXT_BANNER } from "../dump-banner.js";
import { bodyDumpEnabled as knobBodyDumpEnabled, imageRescaleMaxDim } from "../knobs.js";
import { dumpsDir } from "../paths.js";
import { isStrictReasoningEcho, modelIdOf, normalizeStrictEchoBody } from "../strict-echo.js";
Expand Down Expand Up @@ -1488,7 +1489,7 @@ export async function* runCompressLoop(
const dumpDir = dumpsDir();
fs.mkdirSync(dumpDir, { recursive: true });
const sid = (ctx.session.id ?? "unknown").replace(/[^a-zA-Z0-9_-]/g, "_");
fs.writeFileSync(path.join(dumpDir, `req-${Date.now()}-${sid}-REREQUEST.json`), JSON.stringify(newBody, null, 2));
fs.writeFileSync(path.join(dumpDir, `req-${Date.now()}-${sid}-REREQUEST.json`), `${DUMP_PLAINTEXT_BANNER}\n${JSON.stringify(newBody, null, 2)}`);
} catch { /* best-effort */ }
}
let respResult: { response: Response; clearTimer: () => void };
Expand Down
3 changes: 2 additions & 1 deletion src/server/handle.ts
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ import { FORCE_TEXT_PROTOCOL } from "./inject.js";
import { prepareAnthropic } from "./prepare-anthropic.js";
import { prepareOpenai } from "./prepare-openai.js";
import { prepareGoogle, prepareGoogleCountTokens } from "./prepare-google.js";
import { DUMP_PLAINTEXT_BANNER } from "../dump-banner.js";
import { prepareResponses, prepareResponsesCompact } from "./prepare-responses.js";
import { bodyDumpEnabled, isModelDiscoveryPath, isV1InternalControlPlanePath, logDumpFailure, logUnrecognizedPath } from "./observability.js";
import { adminGateDenial, handleAdminRoute, isManagementPath } from "./admin.js";
Expand Down Expand Up @@ -2550,7 +2551,7 @@ async function handleRequest(
Object.fromEntries(Object.entries(req.headers).map(([k, v]) => [k, Array.isArray(v) ? v.join(",") : String(v)])),
);
const hdrText = Object.entries(hdrs).map(([k, v]) => `${k}: ${v}`).join("\n");
fs.writeFileSync(path.join(rawDir, `${Date.now()}-${safeSessionId(session.id)}-INCOMING.txt`), `${req.method} ${maskUrlsInText(req.url ?? "")}\n${hdrText}\n\n${bodyBuffer.toString("utf8")}`);
fs.writeFileSync(path.join(rawDir, `${Date.now()}-${safeSessionId(session.id)}-INCOMING.txt`), `${DUMP_PLAINTEXT_BANNER}\n${req.method} ${maskUrlsInText(req.url ?? "")}\n${hdrText}\n\n${bodyBuffer.toString("utf8")}`);
gcDumpDirIfConfigured(rawDir);
} catch (err) { logDumpFailure("INCOMING dump", err); }
}
Expand Down
7 changes: 4 additions & 3 deletions src/server/observability.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@ import { log as loggerLog } from "../logger.js";
import { maskUrlsInText } from "../log-mask.js";
import { bodyDumpEnabled as knobBodyDumpEnabled } from "../knobs.js";

// Body dumps (dumps/req-*.json, raw/*-REQ.txt, raw/*-RES.txt, raw/*-INCOMING.txt,
// req-*-REREQUEST.json) write the full plaintext request body and are off by
// default. They are decoupled from --debug (verbose logging) and enabled only
// Body dumps (dumps/req-*.json, raw/*-REQ.txt, raw/*-INCOMING.txt,
// req-*-REREQUEST.json) write the full plaintext request body; raw/*-RES.txt
// carries the response status + masked headers only. All are off by default.
// They are decoupled from --debug (verbose logging) and enabled only
// with ACP_DUMP_BODY=1 / diagnostics.dumpBody so `bili <client>` users don't
// leak conversation bodies to disk by default (#276).
export function bodyDumpEnabled(): boolean {
Expand Down
10 changes: 6 additions & 4 deletions src/server/relay.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ import { clearUpstreamAlertsForHost, recordUpstreamAlert, recordUpstreamCircuitF
import { describeUpstreamFailure, formatUpstreamError, proxyDispatcher, recordUpstreamConnection } from "../upstream-proxy.js";
import { withUpstreamFailure, type UpstreamFailureKind } from "../upstream-fail.js";
import { applyEstimateCalibration, currentCalibrationFactor, inspectContextOverflow, MIN_ASSUMED_WINDOW_TOKENS, normalizeUpstreamOrigin, usageOutputTotal, usageTotals, type ContextOverflowInfo, type WireProtocol } from "../util.js";
import { DUMP_PLAINTEXT_BANNER } from "../dump-banner.js";
import { recordPckAblation } from "../pck-probe.js";
import { applyOutputSteering, applyOutputSteeringJson } from "../output-steering.js";
import { estimateWireOverhead } from "./budget.js";
Expand Down Expand Up @@ -490,9 +491,9 @@ export async function forward(
const out = path.join(dumpDir, `req-${Date.now()}-${safeSessionId(sid)}.json`);
try {
const pretty = JSON.stringify(JSON.parse(wireText), null, 2);
fs.writeFileSync(out, pretty);
fs.writeFileSync(out, `${DUMP_PLAINTEXT_BANNER}\n${pretty}`);
} catch {
fs.writeFileSync(out, wireText);
fs.writeFileSync(out, `${DUMP_PLAINTEXT_BANNER}\n${wireText}`);
}
log("info", `[debug] forwarded body written to ${out}`);
gcDumpDirIfConfigured(dumpDir);
Expand All @@ -513,7 +514,8 @@ export async function forward(
// headers/exact body bytes; response status+headers) so two consecutive
// requests can be byte-diffed to locate a cache-breaker that the JSON body
// dump (which re-formats and omits headers) may hide. Enabled with
// ACP_DUMP_BODY=1 (credential header values + non-public hosts masked).
// ACP_DUMP_BODY=1 (credential header values + non-public hosts masked;
// #4482: body is plaintext by design — the leading banner warns).
const rawBase =
bodyDumpEnabled()
? (() => {
Expand All @@ -538,7 +540,7 @@ export async function forward(
? wireBody
: Buffer.from(wireBody).toString("utf8");
const reqPath = `${rawBase}-REQ.txt`;
fs.writeFileSync(reqPath, `${req.method ?? "POST"} ${maskUrlForLog(upstreamUrl)}\n${hdrText}\n\n${bodyText}`);
fs.writeFileSync(reqPath, `${DUMP_PLAINTEXT_BANNER}\n${req.method ?? "POST"} ${maskUrlForLog(upstreamUrl)}\n${hdrText}\n\n${bodyText}`);
log("info", `[debug] RAW request dump: ${reqPath}`);
gcDumpDirIfConfigured(path.dirname(rawBase));
} catch (err) { logDumpFailure("REQ dump", err); }
Expand Down
13 changes: 9 additions & 4 deletions tests/error-dump.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { DUMP_PLAINTEXT_BANNER, stripDumpBanner } from "../src/dump-banner.js";
import { dumpRejectedBody, dumpSummaryRejection } from "../src/error-dump.js";
import { setLogCapture } from "../src/logger.js";
import { rmrf } from "./tmp-rm.ts";
Expand Down Expand Up @@ -57,7 +58,9 @@ test("on: JSON body is pretty-printed and parseable", () => {
const files = errFiles();
assert.equal(files.length, 1);
assert.match(files[0]!, /^err-\d+-s1-400\.json$/);
const parsed = JSON.parse(fs.readFileSync(out!, "utf8")) as { model: string; messages: unknown[] };
const text = fs.readFileSync(out!, "utf8");
assert.ok(text.startsWith(`${DUMP_PLAINTEXT_BANNER}\n`), "#4482: plaintext warning banner must lead the file");
const parsed = JSON.parse(stripDumpBanner(text)) as { model: string; messages: unknown[] };
assert.equal(parsed.model, "m");
assert.equal(parsed.messages.length, 1);
} finally {
Expand All @@ -70,7 +73,7 @@ test("on: non-JSON body passes through raw", () => {
try {
const out = dumpRejectedBody(400, "s1", "<html>bad gateway</html>");
assert.ok(out);
assert.equal(fs.readFileSync(out!, "utf8"), "<html>bad gateway</html>");
assert.equal(fs.readFileSync(out!, "utf8"), `${DUMP_PLAINTEXT_BANNER}\n<html>bad gateway</html>`);
} finally {
restoreDumpEnv();
}
Expand Down Expand Up @@ -139,7 +142,9 @@ test("#1993 summary dump on: envelope carries status plus both sides as strings"
const files = summaryErrFiles();
assert.equal(files.length, 1);
assert.match(files[0]!, /^summary-err-\d+-s1-400\.json$/);
const parsed = JSON.parse(fs.readFileSync(out!, "utf8")) as { status: number; request: unknown; response: unknown };
const text = fs.readFileSync(out!, "utf8");
assert.ok(text.startsWith(`${DUMP_PLAINTEXT_BANNER}\n`), "#4482: plaintext warning banner must lead the file");
const parsed = JSON.parse(stripDumpBanner(text)) as { status: number; request: unknown; response: unknown };
assert.equal(parsed.status, 400);
assert.equal(typeof parsed.request, "string");
assert.equal(typeof parsed.response, "string");
Expand All @@ -159,7 +164,7 @@ test("#1993 summary dump on: non-JSON sides stay raw, oversized side is capped",
const text = fs.readFileSync(out!, "utf8");
assert.match(text, /\[truncated: \d+ more character\(s\)\]/);
assert.ok(text.length < 2048 * 3 + 512, `capped file should be far smaller than raw (${text.length})`);
const parsed = JSON.parse(text) as { request: unknown; response: unknown };
const parsed = JSON.parse(stripDumpBanner(text)) as { request: unknown; response: unknown };
assert.equal(parsed.request, "<html>req</html>");
assert.ok(typeof parsed.response === "string" && (parsed.response as string).includes("[truncated:"));
} finally {
Expand Down
5 changes: 4 additions & 1 deletion tests/issue1993-preflight-400.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import type { ProxyOptions } from "../src/config.ts";
import { SessionStore, _setStoreForTest } from "../src/persist.ts";
import { _setForTest as setRegistryForTest } from "../src/registry.ts";
import { listSessions } from "../src/session.ts";
import { DUMP_PLAINTEXT_BANNER, stripDumpBanner } from "../src/dump-banner.ts";
import { setLogCapture } from "../src/logger.ts";
import { rmrf } from "./tmp-rm.ts";

Expand Down Expand Up @@ -319,7 +320,9 @@ test("#1993 BILI_DUMP_4XX persists the rejected summary exchange (request + resp

const files = fs.readdirSync(dumpDir).filter((f) => f.startsWith("summary-err-") && f.endsWith("-400.json"));
assert.ok(files.length >= 1, `expected at least one summary-err dump, dir has: ${JSON.stringify(fs.readdirSync(dumpDir))}`);
const parsed = JSON.parse(fs.readFileSync(path.join(dumpDir, files[0]!), "utf8")) as { status: number; request: unknown; response: unknown };
const text = fs.readFileSync(path.join(dumpDir, files[0]!), "utf8");
assert.ok(text.startsWith(`${DUMP_PLAINTEXT_BANNER}\n`), "#4482: plaintext warning banner must lead the dump");
const parsed = JSON.parse(stripDumpBanner(text)) as { status: number; request: unknown; response: unknown };
assert.equal(parsed.status, 400);
assert.ok(typeof parsed.request === "string" && parsed.request.includes("instructions"), "dump must carry the request bytes");
assert.ok(typeof parsed.response === "string" && parsed.response.includes("request too large: 523200"), "dump must carry the response bytes");
Expand Down
Loading
Loading