diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 1528ab3fe4..6daf2f3136 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -232,7 +232,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `fakeCompletion` | object | {} (all defaults) | — | Retry loop that re-requests when a final chunk arrives truncated. | | `fakeCompletion.retries` | number | 0 (opt-in) | BILI_FAKE_COMPLETION_RETRIES | How many times a truncated final completion may be re-requested; 0 disables the loop. | | `fakeCompletion.bufCapBytes` | number | 16777216 | BILI_FAKE_BUF_CAP | Buffer size cap for the retry loop's accumulated stream. | -| `codexCompact` | "intercept" \| "pass" | "intercept" | BILI_CODEX_COMPACT | Handle Codex's compaction requests inside bili (intercept, default) or forward them upstream (pass): the explicit forms (compaction_trigger + /responses/compact) and the inline auto-compact that transits as an ordinary turn, detected by its SUMMARIZATION_PROMPT template (#3008) — a user-overridden compact_prompt falls out of detection and transits verbatim; a compaction bili only observed (pass/gate) is rebased post-hoc (#2372). | +| `codexCompact` | "intercept" \| "pass" | "intercept" | BILI_CODEX_COMPACT | Handle Codex's compaction requests inside bili (intercept, default) or forward them upstream (pass): the explicit forms (compaction_trigger + /responses/compact) and the inline auto-compact that transits as an ordinary turn, detected by its SUMMARIZATION_PROMPT template (#3008) — a user-overridden compact_prompt falls out of detection and transits verbatim; a compaction bili only observed (pass/gate) is rebased post-hoc (#2372). The forged handoff is coverage-complete (#4035): besides the active block summaries it carries the recent work since the last fold (the uncovered tail) — trigger form as structured items in the v2 blob envelope (expanded on echo), prompt form rendered into the adopted summary text. | | `ccrRetrievalTtlMs` | number | 600000 (0 disables retrieval) | BILI_CCR_RETRIEVAL_TTL_MS | Lifetime of CCR retrieval pointers before originals stop being fetchable via acp_retrieve. | | `decompressTmpCap` | number | 50 | BILI_DECOMPRESS_TMP_CAP | Size cap (blocks) for temporary decompressed content held in-session. | @@ -977,7 +977,7 @@ Security boundary: the file controls ONLY *when* to upgrade and *which published - **Type:** `string` ("intercept" | "pass") / `number` / `number` - **Default:** `"intercept"` / `600000` / `50` - **Status:** ACTIVE -- **Description:** Top-level scalars. `codexCompact`: whether bili intercepts codex native-compaction requests and forges a local ACP handoff, or passes them upstream (twin `BILI_CODEX_COMPACT`; read per request, so either tier flips without restart). Scope (#3008): interception covers the explicit forms (the `/compact` trigger item and `POST /responses/compact`) and the inline auto-compact that transits as an ordinary turn, detected by its SUMMARIZATION_PROMPT template; a user-overridden compact_prompt falls out of detection and transits verbatim. Compaction bili only observed (pass mode / gate refusal) is rebased post-hoc onto the compacted history (#2372). `ccrRetrievalTtlMs`: expiry for queued-but-undelivered `acp_retrieve` injections, dropped loudly on expiry (twin `BILI_CCR_RETRIEVAL_TTL_MS`; `0` disables). `decompressTmpCap`: max concurrent decompress temp files (twin `BILI_DECOMPRESS_TMP_CAP`). +- **Description:** Top-level scalars. `codexCompact`: whether bili intercepts codex native-compaction requests and forges a local ACP handoff, or passes them upstream (twin `BILI_CODEX_COMPACT`; read per request, so either tier flips without restart). Scope (#3008): interception covers the explicit forms (the `/compact` trigger item and `POST /responses/compact`) and the inline auto-compact that transits as an ordinary turn, detected by its SUMMARIZATION_PROMPT template; a user-overridden compact_prompt falls out of detection and transits verbatim. Compaction bili only observed (pass mode / gate refusal) is rebased post-hoc onto the compacted history (#2372). The forged local handoff is coverage-complete (#4035): in addition to the active-block summaries, the recent work between the last fold and compaction time (the uncompressed tail — assistant progress, tool drafts/results) rides in the forged response — trigger form as structured items inside the forged compaction blob (v2 JSON envelope, expanded back in place on echo, deduping copies codex itself still retains), prompt form rendered as a labeled section inside the adopted summary text; pre-#4035 only the summaries were handed off and the unfolded tail was silently lost from the model's view after a local forge compaction. `ccrRetrievalTtlMs`: expiry for queued-but-undelivered `acp_retrieve` injections, dropped loudly on expiry (twin `BILI_CCR_RETRIEVAL_TTL_MS`; `0` disables). `decompressTmpCap`: max concurrent decompress temp files (twin `BILI_DECOMPRESS_TMP_CAP`). Three more #2030 keys extend existing blocks: [`mitm.handshakeTimeoutMs`](#client-integration) (default `10000`, twin `BILI_MITM_HANDSHAKE_TIMEOUT_MS`), [`compat.noCacheControl`](#compat), and [`compat.keepResponseId`](#compat). @@ -1925,7 +1925,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_ZCODE_SIGNING_FIXED` | `1`/`true` = assume your ZCode build fixed ClientRequestSigningV4 (#1621): the per-entry skips for coding-plan accounts on v3.14+ personal stores flip off and they route again. Temporary escape hatch — there is **no** file equivalent. | | `BILI_CLAUDE_UPSTREAM` | claude direct mode: your relay endpoint, when `ANTHROPIC_BASE_URL` already points at a relay the launcher would otherwise bypass. | | `BILI_CLAUDE_NATIVE_PORT` | Pin an **exact** port for the claude native lane's hook-spawned proxy (#964/#1660): strict-port semantics — a squatter on the port is refused loudly instead of hopping — and the baked managed `ANTHROPIC_BASE_URL` points at it. Without it the hook rides the self-managed zone (`BILI_ZONE_PORT` base + per-lane sticky) and re-pins the managed URL to the live origin each session, so port drift self-heals (#1660). | -| `BILI_CODEX_COMPACT` | Codex native-compaction handling. Default `intercept`: bili intercepts codex's compaction requests and forges a local handoff to the ACP state when the safety gate passes (transform ok + steady-state usage < 90% of the window + at least one active compressed block) — trigger form forges a 2-frame SSE, endpoint form forges `{output}` — and never contacts upstream. Scope (#3008): covers the explicit forms (compaction_trigger + /responses/compact) and the inline auto-compact that transits as an ordinary turn, detected by its SUMMARIZATION_PROMPT template — a user-overridden compact_prompt falls out of detection and transits verbatim; a compaction bili only observed (pass/gate) is rebased post-hoc (#2372). Forged ACP summaries are re-injected as a history-borne handoff message (developer-message fallback) so compressed content stays visible after codex truncates its history. Set `pass` to opt out and forward codex's compaction requests upstream (native compaction backstops). On any gate failure the request passes through untouched. Codex client detection (which traffic this applies to, also used for window clamping and session-identity fingerprinting): the User-Agent must either start with a registered prefix (`codex_cli_rs/`, `codex_exec/`, `codex desktop/` — matched case-insensitively, since variants like Codex Desktop send an initial capital, #1169) or contain a whitespace-delimited component starting with lowercase `codex` (unknown variants such as `codex_sdk_ts/…`, #645 — narrowed from a bare substring to a token-level prefix in #1641 so non-codex clients whose UAs merely mention "codex" mid-token or in parens are no longer misclassified; the fallback stays case-sensitive on purpose so "Codex"-shaped relays are excluded, #1106). | +| `BILI_CODEX_COMPACT` | Codex native-compaction handling. Default `intercept`: bili intercepts codex's compaction requests and forges a local handoff to the ACP state when the safety gate passes (transform ok + steady-state usage < 90% of the window + at least one active compressed block) — trigger form forges a 2-frame SSE, endpoint form forges `{output}` — and never contacts upstream. Scope (#3008): covers the explicit forms (compaction_trigger + /responses/compact) and the inline auto-compact that transits as an ordinary turn, detected by its SUMMARIZATION_PROMPT template — a user-overridden compact_prompt falls out of detection and transits verbatim; a compaction bili only observed (pass/gate) is rebased post-hoc (#2372). Forged ACP summaries are re-injected as a history-borne handoff message (developer-message fallback) so compressed content stays visible after codex truncates its history. The forged handoff is coverage-complete (#4035): it carries not only the active-block summaries but also the recent work since the last fold (uncompressed tail) — trigger form as structured items in the forged compaction blob (v2 envelope, expanded on echo), prompt form rendered into the adopted summary text — so a local forge compaction no longer silently drops recent progress/drafts from the model's view (codex's own retention keeps only user messages). Set `pass` to opt out and forward codex's compaction requests upstream (native compaction backstops). On any gate failure the request passes through untouched. Codex client detection (which traffic this applies to, also used for window clamping and session-identity fingerprinting): the User-Agent must either start with a registered prefix (`codex_cli_rs/`, `codex_exec/`, `codex desktop/` — matched case-insensitively, since variants like Codex Desktop send an initial capital, #1169) or contain a whitespace-delimited component starting with lowercase `codex` (unknown variants such as `codex_sdk_ts/…`, #645 — narrowed from a bare substring to a token-level prefix in #1641 so non-codex clients whose UAs merely mention "codex" mid-token or in parens are no longer misclassified; the fallback stays case-sensitive on purpose so "Codex"-shaped relays are excluded, #1106). | --- diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 8088265ac6..8471e2fc4e 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -232,7 +232,7 @@ | `fakeCompletion` | object | {} (all defaults) | — | 末块被截断时重新发起请求的重试环。 | | `fakeCompletion.retries` | number | 0 (opt-in) | BILI_FAKE_COMPLETION_RETRIES | 被截断的末块完成可被重新请求的次数;0 禁用该重试环。 | | `fakeCompletion.bufCapBytes` | number | 16777216 | BILI_FAKE_BUF_CAP | 重试环累积流缓冲的大小上限。 | -| `codexCompact` | "intercept" \| "pass" | "intercept" | BILI_CODEX_COMPACT | 在 bili 内处理 Codex 的压缩请求(intercept,默认)还是转给上游(pass):覆盖显式形态(compaction_trigger + /responses/compact)与以普通请求形态过线的内联自动压缩(按 SUMMARIZATION_PROMPT 模板检测,#3008)—— 用户自定义 compact_prompt 后锚点失配、按原样转发;bili 仅旁路观察到的压缩(pass/安全阀放行)走事后 rebase(#2372)。 | +| `codexCompact` | "intercept" \| "pass" | "intercept" | BILI_CODEX_COMPACT | 在 bili 内处理 Codex 的压缩请求(intercept,默认)还是转给上游(pass):覆盖显式形态(compaction_trigger + /responses/compact)与以普通请求形态过线的内联自动压缩(按 SUMMARIZATION_PROMPT 模板检测,#3008)—— 用户自定义 compact_prompt 后锚点失配、按原样转发;bili 仅旁路观察到的压缩(pass/安全阀放行)走事后 rebase(#2372)。伪造交接是覆盖完整的(#4035):除活跃块摘要外,还携带上次折叠以来的近期工作(未压缩尾巴)——trigger 形态以结构化条目放进 v2 blob 信封(回显时展开),prompt 形态渲染进被采纳的摘要文本。 | | `ccrRetrievalTtlMs` | number | 600000 (0 disables retrieval) | BILI_CCR_RETRIEVAL_TTL_MS | CCR 回取指针的有效期,过期后原文不再可通过 acp_retrieve 取回。 | | `decompressTmpCap` | number | 50 | BILI_DECOMPRESS_TMP_CAP | 会话内临时展开内容的大小上限(按块计)。 | @@ -977,7 +977,7 @@ - **类型:** `string`("intercept" | "pass")/ `number` / `number` - **默认:** `"intercept"` / `600000` / `50` - **状态:** ACTIVE -- **说明:** 顶层标量。`codexCompact`:bili 是否拦截 codex 原生 compaction 请求并本地伪造 ACP 交接,还是放行到上游(对应 `BILI_CODEX_COMPACT`;按请求读取,两层任一改动都无需重启)。作用范围(#3008):覆盖显式形态(`/compact` 触发的 `compaction_trigger` 与 `POST /responses/compact`)与以普通请求形态过线的内联自动压缩(按 SUMMARIZATION_PROMPT 模板检测);用户自定义 compact_prompt 后锚点失配、按原样转发。bili 仅旁路观察到的压缩(pass/安全阀放行)走事后 rebase(#2372)。`ccrRetrievalTtlMs`:排队未送达的 `acp_retrieve` 注入的过期时限,过期大声丢弃(对应 `BILI_CCR_RETRIEVAL_TTL_MS`;`0` 禁用)。`decompressTmpCap`:并发 decompress 临时文件上限(对应 `BILI_DECOMPRESS_TMP_CAP`)。 +- **说明:** 顶层标量。`codexCompact`:bili 是否拦截 codex 原生 compaction 请求并本地伪造 ACP 交接,还是放行到上游(对应 `BILI_CODEX_COMPACT`;按请求读取,两层任一改动都无需重启)。作用范围(#3008):覆盖显式形态(`/compact` 触发的 `compaction_trigger` 与 `POST /responses/compact`)与以普通请求形态过线的内联自动压缩(按 SUMMARIZATION_PROMPT 模板检测);用户自定义 compact_prompt 后锚点失配、按原样转发。bili 仅旁路观察到的压缩(pass/安全阀放行)走事后 rebase(#2372)。本地伪造交接是覆盖完整的(#4035):除活跃块摘要外,上次折叠到压缩时刻之间的近期工作(未压缩尾巴——助手进展、工具草稿/结果)随伪造响应携带——trigger 形态作为结构化条目放进伪造的 compaction blob(v2 JSON 信封,回显时就地展开,并去重 codex 自身仍保留的副本),prompt 形态渲染为被采纳摘要文本中的带标签小节;#4035 之前只交接摘要,未折叠的尾巴会在本地伪造压缩后从模型视野中静默丢失。`ccrRetrievalTtlMs`:排队未送达的 `acp_retrieve` 注入的过期时限,过期大声丢弃(对应 `BILI_CCR_RETRIEVAL_TTL_MS`;`0` 禁用)。`decompressTmpCap`:并发 decompress 临时文件上限(对应 `BILI_DECOMPRESS_TMP_CAP`)。 另有三个 #2030 键扩展了既有块:[`mitm.handshakeTimeoutMs`](#客户端接入)(默认 `10000`,对应 `BILI_MITM_HANDSHAKE_TIMEOUT_MS`)、[`compat.noCacheControl`](#compat)、[`compat.keepResponseId`](#compat)。 @@ -1919,7 +1919,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_ZCODE_SIGNING_FIXED` | `1`/`true` = 假设你的 ZCode 构建已修复 ClientRequestSigningV4(#1621):v3.14+ personal store 上 coding-plan 账号的逐条跳过关闭,恢复路由。临时逃生舱 —— **没有**文件等效项。 | | `BILI_CLAUDE_UPSTREAM` | claude 直连模式:当 `ANTHROPIC_BASE_URL` 已指向某个 relay 时,用它指定你的 relay 端点(否则会被旁路)。 | | `BILI_CLAUDE_NATIVE_PORT` | 为 claude 原生通道 hook 拉起的代理钉死**确切端口**(#964/#1660):严格端口语义——口上的占用者被大声拒绝而不是跳口——烘进受管 `ANTHROPIC_BASE_URL`。不设则 hook 骑自管区(`BILI_ZONE_PORT` 基准 + 每 lane 粘性),且每次会话把受管 URL 重钉到存活 origin,端口漂移自愈(#1660)。 | -| `BILI_CODEX_COMPACT` | codex 原生压缩处理。默认 `intercept`:安全门通过时(transform 成功 + 稳态用量 < 窗口 90% + 至少一个活跃压缩块)拦截 codex 的压缩请求,在本地伪造向 ACP 状态的交接——trigger 形态伪造 2 帧 SSE,endpoint 形态伪造 `{output}`——且不接触上游。作用范围(#3008):覆盖显式形态(compaction_trigger + /responses/compact)与以普通请求形态过线的内联自动压缩(按 SUMMARIZATION_PROMPT 模板检测)——用户自定义 compact_prompt 后锚点失配、按原样转发;bili 仅旁路观察到的压缩(pass/安全阀放行)走事后 rebase(#2372)。伪造的 ACP 摘要经历史承载交接消息注入(缺席时 developer 消息兜底),保证 codex 截断历史后压缩内容仍可见。设为 `pass` 可退出,把 codex 的压缩请求转发给上游(原生压缩兜底)。任一安全门失败则原样透传。codex 客户端判定(本项生效范围,同时用于窗口 clamp 与会话身份指纹):User-Agent 须以已注册前缀开头(`codex_cli_rs/`、`codex_exec/`、`codex desktop/`——大小写不敏感,因 Codex Desktop 等变体首字母大写,#1169),或含某个以**小写** `codex` 开头的空白分隔组件(未知变体如 `codex_sdk_ts/…`,#645——#1641 起从裸子串收窄为 token 级前缀,UA 中仅在括号内/路径段提及 "codex" 的非 codex 客户端不再被误判;兜底有意保持大小写敏感,排除 "Codex"-形中继,#1106)。 | +| `BILI_CODEX_COMPACT` | codex 原生压缩处理。默认 `intercept`:安全门通过时(transform 成功 + 稳态用量 < 窗口 90% + 至少一个活跃压缩块)拦截 codex 的压缩请求,在本地伪造向 ACP 状态的交接——trigger 形态伪造 2 帧 SSE,endpoint 形态伪造 `{output}`——且不接触上游。作用范围(#3008):覆盖显式形态(compaction_trigger + /responses/compact)与以普通请求形态过线的内联自动压缩(按 SUMMARIZATION_PROMPT 模板检测)——用户自定义 compact_prompt 后锚点失配、按原样转发;bili 仅旁路观察到的压缩(pass/安全阀放行)走事后 rebase(#2372)。伪造的 ACP 摘要经历史承载交接消息注入(缺席时 developer 消息兜底),保证 codex 截断历史后压缩内容仍可见。伪造交接是覆盖完整的(#4035):除活跃块摘要外,还携带上次折叠以来的近期工作(未压缩尾巴)——trigger 形态作为结构化条目放进伪造的 compaction blob(v2 信封,回显时展开),prompt 形态渲染进被采纳的摘要文本——本地伪造压缩不再把近期进展/草稿从模型视野中静默丢掉(codex 自身的保留机制只保留用户消息)。设为 `pass` 可退出,把 codex 的压缩请求转发给上游(原生压缩兜底)。任一安全门失败则原样透传。codex 客户端判定(本项生效范围,同时用于窗口 clamp 与会话身份指纹):User-Agent 须以已注册前缀开头(`codex_cli_rs/`、`codex_exec/`、`codex desktop/`——大小写不敏感,因 Codex Desktop 等变体首字母大写,#1169),或含某个以**小写** `codex` 开头的空白分隔组件(未知变体如 `codex_sdk_ts/…`,#645——#1641 起从裸子串收窄为 token 级前缀,UA 中仅在括号内/路径段提及 "codex" 的非 codex 客户端不再被误判;兜底有意保持大小写敏感,排除 "Codex"-形中继,#1106)。 | --- diff --git a/src/agent/fork-adopt.ts b/src/agent/fork-adopt.ts index c4a0ba7428..1945331b02 100644 --- a/src/agent/fork-adopt.ts +++ b/src/agent/fork-adopt.ts @@ -134,6 +134,7 @@ const FORK_PROXY_MESSAGE_ID_PREFIX = "msg-proxy-"; const FORK_LEGACY_MARKER_MESSAGE_ID_PREFIX = "marker-"; const FORK_CODEX_COMPACT_ID_PREFIX = "fc_bili_"; const FORK_CODEX_COMPACT_SENTINEL = "bili:acp:"; +const FORK_CODEX_COMPACT_BLOB_V2_PREFIX = `${FORK_CODEX_COMPACT_SENTINEL}v2:`; const FORK_CODEX_FORGED_HANDOFF_HEADER = "[bili] context summary after compaction:"; const FORK_RENDER_TAG_RE = new RegExp("\x3c" + ACP_NAME_ALT + "\\s[^\x3e]*\x3e[^\x3c]*\x3c\\/" + ACP_NAME_ALT + "\x3e|\x3c" + ACP_NAME_ALT + "\\s[^\x3e]*\\/\x3e", "g"); @@ -244,32 +245,78 @@ function forkIsBiliCompactionItem(item: unknown): boolean { return false; } -/** Byte-for-byte replica of codex-compact extractBiliSummary. */ -function forkExtractBiliSummary(item: unknown): string | undefined { +// Byte-for-byte replica of codex-compact extractBiliForge (#4035 v2 envelope: +// summaries + uncovered tail; legacy plain-text blobs expand to summaries). +function forkParseForgeV2Blob(blob: string): { v: 2; summaries: string[]; tail: unknown[] } | undefined { + try { + const parsed = JSON.parse(blob.slice(FORK_CODEX_COMPACT_BLOB_V2_PREFIX.length)) as Partial<{ v: unknown; summaries: unknown; tail: unknown }>; + if (parsed.v !== 2 || !Array.isArray(parsed.summaries) || !parsed.summaries.every((s) => typeof s === "string") || !Array.isArray(parsed.tail)) return undefined; + return { v: 2, summaries: parsed.summaries, tail: parsed.tail }; + } catch { + return undefined; + } +} + +function forkExtractBiliForge(item: unknown): { summaryText: string; tail: unknown[] } | undefined { const it = item as { encrypted_content?: unknown } | null; if (!it || typeof it.encrypted_content !== "string") return undefined; - if (!it.encrypted_content.startsWith(FORK_CODEX_COMPACT_SENTINEL)) return undefined; - const text = it.encrypted_content.slice(FORK_CODEX_COMPACT_SENTINEL.length); - return text.length > 0 ? text : undefined; + const blob = it.encrypted_content; + if (!blob.startsWith(FORK_CODEX_COMPACT_SENTINEL)) return undefined; + if (blob.startsWith(FORK_CODEX_COMPACT_BLOB_V2_PREFIX)) { + const payload = forkParseForgeV2Blob(blob); + if (!payload) return undefined; + return { summaryText: payload.summaries.join("\n\n"), tail: payload.tail }; + } + const text = blob.slice(FORK_CODEX_COMPACT_SENTINEL.length); + return text.length > 0 ? { summaryText: text, tail: [] } : undefined; +} + +// Replica of codex-compact stableFingerprint (key-order-insensitive). +function forkStableFingerprint(value: unknown): string | undefined { + try { + const walk = (v: unknown): unknown => + Array.isArray(v) ? v.map(walk) + : v !== null && typeof v === "object" + ? Object.fromEntries(Object.entries(v as Record).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)).map(([k, val]) => [k, walk(val)])) + : v; + return JSON.stringify(walk(value)); + } catch { + return undefined; + } } -/** Byte-for-byte replica of codex-compact replaceBiliCompactionItems. An - * echoed fc_bili_ compaction item is REPLACED (in place) by a plain user - * message carrying the extracted summary — a history-borne handoff the - * kernel can fold again. Marker items without an extractable blob are - * dropped; non-compaction items pass through untouched. */ -function forkReplaceBiliCompactionItems(input: T[]): { items: T[]; replaced: number; dropped: number } { +// Byte-for-byte replica of codex-compact replaceBiliCompactionItems. An +// echoed fc_bili_ compaction item is REPLACED (in place) by a plain user +// message carrying the extracted summary — a history-borne handoff the kernel +// can fold again — followed by the uncovered tail items the v2 blob carries +// (#4035), skipping any tail item the rest of the input already carries +// verbatim. Marker items without an extractable blob are dropped; +// non-compaction items pass through untouched. +function forkReplaceBiliCompactionItems(input: T[]): { items: T[]; replaced: number; dropped: number; tailItems: number } { + const present = new Set(); + for (const item of input) { + if (forkIsBiliCompactionItem(item)) continue; + const fp = forkStableFingerprint(item); + if (fp !== undefined) present.add(fp); + } const items: T[] = []; let replaced = 0; let dropped = 0; + let tailItems = 0; for (const item of input) { if (!forkIsBiliCompactionItem(item)) { items.push(item); continue; } - const summary = forkExtractBiliSummary(item); - if (summary === undefined) { dropped++; continue; } - items.push({ type: "message", role: "user", content: [{ type: "input_text", text: `${FORK_CODEX_FORGED_HANDOFF_HEADER}\n${summary}` }] } as T); + const handoff = forkExtractBiliForge(item); + if (handoff === undefined) { dropped++; continue; } + items.push({ type: "message", role: "user", content: [{ type: "input_text", text: `${FORK_CODEX_FORGED_HANDOFF_HEADER}\n${handoff.summaryText}` }] } as T); + for (const t of handoff.tail) { + const fp = forkStableFingerprint(t); + if (fp !== undefined && present.has(fp)) continue; + items.push(t as T); + tailItems++; + } replaced++; } - return { items, replaced, dropped }; + return { items, replaced, dropped, tailItems }; } /** Project an outgoing Responses body the same way (#2469 stage 2: pi on diff --git a/src/codex-compact.ts b/src/codex-compact.ts index 75ded778b5..42e2e7724c 100644 --- a/src/codex-compact.ts +++ b/src/codex-compact.ts @@ -1,5 +1,6 @@ import { randomUUID } from "node:crypto"; -import type { CompressionBlock } from "acp-kernel"; +import { baseIdOf, type CompressionBlock } from "acp-kernel"; +import type { ResponseInputItem, ResponsesProjection } from "acp-kernel/wire"; import { statusInputBaseline, type Session } from "./session.js"; import { codexCompactMode as knobCodexCompactMode } from "./knobs.js"; @@ -165,48 +166,235 @@ export function stripBiliCompactionItems(input: T[]): T[] { return input.filter((item) => !isBiliCompactionItem(item)); } -// The summary text a forged blob carries (sentinel-prefixed plaintext). -function extractBiliSummary(item: unknown): string | undefined { +// #1634: exact handoff header the model sees on the responses wire; the +// output-side stripper keys off this constant (single source of truth). +export const CODEX_FORGED_HANDOFF_HEADER = "[bili] context summary after compaction:"; + +// #4035: forged blobs minted since this build carry the UNCOVERED tail +// alongside the block summaries in a v2 JSON envelope. Pre-v2 blobs stay +// sentinel-prefixed PLAINTEXT (older builds minted them) — expansion below +// keeps reading them byte-for-byte, so a mid-session upgrade never loses an +// already-echoed marker. +const CODEX_COMPACT_BLOB_V2_PREFIX = `${CODEX_COMPACT_SENTINEL}v2:`; + +interface CodexForgePayload { + v: 2; + summaries: string[]; + tail: ResponseInputItem[]; +} + +function forgeBlob(payload: CodexForgePayload): string { + return `${CODEX_COMPACT_BLOB_V2_PREFIX}${JSON.stringify(payload)}`; +} + +function parseForgeV2Blob(blob: string): CodexForgePayload | undefined { + try { + const parsed = JSON.parse(blob.slice(CODEX_COMPACT_BLOB_V2_PREFIX.length)) as Partial; + if (parsed.v !== 2 || !Array.isArray(parsed.summaries) || !parsed.summaries.every((s) => typeof s === "string") || !Array.isArray(parsed.tail)) return undefined; + return { v: 2, summaries: parsed.summaries, tail: parsed.tail }; + } catch { + return undefined; + } +} + +interface BiliForgeHandoff { + summaryText: string; + tail: ResponseInputItem[]; +} + +// The full handoff an echoed fc_bili_ blob carries: the fold summaries plus +// (#4035) the uncovered recent tail captured at forge time. Legacy plain-text +// blobs expand to summaries only; marker items with no extractable blob yield +// undefined (drop path). +function extractBiliForge(item: unknown): BiliForgeHandoff | undefined { const it = item as { encrypted_content?: unknown } | null; if (!it || typeof it.encrypted_content !== "string") return undefined; - if (!it.encrypted_content.startsWith(CODEX_COMPACT_SENTINEL)) return undefined; - const text = it.encrypted_content.slice(CODEX_COMPACT_SENTINEL.length); - return text.length > 0 ? text : undefined; + const blob = it.encrypted_content; + if (!blob.startsWith(CODEX_COMPACT_SENTINEL)) return undefined; + if (blob.startsWith(CODEX_COMPACT_BLOB_V2_PREFIX)) { + const payload = parseForgeV2Blob(blob); + if (!payload) return undefined; + return { summaryText: payload.summaries.join("\n\n"), tail: payload.tail }; + } + const text = blob.slice(CODEX_COMPACT_SENTINEL.length); + return text.length > 0 ? { summaryText: text, tail: [] } : undefined; +} + +// Key-order-insensitive fingerprint for the echo-time dedup below: the tail +// items were serialized by bili at forge time, the retained copies were +// (re)serialized by codex — same logical object, possibly different key +// order, so a naive stringify comparison would miss the overlap. +function stableFingerprint(value: unknown): string | undefined { + try { + const walk = (v: unknown): unknown => + Array.isArray(v) ? v.map(walk) + : v !== null && typeof v === "object" + ? Object.fromEntries(Object.entries(v as Record).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)).map(([k, val]) => [k, walk(val)])) + : v; + return JSON.stringify(walk(value)); + } catch { + return undefined; + } } // An echoed fc_bili_ compaction item is REPLACED (in place) by a plain user // message carrying the extracted summary — a history-borne handoff the kernel -// can fold again and codex's retention keeps. Rare bounded duplication (a -// block whose anchors survived the truncation also renders) is accepted over -// data loss. Marker items without an extractable blob (id-prefix-only, e.g. -// minted by an older build) are still dropped; real OpenAI blobs pass through +// can fold again and codex's retention keeps — followed (#4035) by the +// uncovered tail items that ride behind it verbatim. Tail items codex ITSELF +// still carries verbatim in the same input (its retention keeps recent user +// messages) are skipped — carrying them too would double the newest turns +// until the next fold; only what codex dropped (assistant progress, tool +// drafts/results) is the gap this fills. Rare bounded duplication (a block +// whose anchors survived the truncation also renders) is accepted over data +// loss. Marker items without an extractable blob (id-prefix-only, e.g. minted +// by an older build) are still dropped; real OpenAI blobs pass through // untouched. -// #1634: exact handoff header the model sees on the responses wire; the -// output-side stripper keys off this constant (single source of truth). -export const CODEX_FORGED_HANDOFF_HEADER = "[bili] context summary after compaction:"; - -export function replaceBiliCompactionItems(input: T[]): { items: T[]; replaced: number; dropped: number } { +export function replaceBiliCompactionItems(input: T[]): { items: T[]; replaced: number; dropped: number; tailItems: number } { + const present = new Set(); + for (const item of input) { + if (isBiliCompactionItem(item)) continue; + const fp = stableFingerprint(item); + if (fp !== undefined) present.add(fp); + } const items: T[] = []; let replaced = 0; let dropped = 0; + let tailItems = 0; for (const item of input) { if (!isBiliCompactionItem(item)) { items.push(item); continue; } - const summary = extractBiliSummary(item); - if (summary === undefined) { + const handoff = extractBiliForge(item); + if (handoff === undefined) { dropped++; continue; } items.push({ type: "message", role: "user", - content: [{ type: "input_text", text: `${CODEX_FORGED_HANDOFF_HEADER}\n${summary}` }], + content: [{ type: "input_text", text: `${CODEX_FORGED_HANDOFF_HEADER}\n${handoff.summaryText}` }], } as T); + for (const t of handoff.tail) { + const fp = stableFingerprint(t); + if (fp !== undefined && present.has(fp)) continue; + items.push(t as T); + tailItems += 1; + } replaced++; } - return { items, replaced, dropped }; + return { items, replaced, dropped, tailItems }; +} + +// #4035: the resent-history items NO active fold covers — the recent work +// between the last fold and compaction time (assistant progress, tool drafts +// and results, reasoning). Codex's own retention keeps only USER messages +// (remote v2 form) or adopts a prose summary (inline prompt form), so without +// this carry the unfolded tail vanishes from the model's view after a local +// forge compaction — exactly the loss a real upstream compaction never makes +// (it summarizes the WHOLE input). Collection walks the projection LAYOUT +// (one slot per input item, in order): +// - slots without a core id (preamble scaffolding, sys/dev, the trailing +// compaction_trigger) are wire furniture, not conversation content; +// - fully covered slots are already summarized by an active block — +// carrying them would duplicate folded bytes; +// - PARTIALLY covered multi-core slots (split user items #3642, demoted- +// thinking splits) are skipped honestly: carrying the whole item would +// resurrect bytes the kernel folded out of it; +// - everything else rides byte-exact (slot.original), thinking-family bytes +// included verbatim per the signature rule (KDD #10: move, never rewrite). +export function collectUncoveredTailItems( + projection: Pick, + blocks: readonly CompressionBlock[], + opts?: { excludePromptTemplate?: boolean }, +): ResponseInputItem[] { + const covered = new Set(); + for (const b of blocks) { + if (!b.active) continue; + for (const id of b.effectiveMessageIds) covered.add(baseIdOf(id)); + } + const slots = Array.isArray(projection.layout) ? projection.layout : []; + // Prompt form: the LAST user message IS codex's SUMMARIZATION_PROMPT + // template (hasCompactionPrompt anchors on it) — it is the instruction, + // not conversation content, and must not ride into future turns. + let lastUserSlot = -1; + for (let i = 0; i < slots.length; i++) { + const original = slots[i].original as ResponseInputItem | null; + if (original && original.type === "message" && original.role === "user") lastUserSlot = i; + } + const tail: ResponseInputItem[] = []; + for (let i = 0; i < slots.length; i++) { + const slot = slots[i]; + if (opts?.excludePromptTemplate && i === lastUserSlot) continue; + const ids = Array.isArray(slot.coreIds) && slot.coreIds.length > 0 ? slot.coreIds : slot.coreId !== undefined ? [slot.coreId] : []; + if (ids.length === 0) continue; + let anyCovered = false; + for (const id of ids) { + if (covered.has(baseIdOf(id))) { + anyCovered = true; + break; + } + } + if (anyCovered) continue; + tail.push(slot.original); + } + return tail; +} + +// Prompt-form channel is prose: the forged assistant message is adopted as +// codex's summary TEXT, so the uncovered tail renders here as labeled lines +// instead of structured items (the trigger form carries the raw items). No +// truncation — the recent draft IS the content this fixes the loss of. +export const FORGED_TAIL_HEADER = "[Recent work since the last fold (not yet summarized)]"; + +function contentPartText(content: unknown): string { + if (typeof content === "string") return content; + if (!Array.isArray(content)) return ""; + const parts: string[] = []; + for (const p of content) { + const part = p as { type?: unknown; text?: unknown } | null; + if (!part || typeof part !== "object") continue; + if (typeof part.text === "string" && (part.type === "input_text" || part.type === "output_text" || part.type === "text" || part.type === undefined)) parts.push(part.text); + else if (part.type === "input_image" || part.type === "output_image") parts.push("[image]"); + } + return parts.join(""); +} + +export function renderUncoveredTailText(tail: readonly ResponseInputItem[]): string { + const lines: string[] = []; + for (const item of tail) { + const it = item as { type?: unknown; role?: unknown; name?: unknown; call_id?: unknown; arguments?: unknown; output?: unknown; input?: unknown; summary?: unknown; content?: unknown }; + switch (it.type) { + case "reasoning": { + const s = it.summary; + const text = Array.isArray(s) + ? s.map((p) => (p as { text?: unknown } | null)?.text).filter((t): t is string => typeof t === "string").join("") + : ""; + if (text.trim().length > 0) lines.push(`[thinking]\n${text}`); + break; + } + case "message": { + const text = contentPartText(it.content ?? ""); + if (text.trim().length > 0) lines.push(`[${it.role === "assistant" ? "assistant" : "user"}]\n${text}`); + break; + } + case "function_call": + lines.push(`[tool call ${String(it.call_id)}] ${String(it.name)}(${typeof it.arguments === "string" ? it.arguments : JSON.stringify(it.arguments ?? "")})`); + break; + case "function_call_output": + lines.push(`[tool result ${String(it.call_id)}]\n${contentPartText(it.output)}`); + break; + case "custom_tool_call": + lines.push(`[custom tool call ${String(it.call_id)}] ${JSON.stringify(it.input ?? "")}`); + break; + case "custom_tool_call_output": + lines.push(`[custom tool result ${String(it.call_id)}]\n${contentPartText(it.output)}`); + break; + default: + break; + } + } + return lines.filter((l) => l.trim().length > 0).join("\n\n"); } // Safety valve preconditions (#332): evaluable BEFORE the kernel transform so @@ -240,11 +428,18 @@ export function buildTriggerForgeBody( summaryText: string, usage: { inputTokens: number; outputTokens: number; totalTokens: number }, stream: boolean, + // #4035: uncovered recent tail to hand off. Non-empty switches the blob to + // the v2 JSON envelope; empty keeps the legacy plain-sentinel byte shape + // stable across builds (no-tail sessions see no wire change). + tail?: readonly ResponseInputItem[], ): { body: string; contentType: string } { + const encryptedContent = Array.isArray(tail) && tail.length > 0 + ? forgeBlob({ v: 2, summaries: summaryText.length > 0 ? [summaryText] : [], tail: [...tail] }) + : `${CODEX_COMPACT_SENTINEL}${summaryText}`; const compactionItem = { type: "compaction", id: `${CODEX_COMPACT_ID_PREFIX}${randomUUID()}`, - encrypted_content: `${CODEX_COMPACT_SENTINEL}${summaryText}`, + encrypted_content: encryptedContent, }; const usageJson = { input_tokens: usage.inputTokens, diff --git a/src/server.ts b/src/server.ts index f960d90726..cc420c1107 100644 --- a/src/server.ts +++ b/src/server.ts @@ -107,7 +107,7 @@ import { reconcileSystemAnchor } from "./system-anchor.js"; import { isStrictReasoningEcho, modelIdOf, normalizeStrictEchoReasoning, normalizeStrictEchoResponsesInput } from "./strict-echo.js"; export { isStrictReasoningEcho, normalizeStrictEchoReasoning, normalizeStrictEchoResponsesInput }; import { sanitizeResponsesInputIds, dropWhitespaceResponsesMessages, normalizeResponsesMessageItems } from "./loop/adapter-responses.js"; -import { CODEX_COMPACT_HEALTH_RATIO, codexCompactMode, codexHealthBarApplies, hasCompactionTrigger, stripBiliCompactionItems, replaceBiliCompactionItems, codexCompactGate, codexCompactGatePre, buildTriggerForgeBody, mergeForgedSummaries } from "./codex-compact.js"; +import { CODEX_COMPACT_HEALTH_RATIO, codexCompactMode, codexHealthBarApplies } from "./codex-compact.js"; import { emitPreflightError } from "./stream-error.js"; import { agentProviderRecipes } from "./agent-providers.js"; import { affinityToken, claudeSubagentAgentId, claudeSubagentSplit, clientConversationHeader, codexTurnIdentity, conversationHeaderSource, dshPersonaFingerprintApplies, instructionsFingerprintApplies, openaiSystemTextForPersona, preferPromptCacheKeyIdentity, type ConversationIdentity } from "./session-id.js"; diff --git a/src/server/prepare-responses.ts b/src/server/prepare-responses.ts index 1bbd183c6e..7b66db197e 100644 --- a/src/server/prepare-responses.ts +++ b/src/server/prepare-responses.ts @@ -30,7 +30,7 @@ import { estimateCoreMessages, estimateCoreMessagesUpper } from "../preflight.js import { hoistTrappedToolItems } from "../tool-pair-order.js"; import { isStrictReasoningEcho, modelIdOf, normalizeStrictEchoResponsesInput } from "../strict-echo.js"; import { reconcileSystemAnchor } from "../system-anchor.js"; -import { buildPromptForgeBody, buildTriggerForgeBody, carriesCodexLocalCompactionSummary, codexCompactGate, codexCompactGatePre, codexCompactMode, hasCompactionPrompt, hasCompactionTrigger, isCodexClient, isCodexHost, mergeForgedSummaries, replaceBiliCompactionItems, stripBiliCompactionItems, CODEX_COMPACT_HEALTH_RATIO, CODEX_LOCAL_COMPACTION_MIN_MISSING } from "../codex-compact.js"; +import { buildPromptForgeBody, buildTriggerForgeBody, carriesCodexLocalCompactionSummary, codexCompactGate, codexCompactGatePre, codexCompactMode, collectUncoveredTailItems, FORGED_TAIL_HEADER, hasCompactionPrompt, hasCompactionTrigger, isCodexClient, isCodexHost, mergeForgedSummaries, renderUncoveredTailText, replaceBiliCompactionItems, stripBiliCompactionItems, CODEX_COMPACT_HEALTH_RATIO, CODEX_LOCAL_COMPACTION_MIN_MISSING } from "../codex-compact.js"; import type { ConversationIdentity } from "../session-id.js"; import { pckKeyAbated } from "../pck-probe.js"; import { logRawStripCounts, markerArtifactCount, stripRawStage } from "../marker/engine.js"; @@ -559,22 +559,34 @@ export async function prepareResponses( session.metadata.codexForgedSummaries = captured; markDirty(session); } + // #4035: the recent work BETWEEN the last fold and now sits in neither + // the block summaries nor codex's retention (user messages only in the + // trigger form, prose summary in the prompt form) — carry it in the + // forged handoff or it vanishes from the model's view after a local + // forge compaction. Coverage reads the active folds just materialized + // above (session.state = turn.state before this block). + const uncoveredTail = responsesProjection !== undefined + ? collectUncoveredTailItems(responsesProjection, session.state.blocks, { excludePromptTemplate: compactForm === "prompt" }) + : []; // Codex recomputes its ledger from the next real request, but the // usage we mint here must not read as an empty context: fall back to // estimating the payload itself when lastInputTokens is stale/zero. // The reply honors parsed.stream (JSON body when not streaming). const est = defaultCountTokens(typeof parsed.input === "string" ? parsed.input : JSON.stringify(parsed.input ?? "")); const total = Math.max(session.stats.lastInputTokens, est, 1); + const usage = { inputTokens: total, outputTokens: 0, totalTokens: total }; codexForge = compactForm === "trigger" ? { kind: "trigger", - ...buildTriggerForgeBody(summaries.join("\n\n"), { inputTokens: total, outputTokens: 0, totalTokens: total }, stream), + ...buildTriggerForgeBody(summaries.join("\n\n"), usage, stream, uncoveredTail), } : { kind: "prompt", - ...buildPromptForgeBody(summaries.join("\n\n"), { inputTokens: total, outputTokens: 0, totalTokens: total }, stream), + ...buildPromptForgeBody( + summaries.join("\n\n") + (uncoveredTail.length > 0 ? `\n\n${FORGED_TAIL_HEADER}\n\n${renderUncoveredTailText(uncoveredTail)}` : ""), + usage, stream), }; - log("info", `[${sessionId}] codex compact intercepted (${compactForm} form); forged ${stream ? "SSE" : "JSON"} with ${summaries.length} block summary(s), upstream not contacted`); + log("info", `[${sessionId}] codex compact intercepted (${compactForm} form); forged ${stream ? "SSE" : "JSON"} with ${summaries.length} block summary(s) + ${uncoveredTail.length} uncovered tail item(s)${uncoveredTail.length > 0 ? ` (~${Math.round(JSON.stringify(uncoveredTail).length / 1024)} KiB)` : ""}, upstream not contacted`); } // #1479: Responses-wire twin of the #762 repair — fold + kernel round-trip diff --git a/tests/codex-compact-e2e.test.ts b/tests/codex-compact-e2e.test.ts index c5762929b0..cffb21907d 100644 --- a/tests/codex-compact-e2e.test.ts +++ b/tests/codex-compact-e2e.test.ts @@ -649,3 +649,120 @@ test("prompt-form detection is codex-only: same template from a non-codex client assert.ok(await reachesUpstreamRequests(h, 1, 2_000), "non-codex UA → ordinary turn"); }); }); + +// --------------------------------------------------------------------------- +// Issue #4035: the trigger/prompt forge handed off ONLY the active-block +// summaries. The recent work between the last fold and compaction time (here: +// conversation messages 2..6 — assistant progress / drafts not yet folded) sat +// in neither the summaries nor codex's retention (remote v2 keeps USER +// messages only; inline adopts a prose summary), so after a local forge +// compaction it vanished from the model's view — the model regressed to older +// progress. A real upstream compaction summarizes the WHOLE input; the local +// forge must be coverage-complete. The forged blob now carries the uncovered +// tail (v2 envelope); on echo it expands back in place, deduping copies codex +// itself still retains. +// --------------------------------------------------------------------------- + +import { FORGED_TAIL_HEADER } from "../src/codex-compact.ts"; + +const V2_PREFIX = "bili:acp:v2:"; + +test("e2e #4035 (trigger form): forged handoff carries the uncovered recent tail end-to-end", async () => { + await withHarness({ mode: "intercept", firstTurnTokens: 1000 }, async (h) => { + const afterSetup = await setupCompressedSession(h); + + // Turn 2: native auto-compact trigger → intercepted + forged. + const r2 = await fetch(h.url, { + method: "POST", + headers: { "content-type": "application/json", "user-agent": CODEX_UA }, + body: JSON.stringify({ model: "gpt-resp", stream: true, session_id: SESSION, instructions: "You are the test coding agent.", input: [...conversation(), { type: "compaction_trigger" }] }), + }); + assert.equal(r2.status, 200); + const frames = (await r2.text()).split("\n\n").filter((f) => f.startsWith("data: ")); + const e1 = JSON.parse(frames[0]!.slice("data: ".length)) as { item: { type: string; id: string; encrypted_content: string } }; + assert.equal(e1.item.type, "compaction"); + assert.ok(e1.item.encrypted_content.startsWith(V2_PREFIX), "uncovered tail present → v2 envelope"); + const payload = JSON.parse(e1.item.encrypted_content.slice(V2_PREFIX.length)) as { v: number; summaries: string[]; tail: unknown[] }; + assert.equal(payload.v, 2); + assert.ok(payload.summaries.some((s) => s.includes("MAIN-SUMMARY-SETUP")), "fold summary in the payload"); + const tailJson = JSON.stringify(payload.tail); + for (let i = 2; i <= 6; i++) assert.ok(tailJson.includes(`WORK_${i}_content_`), `uncovered message ${i} carried in the forged blob`); + assert.ok(!tailJson.includes("WORK_0_content_") && !tailJson.includes("WORK_1_content_"), "covered messages are NOT re-carried (already summarized by the active block)"); + assert.equal(h.bodies.length, afterSetup, "upstream untouched (zero-egress contract holds)"); + + // Turn 3: codex replays [forged item, retained recent user msgs, new + // turn]. The echo must expand into the summary handoff PLUS the + // previously-lost tail — in wire order — while deduping the retained + // copies (messages 5 & 6 ride once, at codex's position). + const r3 = await fetch(h.url, { + method: "POST", + headers: { "content-type": "application/json", "user-agent": CODEX_UA }, + body: JSON.stringify({ model: "gpt-resp", stream: true, session_id: SESSION, instructions: "You are the test coding agent.", input: [e1.item, ...conversation().slice(-2), { type: "message", role: "user", content: "continue the work" }] }), + }); + assert.equal(r3.status, 200); + await r3.text(); + assert.equal(h.bodies.length, afterSetup + 1, "post-forge turn forwarded exactly once"); + + const fwd = h.bodies[h.bodies.length - 1]; + assert.ok(!fwd.includes("fc_bili_"), "echoed bili compaction item replaced before forwarding"); + const idx = (marker: string) => fwd.indexOf(marker); + const handoffIdx = idx("[bili] context summary after compaction"); + assert.ok(handoffIdx >= 0, "summary handoff present"); + let prev = handoffIdx; + for (let i = 2; i <= 6; i++) { + const at = idx(`WORK_${i}_content_`); + assert.ok(at >= 0, `previously-lost message ${i} rides back in the forwarded input`); + assert.ok(at > prev, `message ${i} keeps wire order behind the handoff`); + prev = at; + } + // Count ITEMS carrying the marker (each marker text repeats inside its + // message body), not raw occurrences: a doubled message would show two. + const fwdItems = (JSON.parse(fwd) as { input: unknown[] }).input; + const carriers = (marker: string) => fwdItems.filter((i) => JSON.stringify(i).includes(marker)).length; + assert.equal(carriers("WORK_5_content_"), 1, "codex-retained message 5 rides exactly once (tail copy deduped)"); + assert.equal(carriers("WORK_6_content_"), 1, "codex-retained message 6 rides exactly once (tail copy deduped)"); + assert.ok(!fwd.includes("WORK_0_content_") && !fwd.includes("WORK_1_content_"), "covered content stays summarized, not re-sent raw"); + }); +}); + +test("e2e #4035 (trigger form): no uncovered tail → legacy plain blob (byte-stable no-regression case)", async () => { + await withHarness({ mode: "intercept", firstTurnTokens: 1000 }, async (h) => { + const afterSetup = await setupCompressedSession(h); + // Every message is covered by an active block only if the fold spans + // them all — instead, send a compact whose history IS fully covered: + // replay just the two folded messages + trigger. Their ids are covered + // by the setup block, so the tail collection must come up empty. + const r2 = await fetch(h.url, { + method: "POST", + headers: { "content-type": "application/json", "user-agent": CODEX_UA }, + body: JSON.stringify({ model: "gpt-resp", stream: true, session_id: SESSION, instructions: "You are the test coding agent.", input: [...conversation().slice(0, 2), { type: "compaction_trigger" }] }), + }); + assert.equal(r2.status, 200); + const frames = (await r2.text()).split("\n\n").filter((f) => f.startsWith("data: ")); + const e1 = JSON.parse(frames[0]!.slice("data: ".length)) as { item: { type: string; id: string; encrypted_content: string } }; + assert.equal(e1.item.type, "compaction", "still forged (active block to hand off)"); + assert.ok(!e1.item.encrypted_content.startsWith(V2_PREFIX), "empty tail → legacy plain-sentinel shape"); + assert.ok(e1.item.encrypted_content.startsWith("bili:acp:"), "sentinel intact"); + assert.ok(e1.item.encrypted_content.includes("MAIN-SUMMARY-SETUP"), "summary still carried"); + }); +}); + +test("e2e #4035 (prompt form): forged assistant text appends the rendered uncovered tail", async () => { + await withHarness({ firstTurnTokens: 100, mode: "intercept" }, async (h) => { + const after = await setupCompressedSession(h); + const response = await fetch(h.url, { + method: "POST", + headers: { "content-type": "application/json", "user-agent": CODEX_UA }, + body: JSON.stringify(promptFormBody(SESSION)), + }); + assert.equal(response.status, 200); + const text = await response.text(); + assert.match(response.headers.get("content-type")!, /text\/event-stream/); + assert.ok(text.includes("MAIN-SUMMARY-SETUP-CONTEXT-FOLDED"), "block summary present"); + assert.ok(text.includes(FORGED_TAIL_HEADER), "rendered tail section header present"); + assert.ok(text.includes("WORK_6_content_"), "newest uncovered draft rendered into the adopted summary text"); + assert.ok(!text.includes(CODEX_COMPACT_PROMPT_ANCHOR), "the prompt template itself is excluded from the handoff"); + await new Promise((r) => setTimeout(r, 60)); + assert.equal(h.bodies.length, after, "upstream untouched"); + }); +}); diff --git a/tests/codex-compact.test.ts b/tests/codex-compact.test.ts index ddf350218d..a71f6569e0 100644 --- a/tests/codex-compact.test.ts +++ b/tests/codex-compact.test.ts @@ -285,3 +285,156 @@ test("buildPromptForgeBody: one assistant message codex adopts as its summary (# assert.equal(parsed.output[0]!.content[0]!.text, summary); assert.equal(parsed.usage.total_tokens, 10); }); + +// --------------------------------------------------------------------------- +// #4035: trigger/prompt forge handed off ONLY the fold summaries — the recent +// work between the last fold and compaction time (assistant progress, tool +// drafts/results) sat in neither the summaries nor codex's retention (user +// messages only / prose summary), so it vanished from the model's view after +// a local forge compaction. Fix: coverage-aware handoff — the forged blob +// carries the uncovered tail (v2 envelope), the prompt form renders it into +// the adopted summary text, and echo-time expansion splices it back in place +// while deduping copies codex itself still retains. +// --------------------------------------------------------------------------- + +import { + collectUncoveredTailItems, + renderUncoveredTailText, + FORGED_TAIL_HEADER, + CODEX_FORGED_HANDOFF_HEADER, +} from "../src/codex-compact.ts"; + +type Slot = { original: unknown; coreId?: string; coreIds?: string[] }; + +function slot(original: unknown, coreId?: string, coreIds?: string[]): Slot { + return coreIds !== undefined ? { original, coreIds } : { original, coreId }; +} + +function coveredBlock(ids: string[], active = true): CompressionBlock { + return { ...block(active), effectiveMessageIds: ids }; +} + +test("collectUncoveredTailItems: coverage-aware selection over the wire layout", () => { + const coveredA = { type: "message", role: "user", content: [{ type: "input_text", text: "covered A" }] }; + const coveredB = { type: "message", role: "assistant", content: [{ type: "output_text", text: "covered B" }] }; + const subCovered = { type: "message", role: "assistant", content: [{ type: "output_text", text: "sub-id covered" }] }; + const trigger = { type: "compaction_trigger" }; + const dev = { type: "message", role: "developer", content: [{ type: "input_text", text: "dev" }] }; + const partial = { type: "message", role: "user", content: "split item" }; + const tailA = { type: "message", role: "assistant", content: [{ type: "output_text", text: "recent draft" }] }; + const tailB = { type: "function_call", id: "fc_1", call_id: "call_1", name: "bash", arguments: "{\"command\":\"ls\"}" }; + const tailC = { type: "function_call_output", call_id: "call_1", output: "ok" }; + const tailD = { type: "reasoning", id: "rs_1", summary: [{ type: "summary_text", text: "thinking bytes" }] }; + const projection = { layout: [ + slot(coveredA, "id-a"), + slot(coveredB, "id-b"), + slot(subCovered, "id-z#2"), + slot(trigger), + slot(dev), + slot(partial, undefined, ["id-c", "id-d"]), + slot(tailA, "id-e"), + slot(tailB, "id-f"), + slot(tailC, "id-g"), + slot(tailD, "id-h"), + ] }; + const blocks = [ + coveredBlock(["id-a", "id-b"]), + coveredBlock(["id-c"]), + coveredBlock(["id-z"]), + coveredBlock(["id-e"], false), + ]; + const tail = collectUncoveredTailItems(projection as never, blocks); + assert.deepEqual(tail, [tailA, tailB, tailC, tailD], "only uncovered slots ride, byte-exact, in wire order"); + assert.deepEqual(collectUncoveredTailItems({ layout: [slot(coveredA, "id-a")] } as never, [coveredBlock(["id-a"])]), [], "fully covered → empty"); + assert.deepEqual(collectUncoveredTailItems({ layout: [slot(coveredA)] } as never, [coveredBlock([])]), [], "core-less slot (preamble/sys-dev/unknown-type routing) is wire furniture → never carried"); +}); + +test("collectUncoveredTailItems: prompt template excluded only when asked", () => { + const u1 = { type: "message", role: "user", content: [{ type: "input_text", text: "early question" }] }; + const a1 = { type: "message", role: "assistant", content: [{ type: "output_text", text: "answer" }] }; + const template = { type: "message", role: "user", content: [{ type: "input_text", text: `${CODEX_COMPACT_PROMPT_ANCHOR} Create a handoff summary.` }] }; + const projection = { layout: [slot(u1, "1"), slot(a1, "2"), slot(template, "3")] }; + assert.deepEqual(collectUncoveredTailItems(projection as never, []), [u1, a1, template], "default: everything uncovered rides"); + assert.deepEqual(collectUncoveredTailItems(projection as never, [], { excludePromptTemplate: true }), [u1, a1], "prompt form: the template is the instruction, not content"); +}); + +test("buildTriggerForgeBody: non-empty tail switches the blob to the v2 envelope; empty stays legacy-byte-stable", () => { + const usage = { inputTokens: 100, outputTokens: 0, totalTokens: 100 }; + const tail = [{ type: "message", role: "assistant", content: [{ type: "output_text", text: "draft" }] }]; + const { body } = buildTriggerForgeBody("SUMMARY-TEXT", usage, false, tail); + const parsed = JSON.parse(body) as { output: { type: string; id: string; encrypted_content: string }[] }; + const blob = parsed.output[0]!.encrypted_content; + assert.ok(blob.startsWith("bili:acp:v2:"), "v2 prefix"); + const payload = JSON.parse(blob.slice("bili:acp:v2:".length)) as { v: number; summaries: string[]; tail: unknown[] }; + assert.equal(payload.v, 2); + assert.deepEqual(payload.summaries, ["SUMMARY-TEXT"]); + assert.deepEqual(payload.tail, tail); + + const legacy = JSON.parse(buildTriggerForgeBody("S", usage, false).body) as { output: { encrypted_content: string }[] }; + assert.equal(legacy.output[0]!.encrypted_content, "bili:acp:S", "no tail arg → legacy plain shape unchanged"); + const empty = JSON.parse(buildTriggerForgeBody("S", usage, false, []).body) as { output: { encrypted_content: string }[] }; + assert.equal(empty.output[0]!.encrypted_content, "bili:acp:S", "explicit empty tail → legacy plain shape (byte-stable across builds)"); +}); + +test("replaceBiliCompactionItems: v2 blob expands into handoff + uncovered tail, deduping codex-retained copies", () => { + const retained = { type: "message", role: "user", content: [{ type: "input_text", text: "retained recent question" }] }; + const draft = { type: "message", role: "assistant", content: [{ type: "output_text", text: "uncovered draft" }] }; + const toolCall = { type: "function_call", call_id: "call_x", name: "bash", arguments: "{}" }; + const payload = { v: 2, summaries: ["S1", "S2"], tail: [draft, retained, toolCall] }; + const v2Item = { type: "compaction", id: `${CODEX_COMPACT_ID_PREFIX}v2`, encrypted_content: `bili:acp:v2:${JSON.stringify(payload)}` }; + const next = { type: "message", role: "user", content: "next" }; + const { items, replaced, dropped, tailItems } = replaceBiliCompactionItems([v2Item, retained, next]); + assert.equal(replaced, 1); + assert.equal(dropped, 0); + assert.equal(tailItems, 2, "the codex-retained copy is deduped, the two dropped-by-codex items ride"); + assert.equal(items.length, 5, "handoff + 2 tail + 2 originals, position-preserving"); + const handoff = items[0]! as { type: string; role: string; content: { type: string; text: string }[] }; + assert.equal(handoff.role, "user"); + assert.ok(handoff.content[0]!.text.startsWith(CODEX_FORGED_HANDOFF_HEADER), "handoff header"); + assert.ok(handoff.content[0]!.text.includes("S1\n\nS2"), "summaries joined verbatim"); + assert.deepEqual(items[1], draft, "order preserved: draft before tool call"); + assert.deepEqual(items[2], toolCall); + assert.ok(items.includes(retained) && items.includes(next), "originals untouched"); + + // Same logical item, different key order (codex re-serializes) — the + // fingerprint must still catch the overlap. + const reordered = { role: "user", content: [{ text: "retained recent question", type: "input_text" }], type: "message" }; + const r2 = replaceBiliCompactionItems([v2Item, reordered]); + assert.equal(r2.tailItems, 2, "key order must not defeat the dedup"); +}); + +test("replaceBiliCompactionItems: malformed v2 blobs drop (fail-safe); legacy plaintext still expands", () => { + const badJson = { type: "compaction", id: `${CODEX_COMPACT_ID_PREFIX}a`, encrypted_content: "bili:acp:v2:{not json" }; + const badVersion = { type: "compaction", id: `${CODEX_COMPACT_ID_PREFIX}b`, encrypted_content: `bili:acp:v2:${JSON.stringify({ v: 3, summaries: [], tail: [] })}` }; + const badShape = { type: "compaction", id: `${CODEX_COMPACT_ID_PREFIX}c`, encrypted_content: `bili:acp:v2:${JSON.stringify({ v: 2, summaries: "s", tail: [] })}` }; + const legacy = { type: "compaction", id: `${CODEX_COMPACT_ID_PREFIX}d`, encrypted_content: "bili:acp:legacy summary" }; + const { items, replaced, dropped, tailItems } = replaceBiliCompactionItems([badJson, badVersion, badShape, legacy]); + assert.equal(dropped, 3, "all three malformed v2 shapes drop"); + assert.equal(replaced, 1, "legacy plaintext still expands"); + assert.equal(tailItems, 0, "legacy blobs carry no tail"); + assert.equal(items.length, 1); + assert.ok(JSON.stringify(items[0]).includes("legacy summary")); +}); + +test("renderUncoveredTailText: labeled lines per item kind; unrenderable types filtered", () => { + const out = renderUncoveredTailText([ + { type: "reasoning", summary: [{ type: "summary_text", text: "T1" }, { type: "summary_text", text: "T2" }] }, + { type: "message", role: "user", content: [{ type: "input_text", text: "Q" }, { type: "input_image", image_url: "x" }] }, + { type: "message", role: "assistant", content: [{ type: "output_text", text: "A" }] }, + { type: "function_call", call_id: "c1", name: "bash", arguments: "{\"cmd\":\"ls\"}" }, + { type: "function_call_output", call_id: "c1", output: "ok" }, + { type: "custom_tool_call", call_id: "c2", input: { a: 1 } }, + { type: "custom_tool_call_output", call_id: "c2", output: "done" }, + { type: "extra_tools" }, + ] as never); + assert.ok(out.includes("[thinking]\nT1T2"), "reasoning summary parts joined"); + assert.ok(out.includes("[user]\nQ[image]"), "user text + image placeholder"); + assert.ok(out.includes("[assistant]\nA")); + assert.ok(out.includes('[tool call c1] bash({"cmd":"ls"})'), "tool call labeled with id/name/args"); + assert.ok(out.includes("[tool result c1]\nok")); + assert.ok(out.includes('[custom tool call c2] {"a":1}')); + assert.ok(out.includes("[custom tool result c2]\ndone")); + assert.ok(!out.includes("extra_tools"), "unrenderable types filtered out"); + assert.equal(renderUncoveredTailText([]), ""); + assert.ok(FORGED_TAIL_HEADER.startsWith("["), "header is a bracketed label"); +}); diff --git a/website/config-reference/diagnostics.yaml b/website/config-reference/diagnostics.yaml index 2ca8842960..12512fc360 100644 --- a/website/config-reference/diagnostics.yaml +++ b/website/config-reference/diagnostics.yaml @@ -134,8 +134,8 @@ entries: group: diagnostics id: tt.ccx site_group: adv - desc_en: "Handle Codex's compaction requests inside bili (intercept, default) or forward them upstream (pass): the explicit forms (compaction_trigger + /responses/compact) and the inline auto-compact that transits as an ordinary turn, detected by its SUMMARIZATION_PROMPT template (#3008) — a user-overridden compact_prompt falls out of detection and transits verbatim; a compaction bili only observed (pass/gate) is rebased post-hoc (#2372)." - desc_zh: "在 bili 内处理 Codex 的压缩请求(intercept,默认)还是转给上游(pass):覆盖显式形态(compaction_trigger + /responses/compact)与以普通请求形态过线的内联自动压缩(按 SUMMARIZATION_PROMPT 模板检测,#3008)—— 用户自定义 compact_prompt 后锚点失配、按原样转发;bili 仅旁路观察到的压缩(pass/安全阀放行)走事后 rebase(#2372)。" + desc_en: "Handle Codex's compaction requests inside bili (intercept, default) or forward them upstream (pass): the explicit forms (compaction_trigger + /responses/compact) and the inline auto-compact that transits as an ordinary turn, detected by its SUMMARIZATION_PROMPT template (#3008) — a user-overridden compact_prompt falls out of detection and transits verbatim; a compaction bili only observed (pass/gate) is rebased post-hoc (#2372). The forged handoff is coverage-complete (#4035): besides the active block summaries it carries the recent work since the last fold (the uncovered tail) — trigger form as structured items in the v2 blob envelope (expanded on echo), prompt form rendered into the adopted summary text." + desc_zh: "在 bili 内处理 Codex 的压缩请求(intercept,默认)还是转给上游(pass):覆盖显式形态(compaction_trigger + /responses/compact)与以普通请求形态过线的内联自动压缩(按 SUMMARIZATION_PROMPT 模板检测,#3008)—— 用户自定义 compact_prompt 后锚点失配、按原样转发;bili 仅旁路观察到的压缩(pass/安全阀放行)走事后 rebase(#2372)。伪造交接是覆盖完整的(#4035):除活跃块摘要外,还携带上次折叠以来的近期工作(未压缩尾巴)——trigger 形态以结构化条目放进 v2 blob 信封(回显时展开),prompt 形态渲染进被采纳的摘要文本。" site_en: "codexCompact — intercept | pass (default intercept): handle Codex's compaction requests in bili or forward upstream — explicit forms plus the inline auto-compact transiting as an ordinary turn (prompt-form detection, #3008; overridden compact_prompt transits verbatim)." site_zh: "codexCompact —— intercept | pass(默认 intercept):在 bili 内处理 Codex 压缩请求或转给上游 —— 显式形态 + 以普通请求形态过线的内联自动压缩(prompt 文本检测,#3008;自定义 compact_prompt 按原样转发)。" - key: ccrRetrievalTtlMs