Skip to content

[PR] fix: carry the uncovered recent tail in codex forge compaction handoffs (#4035) - #4045

Open
ranxianglei wants to merge 4 commits into
masterfrom
2026-10-21_codex-forge-uncovered-tail
Open

ranxianglei wants to merge 4 commits into
masterfrom
2026-10-21_codex-forge-uncovered-tail

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why (issue #4035)

Codex trigger/prompt forge compaction handed off ONLY the active-block summaries. The recent work between the last fold and compaction time (assistant progress, tool drafts/results not yet folded — 21.7K tokens in the incident session) sat in neither the summaries nor codex's own retention (remote v2 keeps USER messages only; inline prompt form adopts a prose summary), so after a local forge compaction it vanished from the model's view: the model regressed to older progress while bili's local archive still held the content. A real upstream compaction summarizes the WHOLE input — the local forge must be coverage-complete.

Verified against the incident's exact client tag (rust-v0.163.0-alpha.10.2): build_v2_compacted_history retains only user messages (+ narrow conditions), never assistant/tool/reasoning — so the forged blob is the ONLY survival channel for that content.

Fix

  • collectUncoveredTailItems (src/codex-compact.ts): coverage-aware selection over the wire layout. Covered slots stay summarized; partially-covered multi-core slots are skipped honestly (carrying the whole item would resurrect bytes the kernel folded out of it); everything else rides byte-exact — thinking-family included verbatim per KDD release v0.1.3 #10 (move, never rewrite). The prompt-form template message is excluded (instruction, not content).
  • Forged blobs gain a v2 JSON envelope (bili:acp:v2: + {v, summaries, tail}). Empty tails keep the legacy plain-sentinel byte shape stable across builds; pre-v2 blobs still expand, so a mid-session upgrade never loses an already-echoed marker. Malformed v2 → drop (fail-safe).
  • Echo-time expansion splices handoff + tail back in place, deduping key-order-insensitively any tail item codex itself still carries verbatim in the same request (its retention overlaps exactly the newest user turns).
  • Prompt form renders the tail as a labeled section inside the adopted summary text (its channel is prose by protocol).
  • Agent-side fork-adoption replica mirrors the expansion (parity pinned by tests/pi-omp-fork-adopt).
  • Pass-by cleanup (separate commit): 7 dead codex-compact imports dropped from src/server.ts.

Behavior change (disclosed)

After a local forge compaction the model now SEES the previously-dropped recent work until ACP folds it — post-forge context is larger than before, by design (that is the fix). Zero-egress contract unchanged (upstream still untouched); endpoint form unchanged; no-tail sessions see a byte-identical blob to before. No new config surface.

Tests

  • Unit (tests/codex-compact.test.ts): coverage selection matrix (covered/partial/no-core/inactive-block/sub-id normalization), template exclusion, v2 build shapes + legacy byte-stability, v2 expansion order/dedup (incl. reordered-key case), malformed-v2 drop, renderer.
  • E2E regression (tests/codex-compact-e2e.test.ts) — exactly the issue's requested scenario: old fold + uncovered recent draft → trigger → forged blob carries tail (covered content NOT re-carried) → client echo → next forwarded request contains the tail in wire order with retained copies deduped; plus no-tail→legacy-blob and prompt-form rendered-tail e2e cases.
  • Targeted local runs green: codex-compact.test.ts 23/23, codex-compact-e2e.test.ts 23/23, pi-omp-fork-adopt.test.ts 22/22, dsh-authored-compaction + fork-adoption 18/18, typecheck clean. Full suite = CI on this head.

Closes #4035

@ranxianglei ranxianglei changed the title fix: carry the uncovered recent tail in codex forge compaction handoffs (#4035) [PR] fix: carry the uncovered recent tail in codex forge compaction handoffs (#4035) Oct 20, 2026
awork added 2 commits October 21, 2026 03:02
Pass-by cleanup found while working #4035: seven of the ten symbols
imported from ./codex-compact.js were unused (leftovers of the forge
logic moving into prepare-responses); only CODEX_COMPACT_HEALTH_RATIO,
codexCompactMode and isCodexClient are actually referenced.
…fs (#4035)

The trigger/prompt forge handed off ONLY the active-block summaries. The
recent work between the last fold and compaction time (assistant progress,
tool drafts/results not yet folded) sat in neither the summaries nor
codex's retention (remote v2 keeps user messages only; inline adopts a
prose summary), so after a local forge compaction it vanished from the
model's view and the model regressed to older progress — while a real
upstream compaction summarizes the WHOLE input.

- collectUncoveredTailItems: coverage-aware selection over the wire
  layout — slots covered by an active fold stay summarized, partially
  covered multi-core slots are skipped honestly (carrying the whole item
  would resurrect folded bytes), everything else rides byte-exact
  (thinking-family included verbatim per KDD #10); the prompt-form
  template message is excluded as instruction, not content.
- Forged blobs gain a v2 JSON envelope (bili:acp:v2:) carrying summaries +
  tail items; empty tails keep the legacy plain-sentinel byte shape
  stable across builds, pre-v2 blobs still expand (mid-session upgrade).
- Echo-time expansion splices handoff + tail back in place, deduping
  (key-order-insensitively) any tail item codex itself still carries
  verbatim in the same request.
- Prompt form renders the tail as a labeled section inside the adopted
  summary text (its channel is prose by protocol).
- Fork-adoption agent-side replica mirrors the expansion for parity
  (tests/pi-omp-fork-adopt pins it against incomingCoreMessages).

Behavior change (disclosed): after a local forge compaction the model now
SEES the previously-dropped recent work until ACP folds it — post-forge
context is larger than before by design; zero-egress (upstream untouched)
and the endpoint form are unchanged.
@ranxianglei
ranxianglei force-pushed the 2026-10-21_codex-forge-uncovered-tail branch from a4cb6ab to dd8fba1 Compare October 20, 2026 19:05
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-21_codex-forge-uncovered-tail (9755750)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4827-pr.4045.efb0b41

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4045.tgz
npm install -g package

Install instructions are refreshed on each push.

awork added 2 commits October 21, 2026 03:13
…cs-sync gate)

The generated param-ref blocks must come from website/config-reference/*.yaml;
hand-editing CONFIGURATION*.md drifts. Seed now carries the coverage-completeness
disclosure in desc_en/desc_zh; artifacts regenerated.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 检索 open+closed issue/PR:无其他针对 #4035 的并行修复工作。最接近的 #4247(交接文档缺最新助手回复)是另一机制——bili export 快照滞后一拍(snapshotMessages 以入站请求为快照、回复未落盘),子系统与症状均不同,不并入。

标签: bug / codex / scope/wiring / p1(客户端接入层协议缺口;数据丢失类——本地 forge compaction 后近期工作从模型视图消失,核心承诺失灵)。


八问

1. 是什么问题
Codex 本地 forge compaction(trigger/prompt 两种形态)的交接物只装了 active fold 的摘要。最后一次折叠之后、compaction 之前产生的近期工作(助手进度、工具草稿/结果——事故会话里 21.7K tokens)既不在摘要里,也不在 codex 自己的保留里(remote v2 只留 USER 消息,已对照客户端 tag rust-v0.163.0-alpha.10.2 的 build_v2_compacted_history 核实),于是 forge 一完成就从模型视图蒸发,模型回退到旧进度;而 bili 本地归档里内容还在——账实不符。

2. 什么原因导致的
两层根因,都在生产侧:① prepare-responses 的选择表达式只取 blocks.filter(active).map(summary),forge 时从未做覆盖判定,uncovered tail 无人认领;② forged blob 本身是单段明文 sentinel,格式上没有携带结构化 tail 的通道。codex 保留只覆盖最新 user turns,所以 forged blob 是这批内容的唯一生存通道——通道缺一段,内容就丢。

3. 影响范围与严重程度
所有走 codex proxy 模式、命中 intercept 门禁的长会话:已有 active block 且最后折叠后有未折叠消息时触发(trigger 形态有现场实证 bili 0.1.202,prompt 形态同表达式静态证据)。p1:本地接管对外承诺"覆盖完整",实际丢内容,属核心承诺失灵;缓解因素是内容仍存于本地归档、模型可重读文件恢复,物理不可逆丢失未证实。

4. 怎么修复的
在 forge 点做 coverage-aware 收集:collectUncoveredTailItems 按 projection layout 槽位顺序走——已被 active block 覆盖的槽跳过(避免重带已折叠字节)、部分覆盖的多核槽诚实跳过(整条带走会复活被折掉的字节)、无核槽是 wire 脚手架跳过、其余原样字节级带走(thinking-family 逐字移动,符合 KDD #10 move-never-rewrite)。blob 升级为 v2 JSON 信封(bili:acp:v2: + {v,summaries,tail});空 tail 保持 legacy 明文字节形(无 tail 会话 wire 零漂移);旧版明文 blob 仍可展开(会话中途升级安全);malformed v2 丢弃(fail-safe,与既有 id-prefix-only 丢弃路径同类)。echo 时展开把 handoff+tail 原位拼回,并用 key-order 无关指纹对 codex 自己仍逐字携带的副本去重(宁可有界重复、不可丢失;指纹失败时选择携带而非丢弃——安全方向)。prompt 形态的通道按协议是散文,tail 渲染为带标签段落。fork-adopt 端镜像了同一展开逻辑(测试钉住 parity)。另有一个独立 commit 清掉 server.ts 里 7 个死导入(pass-by)。

5. 是否有回归问题
审过的既有行为清单及结论:

  • blob wire 形状:仅 tail 非空才变 v2 JSON,no-tail 字节级不变(unit 钉住 legacy byte-stability)。全仓 encrypted_content 消费者清点:解析 blob 内容的只有 codex-compact.ts 与 fork-adopt.ts(本 PR 同步更新);isBiliCompactionItem/stripBiliCompactionItems 按 id 前缀+sentinel 头识别,v2 blob 仍以 sentinel 开头、识别不受影响;config/preflight/reasoning-guard/responses-ws 里的 encrypted_content 引用都是 reasoning 项字段或缓冲决策,不碰 bili blob 内容。通过。
  • handoff user message 文本:header 常量未动;单元素 summaries join 与旧行为字节相等。通过。
  • echo 展开语义:由「替换为 handoff」变为「handoff+去重后的 tail」——即本修复本身;codex 保留的副本经指纹去重不重复(e2e 断言 carriers()==1)。通过。
  • prompt 形态 adopted summary 变长(追加 tail 段):有意变更,已在 PR 披露。通过。
  • 死导入清理:零行为影响(typecheck 钉住)。通过。
  • 日志行增加 tail 计数/KiB:仅日志。通过。
  • 双压缩模式:该路径 codex-lane 专属(isCodexHost+intercept+gate),plugin 模式会话不触碰。通过。
  • fail-fast/吞错、阈值/时序漂移:无;malformed v2 丢弃是有意的 fail-safe 而非静默吞错;zero-egress 契约不变(e2e 以 h.bodies.length 不变量钉住上游零接触)。

6. 是否破坏缓存
不破坏失效策略。post-forge 上下文比修复前更大——这正是修复本身(此前丢失的内容现在可见,直到 ACP 把它折掉);同一 build 内输出确定性,无-tail 会话出站字节不变,前缀缓存稳定性不受影响。

7. 是否改了设计方向
未转向。这是把既有机制补全到其应有契约:真实上游 compaction 会总结整个输入,本地 forge 必须覆盖完整。机制、边界、数据流都沿用原设计。

8. 是否引入了新的配置
无新配置面。文档就地更新(CONFIGURATION.md en/zh + website diagnostics.yaml 的 codexCompact 行加覆盖完整性说明),符合 owner-gated 纪律。

测试有效性

单测矩阵双向齐全(covered/partial/no-core/inactive-block/sub-id 归一化 × 模板排除 opt-in × v2 构建形状 × malformed 三类 drop);e2e 正是 issue 要求的最小场景全闭环:old fold + uncovered draft → trigger → blob 带 tail 且不重带 covered 内容 → client echo → 下一轮 wire 请求按序含 tail、codex 自留副本去重;另有 no-tail→legacy blob 与 prompt-form 渲染两个 e2e。断言有牙(字节级 includes/excludes、去重计数),不是恒真断言。

CI 状态与合并判定

head 9755750 CI:codex 相关 lane 全绿;5 个红 shard 全部与本 PR 无关——#3769 存储 flake 家族(bare-id loadSync / pre-fold-gate orphan,测试名自带 flake 标记,#4049/#4014 跟踪、#4058 已修)+ 一例 #1365 attach-mode 超时,均为 master 既有抖动家族,逐 job 日志核对过失败测试名。

判定:代码审查通过,可合的前提是先 rebase ——分支落后 master 224 个提交,其中 src/server.ts(#4341/#4185/#4171/#4079 churn)与 src/server/prepare-responses.ts(#4166 mid-history system verbatim,−57 行级改动)正是本 PR 触碰的热文件,rebase 后需过一轮完整 CI 再合。PR 描述对行为变更(post-forge 上下文变大)的披露完整准确,无静默漂移。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codex p1 scope/wiring

Projects

None yet

1 participant