Skip to content

[PR] fix: bound the codex forged-summary handoff ledger (part 1 of #4600) - #4629

Open
ranxianglei wants to merge 14 commits into
masterfrom
2026-10-24_codex-forge-ledger-bound
Open

ranxianglei wants to merge 14 commits into
masterfrom
2026-10-24_codex-forge-ledger-bound

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Part 1 of #4600 — the bounded-handoff mechanics. The injection-policy decision requested in the issue thread (native-compaction boundary write-off / token-budget cap / both) lands in this same branch after the call is made; per the one-issue-one-PR rule there is no separate PR for that part.

What was wrong

Every intercepted codex compaction appends the rendered summaries of all active blocks to session.metadata.codexForgedSummaries (append-only, exact-text dedup), and every request without an echoed bili compaction item re-injects the ENTIRE accumulated list into the developer message (#1064 fallback contract). Blocks nested into newer blocks keep their old entries forever, so the list grows for the life of the session. Field data from #4600: 1,421 entries ≈ 1.21M tokens in a 46-day session → preflight fail-fast 413 against a 375K window; 36 entries grew a 67 KiB request to 422 KiB.

What changed (host-side only, no kernel/ files touched)

  1. Provenance + lineage pruning (src/codex-compact.ts): ledger entries now carry the block they were captured from ({text, blockId}); legacy persisted string lists keep reading through normalizeForgedEntries. pruneForgedEntries drops an entry only when provably redundant: its block is inactive AND a newer block folded the message carrying its summary (summaryMessageId(blockId) ∈ the consumer's effectiveMessageIds) AND that consumer is itself represented (still active — its summary renders live — or holds its own ledger entry). Truncation debt (blocks deactivated only by the post-forge history cut, with nothing consuming them) ALWAYS stays — that is the load-bearing 发现一些bug #1064 fallback. Pruning runs at BOTH sites: forge-time capture (the persisted ledger stops accumulating) and injection time (self-heals state persisted before this fix).
    • Safety property relied on: a kernel block's effectiveMessageIds never contains its OWN acp_summary_<id> (the set is built in unit-projection.ts before block-id allocation; refolds rewrite only summary/topic/runId), so an entry can never be pruned by its own block.
  2. Conversation-evidence gate (carriesConversationTurns): requests that replay no conversation (no assistant/tool/reasoning items — context-only prewarms/sidecars/probes) get NO re-injection at all. They have no replayed history to protect, and previously paid the full accumulated tax, which failed them outright once the list outgrew the window. The gate is view-only: the persisted ledger is untouched.
  3. Observability: warn log when the fallback re-injection eats ≥10% of the model window (FORGED_SUMMARY_INJECT_WARN_PCT, named constant); info/debug logs for prune/skip events.

Behavior changes (disclosed)

Surface Old New Why
session.metadata.codexForgedSummaries persisted shape string[] {text, blockId?}[] provenance needed for lineage pruning; legacy strings read-compat (provenance-less entries are never pruned)
echo-less request that replays no conversation full ledger injected nothing injected no replayed history to protect; the 413 failure mode
superseded entries (nested into a represented newer block) kept forever dropped from injection AND persistence redundant content; the unbounded growth
logs — new debug/info/warn lines as above observability for the p1 class

No wire-format change beyond what the developer-message injection already did; no config surface added (a budget knob belongs to the pending policy decision); no persistence version bump (internal metadata key).

Tests

Out of scope here (awaiting the owner's call in the issue thread)

The #4600 policy options: (1) write off the ledger at native-compaction boundaries, (2) token-budget cap on the injected set (suggested default 25% × window), (3) both (recommended). Lands in this branch after the decision.

ci-check added 2 commits October 25, 2026 07:08
The session-level ledger of forge-time captured block summaries was
append-only with exact-text dedup, and every request without an echoed
bili compaction item re-injected the ENTIRE accumulated list into the
developer message (#1064 fallback). Over a long session that grows
without bound: field data shows 1,421 entries / ~1.21M tokens in a
46-day session failing preflight 413 against a 375K window.

Changes (host-side only, no kernel/ touched):

- Ledger entries now carry provenance ({text, blockId}); legacy
  persisted string lists keep working via normalizeForgedEntries.
- pruneForgedEntries drops an entry only when provably redundant: its
  block is inactive AND a newer block folded the message carrying its
  summary (summaryMessageId in the consumer's effectiveMessageIds) AND
  that consumer is itself represented (still active or holding its own
  ledger entry). Truncation debt stays — it is load-bearing fallback.
  Applied at both the forge-time capture (persisted ledger stops
  accumulating) and injection time (self-heals older persisted state).
- carriesConversationTurns gate: requests replaying no conversation
  (context-only probes/sidecars/prewarms) get no re-injection at all;
  view-only, the persisted ledger is untouched.
- Observability: warn when the fallback re-injection eats >=10% of the
  model window (FORGED_SUMMARY_INJECT_WARN_PCT); info/debug logs for
  prunes and skips.
Small in-pass fix found while working #4600: the import was already
dead on master (only this import line referenced the symbol after the
forge-capture logic moved into prepare-responses).
@ranxianglei ranxianglei changed the title fix: bound the codex forged-summary handoff ledger (part 1 of #4600) [PR] fix: bound the codex forged-summary handoff ledger (part 1 of #4600) Oct 24, 2026
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-24_codex-forge-ledger-bound (7292abb)

Latest build — install this exact version

npm install -g billion-context@0.0.10930-pr.4629.db7c2c5

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
7292abb billion-context@0.0.10930-pr.4629.db7c2c5 build
ce46c44 billion-context@0.0.9691-pr.4629.b62d33a build
f8e135a billion-context@0.0.9555-pr.4629.f67fc09 build
64ae923 billion-context@0.0.9552-pr.4629.1237d94 build
015139a billion-context@0.0.6165-pr.4629.16566a1 build
6f879a3 billion-context@0.0.6078-pr.4629.d886152 build
96e7bfb billion-context@0.0.5980-pr.4629.6281d69 build
1de0c43 billion-context@0.0.5880-pr.4629.ccf7ead build
20b0fe8 billion-context@0.0.5813-pr.4629.5f04c3b build
4f0bb13 billion-context@0.0.5661-pr.4629.63a7403 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4629.tgz
npm install -g package

Install instructions are refreshed on each push.

ci-check added 2 commits October 25, 2026 07:37
…by this PR)

The entry-shape interface is referenced only inside codex-compact.ts
(normalize/merge/prune signatures); no file imports it. INTERNAL_ONLY
per scripts/check-unused-exports.mjs.
Same one-word de-export as #4606 commit 1 (#4599 inventory item 1):
the type is used only inside client-build-note.ts. Duplicate fix kept
on-branch per storm precedent; drop this commit when rebasing after
#4606 merges.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:本 head(af47e02)的 unused-exports 红分两行——① ClientBuildCheck INTERNAL_ONLY = 与 master head 146b34f27 逐字同一报错行(#4599 盘点第 1 条,修复载体 #4606 在途)→ 重复,按风暴先例分支内独立去 export、#4606 合入后 rebase 丢弃;② ForgedSummaryEntry INTERNAL_ONLY = 本 PR 新引入(不重复,直接修)。未开新 issue。

八问

  1. 是什么问题:CI 的 unused-exports 门禁红——两个类型只在自己文件里使用却带着 export;一个是 master 继承的常红行,一个由本 PR 新增。
  2. 什么原因导致的:① tier-5 train 落 master 时留下一个仅文件内使用的 type-only 导出,de-export 清扫漏了它([PR] dsh lane: note in-session when running an unverified @deepseek-ai/dsh client build (soft, exact-match) #4545 引入);② 本 PR 的 entry-shape 接口 ForgedSummaryEntry 只在 src/codex-compact.ts 内部被引用(normalize/merge/prune 三个签名),却随函数一起导出。
  3. 影响范围与严重程度:仅 CI 信号层(门禁误红),用户侧零影响;两个都是类型符号,无运行时产物差异。
  4. 怎么修复的:各一词 de-export(去掉前导 export,声明保留——两类型仍被本文件使用)。两个独立标注 commit:2ef725c0a(本 PR 自身发现)+ 74e5ff0c4(继承行,与 [PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606 commit 1 同法)。本地门禁复现 ✓ unused-export gate: clean (6 waived) exit 0,已 push(新 head 74e5ff0c4)。
  5. 是否有回归问题:无。全仓 grep(src/scripts/tests)确认两符号零外部引用,纯类型导出面收窄;高危模式逐项过:无语义漂移、无默认值/阈值/时序/输出格式变化、无 fail-fast 变吞错。行为变更披露:无(旧→新零行为差异,不涉及 wire/持久化/配置面)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。
  8. 是否引入新配置:没有。

CI 状态(新 head 74e5ff0c4)

结论:转发的红已修复推送,PR 回到「等 #4606」状态,无需 owner 动作。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

判定:继承性失败,分支侧不动手,等 #4606 合入后 rebase。

证据(run 40143511023,head 74e5ff0c4):ubuntu-22 分片 1 与 windows-22 分片 1 日志逐字同一报错行——
process-spawning test files missing from LIVE_FILES in scripts/run-ci-tests.mjs: tests/issue3890-parent-gone-confirm-drain.test.ts. Register them there after reviewing their concurrency assumptions (#1248)
pretest 自检直接 exit 1,一行真实测试都没跑;即 #4599 盘点第 2 条。本 PR diff 只碰 5 个文件(3 src + 2 codex-compact 测试),不触及 scripts/run-ci-tests.mjs,tripwire 输出也未列出本 PR 的任何测试文件——红与 master head 146b34f27 同因。另:attempt 1 修的 unused-exports 在本 run 已绿(本次转发里没有它),静态门禁全绿。

八问

  1. 是什么问题:11 个 test 分片全部死在跑测试前的自检——一个会起子进程的测试文件没登记进 LIVE_FILES 白名单,门禁按设计 fail-fast。
  2. 什么原因导致的:tier-5 train([PR] merge: tier-5 train — dsh prepare/streaming domain (7 PRs: bili_final terminal marker, auto-fold storm gate, wire-only guard) #4590)新增该测试文件时漏登记,master 面按设计不跑重型 test 车道所以没显形,任何 rebase 过它的 PR head 全中;修复在 [PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606 commit 2,等 owner 合并。
  3. 影响范围与严重程度:所有在途 PR 的 test 矩阵(信号层停摆);本 PR 自身代码零失败证据——没有任何一条真实测试跑过、更没有失败。
  4. 怎么修复的:本车道不修——① 登记是 [PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606 的在途内容,分支内重复修违反「一个根因一个 PR」且合入后 rebase 还得丢弃;② 单独修掉 tripwire 会让全部分片开始真跑套件,把 ~35 个既有 born-red 用例显形成真实失败,矩阵更红而非更绿;③ 登记本身要求先做并发假设评审(tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248),属 [PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606 范围。
  5. 是否有回归问题:不涉及(未改任何代码)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。
  8. 是否引入新配置:没有。

收工判定:auto-fix 车道 2/2 用完。本 PR 侧能修的红(unused-exports ×2)已在 attempt 1 修掉并转绿;剩余 11 片红全部继承自 master,唯一解锁动作 = owner 合并 #4606,之后本分支 rebase 重触发即绿(届时若 born-red 集群仍红,按 #4599 清单逐机制 triage,不在本 PR 范围)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 检索 open+closed issue/PR 无同机制条目(forged-summary ledger 有界化);相关项已互链:父 issue #4600(本 PR 为其 part 1)、#1064(回注 fallback 出处)、#2372(codex 本地 auto-compaction 路径)、#4594(context-only 辅助流量形状)。一个 issue(#4600)→ 一个 PR,符合 loop guard。

标签: bug / codex / scope/wiring / p1(handoff ledger 只增不减属缓存泄露类,按 owner 修订 2026-10-19 一律 p1)

审核进行中,先报一个关键发现(完整八问结论随后):

本 PR 的核心机制——provenance + lineage pruning——在真实 kernel 状态下不会触发,是死机制。

证据链:

  1. prune 的触发条件要求某个更新块的 effectiveMessageIds 含旧块的 acp_summary_<blockId>;
  2. 但 kernel 唯一 fold 投影 kernel/src/unit-projection.ts:135-137 把所有 acp_summary_* id 从 fold 范围剔除(!isSummaryMessageId(id)),effectiveMessageIds 只由该范围 + consumed 块继承集构成(归纳上永远干净);kernel/src/prune.ts 的文档注释也明说 summary id 是 VIEW-ONLY,"must never be persisted into effectiveMessageIds/directMessageIds";
  3. grep kernel/src + src/ 全量:无其他写入点(preflight commitFold 取自 kernel preview state,同一投影);
  4. 实证(真实 kernel createCore + 真实 fold 探针 .tmp/probe-prune-liveness.ts):块 coverage 只有 raw message id;consumer lookup 0 命中;pruneForgedEntries 对账本条目原样保留。

后果:

确实有效的部分:carriesConversationTurns 门控(消除 probe/sidecar 413 类)、warn 可观测性、entry 形状迁移(无害、读兼容)。声称的安全性属性(不自剪)成立——实际比声称更强(summary id 根本不进任何 coverage 集)。

此发现强化 #4600 待决策的必要性:option 2(budget cap)使 413 结构性不可能,与 prune 是否有效无关;option 1(边界核销)覆盖主导模式。part 1 的核心贡献低于正文所述。

进行中:定向测试已过(codex-compact.test.ts 20/20、codex-compact-e2e.test.ts 22/22)→ 正在分支内修 CI 两处 unused-exports 红(本 PR 引入 1 处 + master 继承 1 处)→ push → CI → 八问终局结论。

@ranxianglei ranxianglei added bug Something isn't working codex p1 scope/wiring labels Oct 25, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

PR 审核终局结论(独立重审,不复述 issue 侧结论)

总判定:hold,暂不合并。 代码质量达到合并标准、CI 红已处置(floors 5/8),但核心机制(provenance + lineage pruning)在真实 kernel 状态下不触发(floor 10 证据链)——本 PR 现状合入后,#4600 的用户可见问题(长会话 ledger 无界增长 → 413)依然存在。需要 #4600 政策决策先行,同分支收口。

① 方向关(原 issue)

#4600 要修的是「debt without lifecycle」:append-only capture + 对每个无 echo 请求全量回注 → 无界增长。方向判断正确(根因不是表面现象)。Part 1 三机制实测:(a) lineage pruning = 死机制(kernel/src/unit-projection.ts:135-137 把所有 summary id 剔出 fold 范围,effectiveMessageIds 结构上永远干净,prune 的触发条件永不成立);(b) carriesConversationTurns 门控 = 有效;(c) warn 可观测性 = 有效。PR 正文自述主导模式修复等 owner 拍板 option 1/2/3 后同分支落地——即本 PR 现状本就不关 #4600,但正文对 part 1 价值的表述("覆盖 normal fold-rotation sessions")不成立。

② 回归审查(既有行为逐项清点)

既有行为 结论
ledger 持久化形状 string[]→{text,blockId?}[] 有意变更,已披露,读兼容(normalizeForgedEntries 接受全部历史形状含 legacy 字符串)。全仓审计:唯一读写点 = src/server/prepare-responses.ts,零其他消费者 ✓
回注范围:每个无 echo 请求 → 仅携带会话轮次的请求 有意变更,旧→新已披露。方向正确(probe/sidecar 无 replayed history 可保护,消除 413 类)。残留风险低:异形 item 形状漏判 predicate 会丢该轮 handoff 保护——predicate 按设计偏宽松(type-less assistant 也计入),codex 实流量不受影响
forge-time capture 纯 append → merge+prune 真实状态下 prune 恒为 no-op(死机制)→ 行为与改前一致,无回归亦无收益
dev-message 注入文本/顺序(前缀缓存接缝) 会话请求字节级不变(同文本、同序、同 joiner);辅助流量失去注入(一次性前缀变化,且它们正是此前炸窗的一侧)✓
server.ts 死 import 移除 / 新增日志 零行为 / 纯增量

高危模式逐项过:语义漂移无、fail-fast 变吞错无、阈值/时序漂移无(FORGED_SUMMARY_INJECT_WARN_PCT=0.1 仅日志阈值不改行为)、输出格式变化无。

③ 代码审查(主关)

纯函数组(normalize/merge/prune/carriesConversationTurns)本身整洁:命名达意、注释讲为什么带 #NNNN 引用、guarded cast、具名常量、风格与仓库一致。问题不在代码质量,在机制所依赖的数据关系不存在——约 100 行当前不可达代码路径 + 依赖它的测试。

④ 测试有效性

  • e2e「request replaying no conversation gets no re-injection」:真测试——真 processTurn、真 forge、断言 probe 零注入 + 持久化 ledger 原样。钉住了有效行为 ✓
  • 单测 prune 矩阵 + e2e「forge-time capture prunes superseded」:恒真测试——手工注入不可能状态(e2e:liveBlock.effectiveMessageIds.push("acp_summary_b_stale_1", …)),kernel 永不产生该状态;bug 回归时抓不住(bug 恰是「prune 永不触发」,测试却假设它触发)。
  • 缺失:没有测试钉用户可见属性「多 forge 轮转长会话下 ledger 有界」——part 1 提供不了这个属性,这是政策部分必须补的钉。

⑤/⑤b diff 整洁度与代码硬标准

干净:每一行服务本 PR 目的,无关文件/生成物/空白 churn 为零,外部提交硬标准全项通过。唯一不符项在正文表述而非 diff(见①)。

八问

  1. 是什么问题:codex compaction 交接账本只增不减,每个无 echo 请求把全量重注入 developer message → 长会话注入吞掉上下文窗口,超窗后 preflight 直接 413(现场:46 天会话 1421 条 ≈1.21M tokens vs 375K 窗口)。
  2. 什么原因导致的:债务没有生命周期——capture 端 append-only(exact-text dedup 只防重复不防增长),注入端无条件全量,两端都没有核销或上限。
  3. 影响范围与严重程度:p1(缓存泄露类,按 owner 修订 2026-10-19)。触发条件 = 长寿命 codex 会话 + 多次被拦截 compaction;后果是会话完全不可用(413 硬墙),非体验瑕疵。
  4. 怎么修复的:part 1 给债务加生命周期——lineage 核销(死,证据见上)+ 门控非会话流量(有效)+ 可观测性(有效);主导模式修复待 codexForgedSummaries grows without bound and is re-injected in full whenever a request lacks the echoed compaction item #4600 政策拍板(option 1 边界核销 / option 2 预算上限 / option 3 两者)后同分支落地。
  5. 是否有回归问题:无。上表逐项审毕;唯一低危残留(conversation predicate 假阴性丢 handoff)已列明,设计偏宽松、codex 实流量不发生。
  6. 是否破坏缓存:否。会话请求前缀字节稳定;辅助流量失去注入属一次性前缀变化,无惩罚。
  7. 是否改了设计方向:无转向,在既有 发现一些bug #1064 fallback 上加生命周期纪律。但需修正一处方向认知:lineage-pruning 支柱依赖的 kernel 数据关系不存在(summary id 是 VIEW-ONLY,被所有 fold 投影过滤)——该支柱去留应随政策决策一并定。
  8. 是否引入新配置:无。FORGED_SUMMARY_INJECT_WARN_PCT=0.1 是代码常量(仅日志阈值);若日后要可调阈值,按 config-surface 流程提提案。

CI 状态(head 74e5ff0c4)

收工判定与建议

Hold,不合。 明确建议(不甩选择题):

  1. owner 现在拍板 codexForgedSummaries grows without bound and is re-injected in full whenever a request lacks the echoed compaction item #4600 政策——推荐 option 3:边界核销覆盖主导模式 + 预算上限使 413 结构性不可能,两者互补;
  2. 拍板后同分支收口:实现 1+2,移除(或明确降级标注)死的 lineage-pruning 支柱——~100 行不可达代码 + 恒真测试留着会误导后续维护者以为 lineage 核销在工作;替换为真实状态回归钉(多 forge 轮转 e2e,断言 cap/核销下 ledger 有界);
  3. 届时 PR 以闭环形态关 codexForgedSummaries grows without bound and is re-injected in full whenever a request lacks the echoed compaction item #4600,[PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606 解锁后 rebase 重触发转绿即可合并。

ci-check added 2 commits November 13, 2026 20:34
…n codex-compact/server/prepare-responses)

Merging origin/master (99c22d2) into the PR head to clear the
merge-conflict blocker. Three import-line conflicts resolved by union:
- src/codex-compact.ts: keep master's createHash (nativeCompactionKey)
  + PR's baseIdOf/summaryMessageId (forged-ledger provenance).
- src/server.ts: keep master's CommandcodeEnvelopeMeta import; drop the
  dead mergeForgedSummaries import (unused in server.ts body, PR's own
  intent per af47e02).
- src/server/prepare-responses.ts: union of PR's carriesConversationTurns/
  normalizeForgedEntries/pruneForgedEntries/FORGED_SUMMARY_INJECT_WARN_PCT
  + master's nativeCompactionKey. All symbols verified used + exported.

Typecheck (tsc --noEmit) green; targeted codex-compact tests 42/42 pass.
…state

Master advanced 3 commits (pure-SQLite default #6480 + notice gate test)
after the initial conflict merge; re-syncing so the PR head is clean
against current master. No conflicts.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI red triage — PR #4629 (4 failing test shards)

What's failing: 2 tests in tests/notice-registry.test.ts — visibleNotices: fresh install sees the default flip + pck + wm intro and visibleNotices: dismissing one hides only that one. Both expect storage-sqlite-default in the visible notices but it's missing.

Root cause analysis:

  • The notices code (src/notices.ts, src/notice-store.ts, tests/notice-registry.test.ts) is byte-identical between master and this PR — the PR diff (5 files: codex-compact, server, prepare-responses, 2 test files) does not touch any notices file.
  • The test passes locally (5/5) but fails on CI (4 shards: ubuntu-22, ubuntu-24, windows-22×2) — a CI-environment-specific failure.
  • The test was last changed by [PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480 (pure-SQLite default, 2026-11-13, commit 5a80e2b).
  • sqliteDefaultNoticeActive() checks 6 conditions; one is failing in the CI environment but not locally.

Verdict: This is a master-level CI-specific failure, not introduced by this PR. The PR's own code is clean — typecheck + targeted tests (42/42 codex-compact) pass locally.

Action taken:

Remaining: The 4 test shards will stay red until the notices test is fixed on master/ci-fixes. All other checks (GATE workflow: 4 success, 12 queued, 14 skipped, 0 failures) are clean.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI red triage — PR #4629 (2 failing test shards, head ce46c44)

What's failing: 13 tests across 2 Windows shards, all with the same error:

EBUSY: resource busy or locked, unlink '...index.db'

Root cause: Windows CI-environment file locking — the SQLite index.db file is held by the OS (AV/indexer) when the test teardown calls rmrf → unlink. The CI runner's own diagnostic confirms: "persistent external lock (AV/indexer) on this runner" (#4347). A full retry on the same runner reproduced the lock; a fresh runner is needed.

Not a code bug: The PR diff (5 files: codex-compact, server, prepare-responses, 2 test files) does not touch persist/storage paths. The notices-test fix from the previous wake (c6369fb) is in this head and the notices failures are gone.

Action taken:

Remaining: Waiting for the re-run to complete. If the EBUSY reproduces on fresh runners too, the leaked-handle hypothesis (#4347) becomes primary and needs a persistent fix in the test teardown (close SQLite handles before rmrf).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI red triage — PR #4629 (re-run completed, same 2 shards still red)

The rerun-failed-jobs I triggered last run completed. The same 2 Windows shards are still failing with the identical EBUSY error (13 + 10 = 23 teardown-path file-lock failures).

CI's own diagnostic (from the job logs):

"all 10/3 failing subtest(s) are teardown-path transient OS locks (EBUSY/EPERM/... on unlink/rmdir through rimraf): the #2799/#2908 family — an external AV/indexer hold on freshly-written files outliving the tests/tmp-rm.ts retry ladder, never an assertion regression (#4347)."

Root cause: Windows CI runners have an external AV/indexer that holds freshly-written SQLite index.db files. The current retry ladder in tests/tmp-rm.ts sums to ~9.5s ([25,50,100,200,400,800,1600,3200,3200]) — already escalated in #2908 from ~3.2s for the c=4 concurrent scheduling — but the AV/indexer lock now outlasts even that window.

Not a code bug / not this PR's fault:

Why I'm not fixing it in this PR branch:

Status: All 28 other checks are green (ubuntu × 22/24, windows × 22 shards 2/4/5, all static gates, kernel-guard, docs, lock-sync, etc.). Only the 2 Windows EBUSY shards remain red, pending a ci-fixes retry-ladder widening.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (windows-latest, 22, 3), test (windows-latest, 22, 1) — stopping for human review

2 similar comments
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (windows-latest, 22, 3), test (windows-latest, 22, 1) — stopping for human review

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (windows-latest, 22, 3), test (windows-latest, 22, 1) — stopping for human review

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Conflict resolved & pushed — PR #4629

Merged master (a446a3c, 493 commits ahead) into the PR branch. One conflict in src/codex-compact.ts import lines, resolved as union:

  • Kept PR's baseIdOf, summaryMessageId (both used at lines 421, 398/416/432)
  • Kept master's evidenceInputBaseline (used at line 243)
  • Dropped dead statusInputBaseline (import-only, never called in body)

Verified: typecheck clean, 46/46 targeted tests pass (codex-compact + codex-compact-e2e). Pushed as merge commit 7292abb (no force-push).


八问

  1. 是什么问题: PR 分支落后 master 493 个提交,merge state=dirty(冲突),无法合并。
  2. 什么原因导致的: master 在 PR 分支创建后持续前进,5 个 PR 文件(src/codex-compact.ts, src/server.ts, src/server/prepare-responses.ts, 2 个测试文件)在两侧都有改动;其中 src/codex-compact.ts 的 import 行产生文本冲突。
  3. 影响范围与严重程度: 仅 PR 分支的合并状态,不影响任何已发布行为;不解决则 PR 永远 blocked。
  4. 怎么修复的: merge master 进 PR 分支(非 rebase,保留历史);唯一冲突按 union 解决——保留两侧实际使用的符号,删除死导入 statusInputBaseline。
  5. 是否有回归问题: 无。typecheck 全量通过;46/46 定向测试通过;PR diff 仍为原 5 文件,无 kernel/ 改动。
  6. 是否破坏缓存: 不涉及。
  7. 是否改了设计方向: 不涉及。
  8. 是否引入了新的配置: 不涉及。

判定: 冲突已解决,PR 分支已推送到 7292abb,等待 CI 在新 head 上重跑。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: opencode/qwen3.8-27b)

REQUEST_CHANGES — the core "bound the ledger" mechanism (lineage pruning) is dead code against real kernel state: the predicate it relies on is never true, so the persisted ledger still grows unboundedly for the main #4600 scenario. The conversation-evidence gate works and is correct, but it only removes the 413 for context-only probes/sidecars — it does not bound the ledger or fix the 413 for real conversation requests.

Blocking: pruneForgedEntries never fires (src/codex-compact.ts:401)

The pruning drops an entry only when some consumer block's effectiveMessageIds contains the entry's block summary id:

  • src/codex-compact.ts:411 registers the lookup key summaryMessageId(e.blockId) (= acp_summary_<blockId>);
  • src/codex-compact.ts:414-425 scans each block's effectiveMessageIds for that key;
  • src/codex-compact.ts:427 prunes when a represented consumer is found.

But the kernel never puts an acp_summary_* id into any block's effectiveMessageIds:

  • kernel/src/unit-projection.ts:181: const rangeMessageIds = adjustedIds.filter((id) => !isSummaryMessageId(id) && !liveCarrierIds.has(id)); — summary ids are explicitly excluded from the fold range, and effectiveMessageIds is seeded from rangeMessageIds (line 237) plus the consumed blocks' raw ids (lines 240-242).
  • kernel/src/prune.ts:17 documents summaryMessageId as "VIEW-ONLY ... must never be persisted into effectiveMessageIds/directMessageIds".
  • kernel/tests/regression-promote-after-prune.test.ts:465 pins the invariant: assert.ok(!isSummaryMessageId(id), 'synthetic id leaked into block: ' + id).

The kernel records block lineage in directBlockIds, not in effectiveMessageIds with summary ids. I verified this empirically against the PR's exact kernel (0.0.199, built at the PR head): folding a range that consumes block b2 produces a new block with directBlockIds = [b2] and effectiveMessageIds = [raw-3, raw-4] — no acp_summary_b2. Running the PR's predicate against that real state finds zero consumers, so the b2 ledger entry is never pruned.

Consequence: the persisted codexForgedSummaries ledger still accumulates forever for the #4600 field scenario (blocks nested into newer blocks across a long-running conversation — the 1,421-entry / 1.21M-token case). Both pruning sites (injection-time and forge-time in src/server/prepare-responses.ts) call this dead predicate. The e2e test tests/codex-compact-e2e.test.ts ("forge-time capture prunes superseded ledger entries") fabricates the state the kernel forbids — liveBlock.effectiveMessageIds.push("acp_summary_b_stale_1", ...) — so it passes while the real behavior is broken. The PR's "safety property" note (a block's effectiveMessageIds never contains its own summary id) is true but proves only no-self-pruning; it does not establish that a consumer carries the old block's summary id.

Suggested fix: key supersession on the kernel's actual lineage record — an entry for block X is redundant iff X is inactive AND some block Y is represented (Y.active, or Y holds its own ledger entry) AND X is reachable from Y via the transitive directBlockIds chain (that is how the kernel records "Y consumed X"). Truncation debt (a block deactivated by the post-forge history cut with no consumer) then stays, exactly as the PR intends. Re-verify against a real applyCompression-produced state (not a hand-built block) so the regression test pins real kernel behavior.

What works (no change needed)

  • carriesConversationTurns gate (src/codex-compact.ts:451) — correct; context-only requests pay nothing, and it errs toward injecting (lenient by design, matching the comment). The e2e probe test is valid.
  • normalizeForgedEntries / mergeForgedSummaries shape change (string[] → {text, blockId?}[]) with legacy read-compat — correct; the only reader of the ledger is src/server/prepare-responses.ts, and the shape change is disclosed.
  • Observability warn log (FORGED_SUMMARY_INJECT_WARN_PCT, a code-internal named constant, not a config key) — fine.

结构化八问

  1. 是什么问题:PR 声称"bound the codex forged-summary handoff ledger",但核心谱系剪枝(pruneForgedEntries)在真实 kernel 状态下永不触发——它依赖的谓词(consumer 的 effectiveMessageIds 含 acp_summary_<blockId>)在 kernel 中恒为假。
  2. 什么原因导致的:kernel 从不把 acp_summary_* 写进任何 block 的 effectiveMessageIds(unit-projection.ts:181 显式过滤;prune.ts 标注 VIEW-ONLY;kernel 自身回归测试钉死"synthetic id never leaks into block")。kernel 用 directBlockIds 记录谱系。PR 把"block 的 effectiveMessageIds 不含自己的 summary id"(真,仅防自剪枝)误当成了"consumer 含旧 block 的 summary id"(假)。
  3. 影响范围与严重程度:持久化账本在 #4600 主场景(长会话嵌套 block)下仍无界增长——p1 未修复;注入时与 forge 时两个剪枝点都调用死谓词。e2e 测试伪造了 kernel 禁止的状态故通过,真实行为是坏的。会话证据门有效,但只消除 context-only probe/sidecar 的 413,不消除增长,也不消除真实会话请求的 413。
  4. 怎么修复的:把取代判定键到 kernel 真实谱系记录上——entry X 冗余当且仅当 X inactive 且存在被代表的 Y(Y.active 或 Y 有自己的账本 entry)使 X 可从 Y 经 transitive directBlockIds 链到达;用真实 applyCompression 状态(非手搓 block)重验并让回归测试钉住真实 kernel 行为。
  5. 是否有回归问题:本 PR 未合入,无既有行为被破坏。按建议修复后剪枝语义与 PR 原意一致(truncation debt 保留、被取代 entry 剪除),仅换正确信号,无额外回归。
  6. 是否破坏缓存:不涉及——账本是 session.metadata 内部键,非缓存/重试/增量游标。
  7. 是否改了设计方向:不涉及(#1064 handoff 契约不变;token 预算上限仍待 owner 在 issue 线程决策)。
  8. 是否引入了新的配置:无新配置(FORGED_SUMMARY_INJECT_WARN_PCT 是代码内命名常量,非 env/config 键)。

判定:REQUEST_CHANGES — 核心机制是死代码,PR 未达成其声称的目标(bound the ledger);按 directBlockIds 谱系重写剪枝谓词、补真实 kernel 状态的回归测试后再审。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review of PR #4629 (bound the codex forged-summary handoff ledger, part 1 of #4600) is complete.

Verdict: REQUEST_CHANGES (blocking). Posted as a comment review (GitHub blocks request-changes on my own account's PR): #4629 (review)

The core "bound the ledger" mechanism (lineage pruning) is dead code against real kernel state:

  • pruneForgedEntries (src/codex-compact.ts:401) prunes only when a consumer block's effectiveMessageIds contains acp_summary_<blockId> (lines 411/414-425/427). The kernel never puts summary ids there — kernel/src/unit-projection.ts:181 filters them out of the fold range, kernel/src/prune.ts:17 marks them VIEW-ONLY, and the kernel's own test kernel/tests/regression-promote-after-prune.test.ts:465 pins "synthetic id never leaks into block."
  • Verified empirically against the PR's exact kernel (0.0.199, built at head): a block consuming b2 gets directBlockIds=[b2], effectiveMessageIds=[raw-3,raw-4] — the PR's predicate finds zero consumers, so nothing is ever pruned. The persisted ledger still grows unboundedly (codexForgedSummaries grows without bound and is re-injected in full whenever a request lacks the echoed compaction item #4600's p1 not fixed); both pruning sites (injection-time + forge-time) call the dead predicate. The e2e test fabricates the forbidden state (effectiveMessageIds.push("acp_summary_…")), so it passes while real behavior is broken.
  • Suggested fix: key supersession on the kernel's actual lineage record — the transitive directBlockIds chain (an entry for X is redundant iff X inactive AND a represented Y reaches X via directBlockIds) — and re-test against a real applyCompression-produced state, not a hand-built block.

What works (no change needed): the carriesConversationTurns gate (removes the 413 for context-only probes/sidecars), the string[]→{text,blockId?}[] shape change with legacy read-compat (only reader is prepare-responses.ts), and the observability warn.

结构化八问:

  1. 是什么问题:PR 声称 bound the ledger,但核心谱系剪枝在真实 kernel 状态下永不触发(谓词恒假)。
  2. 什么原因导致的:kernel 从不把 acp_summary_* 写进任何 block 的 effectiveMessageIds(unit-projection.ts:181 显式过滤;prune.ts 标注 VIEW-ONLY;kernel 自身回归测试钉死);kernel 用 directBlockIds 记录谱系。PR 把"block 不含自己的 summary id"(真,仅防自剪枝)误当成"consumer 含旧 block 的 summary id"(假)。
  3. 影响范围与严重程度:持久化账本在 codexForgedSummaries grows without bound and is re-injected in full whenever a request lacks the echoed compaction item #4600 主场景(长会话嵌套 block)下仍无界增长,p1 未修复;注入时与 forge 时两个剪枝点都调用死谓词;e2e 伪造了 kernel 禁止的状态故通过。会话证据门有效,但只消除 context-only probe/sidecar 的 413,不消除增长。
  4. 怎么修复的:把取代判定键到 transitive directBlockIds 谱系上,用真实 applyCompression 状态(非手搓 block)重验并让回归测试钉住真实 kernel 行为。
  5. 是否有回归问题:本 PR 未合入,无既有行为被破坏;按建议修复后剪枝语义与 PR 原意一致(truncation debt 保留、被取代 entry 剪除),仅换正确信号。
  6. 是否破坏缓存:不涉及——账本是 session.metadata 内部键,非缓存/重试/增量游标。
  7. 是否改了设计方向:不涉及(发现一些bug #1064 handoff 契约不变;token 预算上限仍待 owner 在 issue 线程决策)。
  8. 是否引入了新的配置:无(FORGED_SUMMARY_INJECT_WARN_PCT 是代码内命名常量,非 env/config 键)。

判定:REQUEST_CHANGES — 核心机制是死代码,PR 未达成其声称的目标(bound the ledger);按 directBlockIds 谱系重写剪枝谓词、补真实 kernel 状态的回归测试后再审。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codex p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant