Skip to content

test: pin fold re-request / next-turn prefix parity guard (#2929) - #2933

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-12_issue2929-fold-seam-guard-test
Oct 12, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-12_issue2929-fold-seam-guard-test

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

References #2929 (codex hit-rate analysis) — this is the bili-side deliverable of that analysis; the issue itself stays open pending relay-side data.

What this is

A single new hermetic test (tests/issue2929-codex-fold-double-miss.test.ts) that pins the prefix-parity invariant between the two outbound-body construction paths that share one session:

  1. the compress loop's post-fold intra-turn re-request (refreshFolded → adapter buildRequest), and
  2. the next steady client turn's normal forward (prepare-responses).

It asserts both bodies are byte-identical through the summary anchor (the exact property whose violation produces the #1548/#1592/#377 seam family — "post-fold cache floor reset to the stable head").

Why (from #2929)

The reported 79% codex hit rate turned out to be dominated by a systematic pattern: after any fold whose post-fold prefix was FIRST served on an intra-turn loop re-request, the next client turn cached only the instructions baseline (~14.5K), then recovered fully on the following turn (6/6 folds behaved identically). Folds whose first exposure was a steady forward chained perfectly.

Full static audit of every candidate asymmetry (strict-echo gating, nudge placement, stripKernelSummaries carrier logic, repairResponsesAssistantOrdering, dropReasoning mirroring, item-id handling — both paths share patchResponsesInputWithToolImages) plus this probe (run with production-faithful fixtures: native ids on all items, persisted reasoning items, nudge + proxy-tool round + mid-turn compress) shows current master emits identical bytes on both paths — no reproducible bili-side wire defect remains. The residual evidence points at the upstream relay's cache routing/sharding semantics (the deployment is multi-instance per #1724 logs; baseline-only hits plus invisible baseline drift 14508→10948 are inconsistent with a single byte-prefix store).

So there is no behavior fix to ship here — but the invariant above is load-bearing and currently un-pinned. If any future change to either path breaks it, this test fails with the first divergent item index instead of the regression surfacing as production cache misses.

Scope

  • 1 new test file, ~255 lines, zero source changes, no config surface, no kernel/pi-subagents changes.
  • Hermetic: local mock Responses upstream, listen(0) ports, no env gates, runs in <5s.
  • Verified green locally on rebased master head (3 fixture variants: plain / +reasoning items / +native ids).

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-12_issue2929-fold-seam-guard-test (2cba06e)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3269-pr.2933.42abdc6

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2933.tgz
npm install -g package

Install instructions are refreshed on each push.

ework-agent added 2 commits October 12, 2026 21:09
Issue #2929 found a systematic cache double-miss after folds whose post-fold
prefix was first served on an intra-turn loop re-request: the next client turn
hit only the instructions baseline. Static audit plus this probe show the two
construction paths (loop refreshFolded rebuild vs steady prepare-responses
forward) are byte-equivalent through the summary anchor on current master for
production-faithful clients (native item ids, reasoning items, nudge, proxy
tool round). This guard pins that parity so future changes to either path fail
loudly instead of regressing into the #1548/#1592 seam family.
… class)

The new guard defined its own bare-rmSync rmrf while all 143 sibling suites
import tests/tmp-rm.ts, whose retry ladder exists because loaded CI runners
(#1646/#1910) and Windows teardown under c=4 (#2908) make bare deletes flaky.
Aligns the file with repo convention; no behavior change.
@ranxianglei
ranxianglei force-pushed the 2026-10-12_issue2929-fold-seam-guard-test branch from 05ec446 to 2cba06e Compare October 12, 2026 13:12
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review 完成(这是本 agent 在 #2929 上开的防护性测试 PR,按 [PR] 流程核对代码 + 验证 + 收口,未重做)。已把分支 rebase 到当前 master 并补了一个小修,CI 在新 head 上运行中。

① 方向关(审原 issue):#2929 的结论链成立——6/6 Type-A fold 的「基线-only 命中 + 一轮恢复」签名、bili 侧全路径静态审计 + 本地探针字节级实测均指向 relay 多实例缓存语义而非 bili wire 缺陷;「不变量当前成立但无钉」→ 用 hermetic 回归测试钉住,是正确的最小交付。同意不硬造行为修复。

② 重复判定:与既有 seam 家族 pin 有交集但非重复——responses-fold-cache-seam.test.ts(#1548) 钉同一「re-request/next-turn 前缀字节等价」不变量,但其 fixture 是 COMPRESS_AT=6、默认 50K nudge floor(nudge 从未触发)、无前置代理工具轮;本 PR 的 fixture 正是 #2929 的生产触发形态(nudgeGrowthTokens=2000 强制 T1 nudge 进 pre-fold 请求 + 同轮 acp_status 轮 + 每项原生 id + 持久化 reasoning item),且带自校验断言(nudge 没落进 pre-fold 请求就大声失败,不会静默测空)。互补覆盖,保留。

③ 代码审查:单文件 tests/issue2929-codex-fold-double-miss.test.ts(+255,零源码改动,diffstat 核实)。断言有牙:逐 item 字节比较到 summary anchor、失败信息带首个分歧索引;场景自校验(assert (0) nudge 存在)防止 fixture 漂移后假绿;hermetic(mock Responses upstream + listen(0),无 env 门,实测 ~3.5s)。无 spawner 模式 → 不需要 LIVE_FILES 注册(scripts/run-ci-tests.mjs 的 SPAWN_RE 自查已核)。

顺手小修(已推独立标注 commit 2cba06e3d):新文件自带了一个裸 fs.rmSync 版 rmrf,而全部 143 个兄弟 suite 都 import tests/tmp-rm.ts(带 ENOTEMPTY/EBUSY/EPERM/EACCES/EMFILE 重试阶梯,#1646/#1910/#2908 专为高负载 CI 与 Windows c=4 teardown 而设)。已改为共享 helper,无行为变化。

rebase + 定向验证:PR 原 base 落后 master ~20 提交(含 #2735 preflight-codex-shape,动了 codex wire 形状,与本家族相邻)→ 已 rebase 到 9972ff2df,无冲突。本地定向跑(非全量,全量交 CI):

CI:新 head 2cba06e3d 的 CI 运行中(PR Build Artifact 腿已过;旧 head 的 run 因 force-push 自动取消)。红了我会定向排查并跟进。

结构化八问

  1. 是什么问题:codex 命中率问题分析 #2929 定位的 fold 双失配在 bili 侧无可复现缺陷,但「loop 重请求体与下一稳态轮请求体在 summary 锚点内字节等价」这一承重不变量此前只有部分 fixture 形状被钉住,生产触发形态(nudge+轮内工具轮+轮中 fold)无守卫。
  2. 什么原因导致的:不是缺陷而是覆盖缺口——既有 codex 版本压缩后缓存大幅度降低问题 #1548 pin 的 fixture 不含 nudge/代理工具轮,codex 命中率问题分析 #2929 的生产签名恰好走那条未钉路径。
  3. 影响范围与严重程度:纯测试新增,零生产影响;价值在未来——任何改动破坏该不变量时会在 CI 以首个分歧索引失败,而不是变成线上缓存失配才被发现。
  4. 怎么修复的:hermetic 复现生产流量形状(mock upstream 固定索引回 acp_status/compress function call,真实 bili proxy 进程),对两条出站构造路径的捕获体做锚点内逐字节断言,外加场景自校验防假绿。
  5. 是否有回归问题:无源码改动 → 无既有行为可破;唯一触碰面是 CI 调度(新文件被 tests/*.test.ts glob 自动拾取,非 spawner,无需注册),已核实。
  6. 是否破坏缓存:不涉及(测试自身不产生缓存状态;调查反而确认稳态链式缓存正常)。
  7. 是否改了设计方向:否。
  8. 是否引入新的配置:否(用的全是既有配置项,nudgeGrowthTokens 等仅在测试内设置)。

判定:可合——待 CI 全绿(合并由人执行:#2933 )。#2929 本体保持打开,等 dump 复现会话/relay 方答复做最终归因。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI 收口确认:新 head 2cba06e3d(rebase 到 9972ff2df + tmp-rm 小修)的 CI 全绿——17/17 job success、0 skipped(含 ubuntu/windows × node 22/24 双矩阵的 test 腿与 static-gates,无矩阵腿被跳过)。本地定向验证(新测试 + #1548/#1592 相邻 seam 守卫)此前也已 green。

结论不变:可合,合并由人执行:#2933 。#2929 本体保持打开等 relay 侧数据。

@ranxianglei
ranxianglei merged commit d9de589 into master Oct 12, 2026
18 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.333-master.d9de589 (rolling master tag).

npm install -g billion-context@0.0.333-master.d9de589  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei
ranxianglei deleted the 2026-10-12_issue2929-fold-seam-guard-test branch October 15, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant